system

Abstract

A system is provided.SOLUTION: A system including a means for receiving a service request from a user terminal in natural language, a means for registering and managing a list of APIs provided by registrants and a usage method in a database, a means for analyzing the request of the user by using natural language processing and specifying an appropriate API, a means for generating an API calling script by using a generated AI, a means for calling the API by executing the generated script, and a means for returning a calling result to the user.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Currently, when users use multiple online services, they must operate each service individually. This makes operations cumbersome and makes efficient use difficult. Furthermore, there is a lack of a mechanism to automate the linking between different services, so users have to manually select and operate each required service, which is time-consuming. This reduces user convenience and makes it difficult to expect an improvement in conversion rates. [Means for solving the problem]

[0005] To solve the above-mentioned problems, the present invention provides a system that receives service requests from user terminals in natural language and registers and manages a list of APIs provided by registrants and their usage methods in a database. The system also includes a means for analyzing the user's requests using natural language processing and identifying an appropriate API, and generates a script for calling the API using a generation AI. The system then includes a means for executing the generated script and calling the API, and returns the call results to the user, allowing the user to use multiple services seamlessly. The system also includes a means for linking multiple APIs according to the service the user desires, and a means for calling APIs and returning the results through chat-style interactions, allowing the user to use different services in an integrated manner.

[0006] "User terminal" refers to a device such as a computer device, smartphone, or tablet that a user operates when using a service.

[0007] "Service requests" refer to the specific service contents and requests that a user wishes to use, and are often expressed in natural language.

[0008] "Natural language processing" is a technology that allows computers to understand, interpret, and generate human language.

[0009] "API" stands for Application Program Interface, a definition and protocol for different software applications to communicate with each other.

[0010] A "database" is a system for organizing data and efficiently storing, retrieving, and managing it.

[0011] "Generative AI" refers to artificial intelligence that generates new data or scripts based on given input.

[0012] A "script" is program code written to automate a particular task.

[0013] "Error handling" refers to the process by which a program recognizes errors and takes appropriate action.

[0014] "Receiving in natural language" refers to receiving and processing text or voice input from the user in its original form.

[0015] "Chat format" is a text-based interactive communication style in which users and the system send and receive messages alternately.

[0016] "Collaboration" refers to multiple systems or services working together and communicating with each other.

[0017] "Seamless" refers to a state of smooth continuity without interruption, meaning that users can use different services without being aware that they have switched operations between them. [Brief explanation of the drawings]

[0018] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8]FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0019] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0020] First, the terms used in the following description will be explained.

[0021] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0022] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0023] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0024] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0025] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0026] [First embodiment]

[0027] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0028] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0029] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0030] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0031] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0032] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0033] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0034] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0035] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0036] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0037] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0038] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0039] The present invention provides a platform where anyone can register and use an AI account that uses a generating AI. Users can converse with the AI ​​account on this platform and seamlessly use multiple services. Specific embodiments of the present invention are described below.

[0040] Program Generation and Processing Description

[0041] server:

[0042] The server provides a platform environment that users and registered users can access. The main roles of the server are as follows:

[0043] The server registers and manages a list of APIs provided by registrants and their usage methods in a database. Registrants access the server through their terminals and enter information about the APIs they provide. This information includes the API name, endpoint, usage method, required input parameters, and output items.

[0044] The server also receives and analyzes natural language requests from users, using natural language processing technology to understand the service desired by the user and identify the corresponding API.

[0045] Based on the identified API, the server uses a generation AI to generate the appropriate script to call the API as desired by the user. The generated script is then executed by the server to call the specified API.

[0046] Subscriber Device:

[0047] The registrant terminal is a device that connects to the server and registers API information. The registrant accesses the server from the terminal and enters the API name, endpoint, usage method, input parameters, and output items. The registrant also provides additional use cases and prompts to explain in detail how to use the API.

[0048] User device:

[0049] The user terminal is a device that allows users to access the platform and converse with the AI ​​account. The user selects the AI ​​account they want to use from their terminal and inputs the desired service details in chat format. For example, if a user inputs "I would like to reserve a hotel and restaurant for 2 nights and 3 days in Tokyo," the server receives this request.

[0050] The server analyzes the request, identifies the corresponding accommodation and restaurant reservation API, uses the generative AI to generate the necessary script, and calls the API to make the reservation. After the reservation is completed, the server returns the result to the user.

[0051] Examples:

[0052] Example 1: Travel booking

[0053] User:

[0054] The user is planning a three-day, two-night trip to Tokyo and would like to make hotel and restaurant reservations.

[0055] Operation steps:

[0056] 1. User terminal:

[0057] The user chats with the AI ​​account and says, "I'd like to book a hotel and restaurant in Tokyo for 2 nights and 3 days."

[0058] 2. Server:

[0059] The server analyzes the request and identifies the accommodation and restaurant reservation APIs.

[0060] 3. Server:

[0061] The generating AI generates a script for reservations and embeds the necessary parameters (date of stay, number of people, budget, etc.).

[0062] 4. Server:

[0063] Execute the generated script and make an API call.

[0064] 5. Server:

[0065] Receives accommodation and restaurant reservation results and returns the results to the user.

[0066] Example 2: Linking multiple AI accounts

[0067] User:

[0068] Users want their AI assistant to help them shop online and check the weather at the same time.

[0069] Operation steps:

[0070] 1. User terminal:

[0071] Users can chat with the AI ​​account and ask, "Tell me today's shopping recommendations and weather information."

[0072] 2. Server:

[0073] The server identifies the shopping and weather APIs and generates the necessary scripts.

[0074] 3. Server:

[0075] If necessary, add other AI accounts to the conversation and call their respective APIs.

[0076] 4. Server:

[0077] It retrieves shopping and weather information and returns it to the user.

[0078] This system allows users to seamlessly link different services together, greatly improving convenience and efficiency.

[0079] The processing flow will be explained below.

[0080] Step 1:

[0081] Subscriber Device:

[0082] The registrant accesses the server from a terminal and opens the API registration screen. The registrant enters the API name, endpoint, usage method, required input parameters, and output items. In addition, the registrant enters API usage examples and prompts, providing a detailed explanation of how to use the API. After confirming the entered information, the registrant submits the registration to the server.

[0083] Step 2:

[0084] server:

[0085] The server analyzes the received API registration details and stores them in a database. It then verifies the API usage and sample code to ensure there are no problems. Once verification is complete, it returns a registration completion message to the registrant's terminal.

[0086] Step 3:

[0087] User device:

[0088] Users access the platform from their device and select the AI ​​account they want to use. On the selected chat screen, they enter the desired service details in natural language. For example, they could enter, "I would like to book a hotel and restaurant in Tokyo for 2 nights and 3 days."

[0089] Step 4:

[0090] server:

[0091] The server analyzes messages received through the chat system, uses natural language processing to analyze the user's wishes and identify the necessary API, and prepares data to automatically generate API parameters based on the analysis results.

[0092] Step 5:

[0093] server:

[0094] The server calls the generation AI engine and generates a script for calling the API. It provides the "identified API name" and "required parameters" to the generation AI and instructs it to generate the script. It then performs syntax checks and logic verification on the generated script.

[0095] Step 6:

[0096] server:

[0097] Add error handling logic to complete the final script. Run the completed script in the execution environment. Call the specified API and obtain the results in real time. Check that the obtained results follow the correct format.

[0098] Step 7:

[0099] server:

[0100] The acquired data is formatted based on the user's wishes. The formatted results are sent back to the user's device. For example, a message such as "Your hotel reservation for 2 nights and 3 days in Tokyo has been completed. Details are as follows:..." is sent.

[0101] Step 8:

[0102] User device:

[0103] The user can check the results on their device and select the next action. For example, they may send another message saying, "Please make a restaurant reservation as well."

[0104] Step 9:

[0105] server:

[0106] The server receives a new request from the user, prepares a new API call while preserving the existing conversation context, uses the generative AI again to generate the necessary scripts and make additional API calls, and then integrates all the results and returns the combined information to the user.

[0107] Step 10:

[0108] User device:

[0109] The user can check the final result on the device and send feedback if necessary, such as a message saying, "I have confirmed the reservation details. There are no problems."

[0110] Step 11:

[0111] server:

[0112] The server receives the feedback and stores it in a database for service improvement purposes. It then sends a message to the user to end the conversation and terminates the session.

[0113] In this way, multiple services that use generative AI can be used in seamless collaboration.

[0114] Example 1

[0115] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0116] Conventional systems make it difficult for users to seamlessly use a variety of services. In particular, they require the manual integration of multiple application programming interfaces (APIs), a complex and time-consuming process. Furthermore, there was a lack of automated means for analyzing natural language requests, selecting the appropriate API, and generating the corresponding program. As a result, users were required to have high operational skills, and the system was difficult to use.

[0117] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0118] In this invention, the server includes means for receiving a service request from a user terminal in natural language, means for registering and managing a list of application programming interfaces provided by registrants and usage methods in a data storage device, means for analyzing the user's request using natural language processing and identifying an appropriate application programming interface, means for generating a program for calling the application programming interface using generative artificial intelligence, means for executing the generated program and calling the application programming interface, and means for returning the call result to the user. This enables users to seamlessly link and use a variety of services through requests in natural language.

[0119] "User terminal" means an electronic device through which a user inputs service requests and interacts with the system.

[0120] "Service requests" refers to requests or requests made by users regarding the content of services provided through the system.

[0121] "Natural language" refers to the words and forms of expression used by humans on a daily basis, usually expressed in text form.

[0122] A "server" is a computer system that processes requests from user terminals and registrant terminals and provides data and services.

[0123] "Registrant" means a person or entity that provides and manages one or more application programming interfaces (APIs) to a system.

[0124] An "application programming interface (API)" is a set of definitions or protocols that allow software components to communicate.

[0125] "Information management" is the process of using a database to register, update, and delete API information and maintain how it is used.

[0126] "Natural language processing" refers to the use of computers to analyze, understand, and generate natural language, for example through the use of text analysis and language models.

[0127] "Generative AI" is a general term for AI techniques for performing generative tasks (such as text generation and code generation), using, for example, generative models and neural networks.

[0128] "Program generation" is the process of automatically creating code or scripts to perform a specific task.

[0129] An "API call" refers to the act of sending a request from software to an API to use its functions or data.

[0130] A "data storage device" is a hardware or software system for the long-term storage and management of data.

[0131] "Conversational" refers to interactions between users and systems, such as text responses via a chatbot.

[0132] The present invention relates to a system that uses a user terminal, a server, and a registered user terminal to utilize a generative AI model and provide a platform that allows seamless use of a variety of services.

[0133] server

[0134] The server performs the following main functions:

[0135] 1. Receiving a request from a user device

[0136] The server receives service requests from the user terminal in natural language using a communication protocol such as HTTP or WebSocket.

[0137] 2. API information registration and management

[0138] The server uses a data storage device to register and manage a list of application programming interfaces (APIs) provided by registrants and their usage methods in a database. Typically, a relational database such as MySQL or PostgreSQL is used.

[0139] 3. Natural Language Processing

[0140] The server analyzes user requests using natural language processing models such as BERT and GPT-3 to understand the service desired by the user and identify the appropriate API.

[0141] 4. Script generation using generative AI models

[0142] Based on the analysis results, the server uses a generative AI model (e.g., GPT-4) to generate a script for calling the API, including the appropriate endpoints and parameters.

[0143] 5. Calling the API

[0144] The server executes the generated script and calls the specified API, and the response from the API is received by the server and processed as needed.

[0145] 6. Return of results

[0146] Finally, the server returns the results of the API call to the user's device, which is a specific response based on the service content requested by the user.

[0147] Registered device

[0148] The Registrant Terminal performs the following functions:

[0149] 1. Enter API information

[0150] The registrant enters the API name, endpoint, usage method, input parameters, and output items into the server through a terminal, which can be done via a web interface.

[0151] 2. Providing use cases and prompts

[0152] Registrants may provide additional use cases and prompts to provide detailed instructions on how to use the API, thereby helping consumers to more easily use the API.

[0153] User terminal

[0154] The user terminal is responsible for the following functions:

[0155] 1. Enter your service request

[0156] Users input the desired service details in chat format, and the details of the request are sent to the server as a request in natural language.

[0157] 2. Receiving and displaying results

[0158] The results returned from the server are displayed on the user's terminal, and the user performs the next operation based on these results.

[0159] Specific examples

[0160] Example 1: Travel booking

[0161] When a user enters "I would like to book a hotel and restaurant in Tokyo for 2 nights and 3 days," the server analyzes the request and identifies the appropriate accommodation and restaurant reservation API. Using generative AI, it generates the necessary script and calls the API to make the reservation. The results are sent back to the user's device.

[0162] Example 2: Linking multiple AI accounts

[0163] When a user types "Tell me today's shopping recommendations and weather information," the server identifies the APIs for shopping and weather information, generates the necessary scripts, and, if necessary, invites other AI accounts into the conversation and calls their respective APIs. The results are then sent back to the user's device.

[0164] Prompt Sentence Examples

[0165] 1. "Create a 3-day, 2-night trip to Tokyo."

[0166] 2. "I'd like to book a hotel in Tokyo for two nights starting February 20th."

[0167] 3. "Tell me what to buy today and what the weather is like."

[0168] 4. "Provide current weather information along with online shopping."

[0169] As described above, the embodiment of the present invention allows users to seamlessly use multiple services through requests in natural language.

[0170] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0171] Step 1:

[0172] (User request input)

[0173] The user inputs their service request in natural language using the user terminal. For example, they input a request such as "I would like to reserve a hotel and restaurant in Tokyo for 2 nights and 3 days." This request is sent from the user terminal to the server.

[0174] input:

[0175] A user's natural language request (e.g., "I would like to book a hotel and restaurant for a 2-night, 3-day stay in Tokyo.")

[0176] output:

[0177] A natural language request is sent to the server

[0178] Specific behavior:

[0179] The user enters the desired service details in the chat window and clicks the send button. The user's terminal generates an HTTP request to send the entered details to the server.

[0180] Step 2:

[0181] (receive and analyze the request by the server)

[0182] The server receives a natural language request from the user's device. Next, it uses natural language processing technology (e.g., BERT or GPT-3 models) to analyze the request and understand the service desired by the user. From the analysis results, it extracts key information (e.g., location, duration, type of service, etc.).

[0183] input:

[0184] A natural language request received by the server

[0185] output:

[0186] Extracted key information (e.g., "Tokyo," "2 nights and 3 days," "hotel," "restaurant")

[0187] Specific behavior:

[0188] The server receives an HTTP request and feeds the text portion of the request into a natural language processing model, which analyzes the text and extracts important keywords and parameters.

[0189] Step 3:

[0190] (Server-specific API)

[0191] Based on the analysis, the server identifies the appropriate API, for example, "Hotel Reservation API" and "Restaurant Reservation API," which involves querying a database to select the appropriate API from a list of applicable APIs.

[0192] input:

[0193] Parsed key information

[0194] output:

[0195] List of identified APIs (e.g., "Hotel Booking API," "Restaurant Booking API")

[0196] Specific behavior:

[0197] The server queries the database to obtain API information that matches the analysis results, and then lists the endpoints and parameters of the obtained APIs.

[0198] Step 4:

[0199] (Script generation by generative AI)

[0200] The server uses a generative AI model (e.g., GPT-4) to generate a script to call the identified API. This script embeds the necessary parameters (e.g., location, period, number of people, etc.) and makes the API call.

[0201] input:

[0202] List of identified APIs and analyzed key information

[0203] output:

[0204] Generated API call script

[0205] Specific behavior:

[0206] The server inputs the API list and key information to the generation AI and requests it to generate a script. The generation AI generates the script and returns it to the server.

[0207] Step 5:

[0208] (API call by server)

[0209] The server executes the generated script and calls the specified API, the response from the API is received by the server and further processed as needed.

[0210] input:

[0211] Generated API call script

[0212] output:

[0213] Response data from the API (e.g., reservation confirmation information)

[0214] Specific behavior:

[0215] The server runs the generated script, calls the API via an HTTP request, receives the API response, and formats the data if necessary.

[0216] Step 6:

[0217] (Return of results)

[0218] The server returns the results of the API call to the user, who can then view the results on their device.

[0219] input:

[0220] Response data from the API

[0221] output:

[0222] Results displayed on the user's device

[0223] Specific behavior:

[0224] The server converts the received response data into a convenient format and sends it to the user's terminal as an HTTP response. The user can then check the results, such as a reservation completion notification, on their terminal.

[0225] (Application example 1)

[0226] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0227] Conventional online shopping systems have had difficulty seamlessly providing the services users need, such as detailed product information, stock availability, and purchasing procedures. In particular, centralizing multiple services and automatically linking them according to user requests has been a challenge.

[0228] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0229] In this invention, the server includes means for receiving service requests in natural language from a user terminal, means for registering and managing a list of APIs provided by registrants and usage methods in a database, means for analyzing the user's requests using natural language processing and identifying an appropriate API, means for generating a script for calling the API using a generation AI, means for executing the generated script and calling the API, means for returning the call results to the user, and means for the user to receive various services such as detailed product information, stock confirmation, and purchase procedures in the virtual store, thereby enabling the user to seamlessly complete a series of shopping procedures in the virtual store.

[0230] A "user terminal" is a device for inputting service requests in natural language and communicating with a server to receive information and results.

[0231] A "service request" is a request that specifies the specific service content that the user wants to receive.

[0232] "Natural language" is the language used by humans in everyday life, and includes informal words and sentences.

[0233] "API" stands for Application Programming Interface, an interface for exchanging functions between software programs.

[0234] A "database" is a system for systematically storing large amounts of data and efficiently searching, updating, and managing them.

[0235] "Natural language processing" is a technology that allows computers to understand, analyze, and generate human language.

[0236] "Generative AI" is an artificial intelligence technology that automatically generates new scripts and content based on input data.

[0237] A "script" is a program code that contains a set of instructions for performing a particular task or process.

[0238] A "virtual store" is a store operated on the Internet, a virtual space that offers products and services.

[0239] "Checking inventory" means checking how much of a particular product is currently available for purchase.

[0240] The "purchase procedure" refers to the process of all the steps required to purchase the selected product.

[0241] This invention is a system that provides AI accounts using generation AI in virtual stores, allowing users to provide detailed product information, check inventory, and complete purchase procedures seamlessly.

[0242] System Configuration

[0243] The system consists of the following main components:

[0244] 1. Server:

[0245] Technologies used: Python, Flask, TensorFlow

[0246] function:

[0247] A service request is received in natural language from a user terminal.

[0248] A database manages a list of APIs provided by registrants and their usage instructions.

[0249] It uses natural language processing to analyze user preferences and identify the appropriate API.

[0250] Use generative AI to generate scripts for API calls.

[0251] Run the generated script to call the API.

[0252] The call result is returned to the user.

[0253] 2. User terminal:

[0254] Technologies used: Kotlin or Swift (for smartphones), ARCore or ARKit (for smart glasses)

[0255] function:

[0256] The user inputs the service request in natural language and sends it to the server.

[0257] The information and results received from the server are displayed and read aloud.

[0258] Operation overview

[0259] User operation

[0260] Customers access the virtual store using an app on their smartphone or smart glasses, and then make requests by voice or text, such as "Please let me know if the Nike Air Max 2023 is in stock."

[0261] Server-side processing

[0262] 1. The server receives a natural language request from the user.

[0263] 2. Use natural language processing technology to analyze the request and identify the appropriate API.

[0264] 3. Use generative AI (using TensorFlow) to generate scripts for API calls corresponding to the request.

[0265] 4. Execute the generated script and call the product information API to retrieve the required data.

[0266] 5. The acquired information is sent back to the user's terminal.

[0267] Server example

[0268] Example prompt sentence:

[0269] A user wants more information about the following product: "Nike Air Max 2023". Get the required information from your API.

[0270] Specific examples of user terminals

[0271] When a user voice-inputs, "Please tell me the stock status of the Nike Air Max 2023," the request is sent to the server, which analyzes and processes it, sends the results back to the device, and displays a message such as "In stock: 10 left" on the smartphone screen.

[0272] Overall system flow

[0273] This system allows users to seamlessly obtain product information, check inventory, and complete purchase procedures within the virtual store, greatly improving the convenience of shopping. Users input their service requests in a chat format, and the server automatically calls the optimal API to provide the necessary information, making purchasing activities more efficient.

[0274] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0275] Step 1:

[0276] A user accesses a virtual store through a smartphone or smart glasses app and inputs their service request in natural language, for example, "Please tell me the stock status of the Nike Air Max 2023." This input is sent by the device to the server.

[0277] Step 2:

[0278] The server receives the input natural language request and analyzes the request using natural language processing technology. Specifically, it analyzes the input text and extracts intent such as "product information" or "inventory check." The input is a natural language request, and the output is the analyzed intent data.

[0279] Step 3:

[0280] The server identifies an appropriate API from a database based on the analyzed intent data. The database contains a list of multiple APIs and their usage methods. The analyzed intent data is the input, and the identified API is the output.

[0281] Step 4:

[0282] The server uses a generative AI model (using TensorFlow) to generate a script for calling the API. The user's request and the identified API information are input into the generative AI model, and the API call script is output. For example, a "script to call the Nike Air Max 2023 stock confirmation API" is generated.

[0283] Step 5:

[0284] The server executes the generated script and calls the specified API to obtain the required data. As a result of the API call, product information and stock status are obtained. The input is the API call script, and the output is the obtained product data.

[0285] Step 6:

[0286] The server processes the acquired data as necessary and converts it into an appropriate format before returning it to the user. For example, it converts inventory information into an easy-to-understand format such as "In stock: 10 remaining." The input is the acquired product data, and the output is the processed information data.

[0287] Step 7:

[0288] The terminal provides the information received from the server to the user by displaying it on the screen or outputting audio. For example, the display of a smartphone or smart glasses might show "In stock: 10 remaining." The input is the information data returned from the server, and the output is display data or audio data.

[0289] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0290] This invention provides a platform where anyone can use and register an AI account using generative AI. In particular, this invention combines an emotion engine that recognizes the user's emotions to provide more personalized services.

[0291] Program Generation and Processing Description

[0292] server:

[0293] The server provides a platform environment that users and registered users can access. The main roles of the server are as follows:

[0294] The server registers and manages a list of APIs provided by registrants and their usage methods in a database. Registrants access the server via their terminals and enter information about the APIs they provide. This includes the API name, endpoint, usage method, required input parameters, and output items.

[0295] Next, the server receives a request in natural language from the user. It uses natural language processing technology to understand the service desired by the user and identifies the corresponding API. Based on the identified API, it uses generative AI to generate an appropriate script. This script is used to call the API according to the user's request.

[0296] The server also has an emotion engine that analyzes the user's chat content to identify their emotional state. This emotional information is used to adjust the script content generated by the AI. For example, if the user is feeling stressed, the script output can be changed to a soothing message.

[0297] Subscriber Device:

[0298] The registrant terminal is a device that connects to the server and registers API information. The registrant accesses the server from the terminal and enters the API name, endpoint, usage method, input parameters, and output items. It also provides examples and prompts for users to use the API in practice, and provides detailed instructions on how to use the API.

[0299] User device:

[0300] The user terminal is a device that allows users to access the platform and converse with the AI ​​account. The user selects the AI ​​account they want to use from their terminal and inputs the desired service details in chat format. For example, if a user inputs "I would like to reserve a hotel and restaurant for 2 nights and 3 days in Tokyo," the server receives this request.

[0301] The server analyzes the request, identifies the corresponding accommodation and restaurant reservation APIs, uses generative AI to generate the necessary scripts, and calls the APIs to make the reservation. Once the reservation is complete, the server returns the results to the user.

[0302] An emotion engine is also integrated into this process, analyzing the user's emotional state in real time. For example, if the system detects that the user is tired, it can send an additional message along with the booking result, such as "We recommend a relaxing hotel."

[0303] Examples:

[0304] Example 1: Travel booking

[0305] User:

[0306] The user is planning a three-day, two-night trip to Tokyo and wants to make hotel and restaurant reservations. During the chat, the emotion engine detects that the user is feeling stressed.

[0307] Operation steps:

[0308] 1. User terminal:

[0309] The user chats with the AI ​​account and says, "I'd like to book a hotel and restaurant in Tokyo for 2 nights and 3 days."

[0310] 2. Server:

[0311] The server analyzes the request and identifies the accommodation and restaurant reservation APIs.

[0312] 3. Server:

[0313] The generating AI generates a script for reservations and embeds the necessary parameters (date of stay, number of people, budget, etc.).

[0314] The emotion engine detects the user's stress level and adjusts the content of the generated script.

[0315] 4. Server:

[0316] Execute the generated script and make an API call.

[0317] 5. Server:

[0318] It receives the results of accommodation and restaurant reservations and returns them to the user with a message such as, "We recommend a relaxing hotel."

[0319] Example 2: Linking multiple AI accounts

[0320] User:

[0321] Users want to shop online and check the weather at the same time. The emotion engine recognizes that users enjoy themselves.

[0322] Operation steps:

[0323] 1. User terminal:

[0324] Users tell the AI ​​account, "Tell me today's recommended shopping and weather information."

[0325] 2. Server:

[0326] The server identifies the shopping and weather APIs and generates the necessary scripts.

[0327] The emotion engine recognizes the user's enjoyment and adapts the generated script accordingly.

[0328] 3. Server:

[0329] If necessary, add other AI accounts to the conversation and call their respective APIs.

[0330] 4. Server:

[0331] It obtains shopping and weather information and returns it to the user along with a message such as, "Today's weather is sunny. Perfect day for shopping!"

[0332] In this way, by combining generative AI and an emotion engine, the present invention can provide optimal services according to the user's emotional state, further improving user satisfaction and convenience.

[0333] The processing flow will be explained below.

[0334] This invention provides a platform where anyone can use and register an AI account using generative AI. In particular, this invention combines an emotion engine that recognizes the user's emotions to provide more personalized services.

[0335] Program Generation and Processing Description

[0336] server:

[0337] Step 1:

[0338] server:

[0339] A list of APIs provided by registrants and their usage methods are registered and managed in a database. Registrants access the server from their terminal and enter information about the APIs they provide. This includes the API name, endpoint, usage method, required input parameters, and output items. This allows the API corresponding to the service desired by the user to be saved in the database.

[0340] Step 2:

[0341] server:

[0342] It receives requests in natural language from users. It uses natural language processing technology to understand the service desired by the user and identify the appropriate API. For example, if a request such as "I would like to book a hotel and restaurant in Tokyo for 2 nights and 3 days" is received via chat, it will be analyzed.

[0343] Step 3:

[0344] server:

[0345] The emotional engine analyzes the user's emotional state from the chat content, thereby identifying the stress or joy the user is feeling.

[0346] Step 4:

[0347] server:

[0348] It uses generative AI to generate scripts to call the identified APIs, and the generated scripts are tailored to the user's emotional state—for example, if they're feeling stressed, the script will be tailored to be relaxing.

[0349] Step 5:

[0350] server:

[0351] Execute the generated script and call the API. The specified API returns data in real time, and the server receives that data.

[0352] Step 6:

[0353] server:

[0354] It then formats the data it receives based on the user's preferences and generates additional messages as needed, such as "We recommend a hotel where you can relax."

[0355] Step 7:

[0356] server:

[0357] The formatted result is sent back to the user's terminal, where the user can receive the result and select the next action.

[0358] Examples:

[0359] Example 1: Travel booking

[0360] User:

[0361] The user is planning a three-day, two-night trip to Tokyo and wants to make hotel and restaurant reservations. During the chat, the emotion engine detects that the user is feeling stressed.

[0362] Operation steps:

[0363] Step 1:

[0364] User device:

[0365] The user chats with the AI ​​account and says, "I would like to book a hotel and restaurant in Tokyo for 2 nights and 3 days."

[0366] Step 2:

[0367] server:

[0368] The server analyzes the request and identifies the accommodation and restaurant reservation APIs.

[0369] Step 3:

[0370] server:

[0371] The generating AI generates a script for reservations and embeds the necessary parameters (date of stay, number of people, budget, etc.).

[0372] Step 4:

[0373] server:

[0374] The emotion engine detects the user's stress level and adjusts the content of the generated script.

[0375] Step 5:

[0376] server:

[0377] Execute the generated script and make an API call.

[0378] Step 6:

[0379] server:

[0380] It receives hotel and restaurant booking results and formats the results with a message such as "We recommend a relaxing hotel."

[0381] Step 7:

[0382] server:

[0383] The formatted result is sent back to the user's terminal.

[0384] Example 2: Linking multiple AI accounts

[0385] User:

[0386] Users want to shop online and check the weather at the same time. The emotion engine recognizes that users enjoy themselves.

[0387] Operation steps:

[0388] Step 1:

[0389] User device:

[0390] Users tell the AI ​​account, "Tell me today's recommended shopping and weather information."

[0391] Step 2:

[0392] server:

[0393] The server identifies the shopping and weather APIs and generates the necessary scripts.

[0394] Step 3:

[0395] server:

[0396] The emotion engine recognizes the user's enjoyment and adapts the generated script accordingly.

[0397] Step 4:

[0398] server:

[0399] If necessary, add other AI accounts to the conversation and call their respective APIs.

[0400] Step 5:

[0401] server:

[0402] Get shopping and weather information and format the results with a message like "Today's weather is sunny. Great day for shopping!"

[0403] Step 6:

[0404] server:

[0405] The formatted result is sent back to the user's terminal.

[0406] In this way, by combining generative AI and an emotion engine, the present invention can provide optimal services according to the user's emotional state, further improving user satisfaction and convenience.

[0407] Example 2

[0408] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0409] Modern information systems require users to receive a variety of services quickly and in a personalized manner. However, current systems often lack the functionality to analyze the user's emotional state and reflect the results in the content of the service provided. As a result, services provided without considering the user's emotional state may reduce user satisfaction. Furthermore, when providing complex services by linking multiple APIs, there is a lack of an efficient process for identifying APIs and generating scripts. This poses the challenge of requiring a great deal of effort and time to meet user requests.

[0410] The specification process by the specification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes: means for receiving a service request from a user terminal in natural language; means for registering and managing a list of APIs provided by registrants and usage methods in a database; means for analyzing the user's request and identifying an appropriate API using natural language processing; means for generating a script for calling the API using a generative AI model; means for executing the generated script to call the API; means for returning the call result to the user; means including an emotion engine for analyzing the user's emotional state; and means for adjusting the content of the generated script depending on the user's emotional state. This makes it possible to quickly provide personalized services according to the user's emotional state and efficiently link multiple APIs.

[0411] "User terminal" means a device through which a user accesses the system and inputs service requests.

[0412] A "registrant" is a person who registers information about the API that they provide in the system.

[0413] "API" stands for Application Programming Interface, an interface for providing specific functions and data to the outside world.

[0414] A "database" is a system that stores information in an organized manner and allows it to be retrieved when needed.

[0415] "Natural language processing" is a technology that allows computers to understand and process human language.

[0416] A "generative AI model" is an algorithm that uses artificial intelligence to generate scripts for specific tasks or purposes.

[0417] A "script" is program code for automatically executing a series of instructions.

[0418] An "emotion engine" is a technology that analyzes a user's emotional state from their text or input.

[0419] "Call result" refers to the response or data obtained after executing an API.

[0420] "Natural language" refers to the language used by humans on a daily basis, translated into a form that the system can understand.

[0421] "Fusion of multiple APIs" refers to the use of multiple APIs simultaneously or sequentially to provide one or more services.

[0422] This invention is a highly personalized service provision system that utilizes a generative AI model and an emotion engine. This system has the function of calling the optimal API based on the user's wishes and emotional state and returning the results to the user.

[0423] System Overview

[0424] The system includes the following major hardware and software components:

[0425] Server: Responsible for managing the API, parsing requests, generating scripts, running the emotion engine, and returning results.

[0426] User terminal: A device (PC, smartphone, etc.) that allows a user to access the system and input service requests.

[0427] Subscriber terminal: A device used to add API information to the system.

[0428] Database (e.g. MySQL, PostgreSQL): Stores and manages registered API information.

[0429] Natural language processing engines (e.g. Google NLP API, spaCy): used to analyze user preferences.

[0430] A generative AI model (e.g., OpenAI GPT-3) used to generate scripts for API calls.

[0431] Emotion engine (e.g., Microsoft Azure Emotion API): Used to analyze the user's emotional state.

[0432] Detailed System Operation

[0433] 1. Receiving Service Requests:

[0434] The user inputs their desired service request in natural language from their device and sends it to the server. For example, a user might input, "I would like to reserve a hotel and restaurant for 2 nights and 3 days in Tokyo."

[0435] 2. Parse the request:

[0436] The server uses a natural language processing engine to analyze the request and understand the user's wishes, extracting the keywords "hotel reservation" and "restaurant reservation."

[0437] 3. Identify APIs and generate scripts:

[0438] The server identifies the APIs corresponding to "hotel reservation" and "restaurant reservation" from the database. Then, it generates a script using the generative AI model. For example, it sends a prompt such as "Please generate a script using the APIs for hotel reservation and restaurant reservation."

[0439] 4. Applying the Emotion Engine:

[0440] The server analyzes the user's chat content using an emotion engine to identify the user's emotional state. For example, if the user is feeling stressed, the emotion engine conveys that information. The server then adjusts the content of the generated script to include a message such as, "We recommend a relaxing hotel."

[0441] 5. API call and return results:

[0442] The server runs the generated script, calls the appropriate API, receives the response from the API, and returns the results to the user, including the specific reservation information and an optional message (e.g., "We recommend a relaxing hotel").

[0443] Specific examples

[0444] Book a trip

[0445] User: Planning a trip, type in chat, "I'd like to book a hotel and restaurant in Tokyo for 2 nights and 3 days."

[0446] Server: Receives the request and analyzes it using a natural language processing engine. It generates an appropriate script using a generative AI model and analyzes the user's emotions using an emotion engine. If stress is detected, it adds a soothing message to the script.

[0447] Execution and return: Call the API to execute the reservation and return the reservation results to the user terminal.

[0448] Multiple API integrations

[0449] User: Wants to shop online and check the weather at the same time, so types in "Tell me today's recommended shopping and weather information."

[0450] Server: Identifies the shopping and weather APIs and generates the necessary scripts. If the emotion engine determines that the user is having fun, it includes a message that matches this excitement.

[0451] Execution and return: Calls multiple APIs and returns the obtained shopping information and weather information to the user along with a message such as "Today's weather is sunny. It's a great day for shopping!"

[0452] In this way, by combining a generative AI model and an emotion engine, the present invention can provide optimal services according to the user's emotional state, thereby improving user satisfaction and realizing efficient and personalized service provision.

[0453] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0454] Step 1: Enter and submit your service request on the user's device

[0455] Specific operation: The user uses the user terminal to input the desired service details in chat format. For example, the user might input "I would like to reserve a hotel and restaurant in Tokyo for 2 nights and 3 days."

[0456] Input: A message of service request written in natural language.

[0457] Output: The request message sent to the server.

[0458] Step 2: Server receives and analyzes the request

[0459] Specific operation: The server receives the request sent from the user terminal and analyzes the request using a natural language processing engine (e.g., Google NLP API, spaCy). Through the analysis, the keywords "hotel reservation" and "restaurant reservation" are extracted.

[0460] Input: A natural language request from the user.

[0461] Output: Parsed preference (e.g. "Hotel reservation", "Restaurant reservation").

[0462] Step 3: Identify the corresponding API

[0463] Specific operation: The server refers to the database and identifies the API corresponding to the analyzed request. For example, it retrieves the "Hotel Reservation" API and the "Restaurant Reservation" API.

[0464] Input: Parsed desired content.

[0465] Output: Identified API (e.g. "Hotel Booking" API, "Restaurant Booking" API).

[0466] Step 4: Generating scripts using generative AI models

[0467] Specific operation: The server uses a generative AI model (e.g., OpenAI GPT-3) to generate a script for API calls based on the user's request. The prompt text is "Please generate a script using the hotel reservation and restaurant reservation APIs."

[0468] Input: The API identified and what you want.

[0469] Data processing: Send prompt text to the generative AI model to generate a script for calling the API.

[0470] Output: The generated script.

[0471] Step 5: Emotional state analysis by the emotion engine

[0472] Specific operation: The server analyzes the user's chat content using an emotion engine (e.g., Microsoft Azure Emotion API) to identify the user's emotional state. For example, it detects whether the user is feeling stressed.

[0473] Input: User chat content.

[0474] Data Computation: Analyze emotional states using an emotion engine.

[0475] Output: Detected emotional state (e.g., stress).

[0476] Step 6: Adjust the script content

[0477] Specific behavior: The server adjusts the content of the generated script based on the detected emotional state. For example, if a user is feeling stressed, it might add a message such as "We recommend a relaxing hotel."

[0478] Input: Generated script and emotional state.

[0479] Data processing: The message part of the script is changed depending on the emotional state.

[0480] Output: The adjusted script.

[0481] Step 7: Making API calls

[0482] Specific operation: The server executes the tailored script, calls the specified API, and receives the response from the API.

[0483] Input: The adjusted script.

[0484] Output: The response data from the API.

[0485] Step 8: Returning the results

[0486] Specific operation: The server returns the response data from the API to the user's device and displays the result, for example, information that the reservation was successful and a message such as "We recommend a relaxing hotel."

[0487] Input: Response data from the API.

[0488] Data processing: Integrate response data and emotional responses.

[0489] Output: Results returned to the user's terminal.

[0490] (Application example 2)

[0491] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0492] Conventional automated service provision systems have difficulty in providing personalized responses that take into account the user's emotions, making it difficult to increase user satisfaction. In particular, when providing services in physical stores, flexible and appropriate responses that take into account the customer's psychological state are required.

[0493] The specification processing by the specification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for receiving a service request from a user terminal in natural language, means for registering and managing a list of APIs provided by registrants and their usage methods in a database, means for analyzing the user's request using natural language processing and identifying an appropriate API, means for generating a script for calling the API using a generation AI, means for executing the generated script and calling the API, means for returning the call result to the user, means for analyzing the user's emotional state, and means for adjusting the content of the generated script based on the emotional state. This makes it possible to provide personalized services according to the user's emotional state.

[0494] A "user terminal" is a device that allows a user to input service requests.

[0495] "Service Desired" refers to the specific service content that the user wishes to use.

[0496] "Natural language" refers to the language that humans use on a daily basis, not a specialized language such as a program language.

[0497] "API" stands for Application Programming Interface, an interface for exchanging functions and data between software programs.

[0498] A "database" is a system for efficiently storing and managing data.

[0499] "Natural language processing" is a technology that allows computers to understand and analyze natural language.

[0500] "Generative AI" is an artificial intelligence technology that generates appropriate responses and results based on input data.

[0501] A "script" is a program code that contains a series of instructions or commands.

[0502] "Emotional state" refers to the user's psychological state or mood, including emotions such as stress and joy.

[0503] "Emotion analysis" is a technology that determines a user's emotional state based on facial expressions, tone of voice, etc.

[0504] An "API call" is an operation that executes a specified API to obtain data or functionality.

[0505] This invention is a system that provides personalized customer service in brick-and-mortar stores, and is realized using smart glasses and emotion analysis sensor devices. The main components of the system are the user terminal, emotion analysis sensor, generative AI, natural language processing, and a server that manages and executes each API.

[0506] Major hardware and software used

[0507] Hardware: smart glasses (e.g., Google Glass), sensor devices for emotion analysis (e.g., Microsoft Azure Kinect)

[0508] software:

[0509] Natural language processing libraries (e.g. spaCy)

[0510] Sentiment analysis engine (e.g. Microsoft Azure Emotion API)

[0511] Generative AI systems (e.g., OpenAI GPT-4)

[0512] API management and databases (e.g. AWS, Firebase)

[0513] System Operation Overview

[0514] 1. Obtaining customer sentiment data

[0515] The smart glasses' camera and emotion-analysis sensors capture the customer's facial expressions and tone of voice in real time, allowing them to determine their emotional state on the spot. For example, if a customer is feeling stressed, that information will be detected.

[0516] 2. Emotional Data Analysis

[0517] The acquired emotional data is sent to the cloud and analyzed by an emotion analysis engine. For example, the Microsoft Azure Emotion API is used to identify emotional states such as "high stress level" or "enjoyed" based on facial expressions and voice characteristics.

[0518] 3. Proposal generation using generative AI

[0519] Based on the analysis results, the generative AI system will make optimal product suggestions. For example, if a customer is feeling stressed, the generative AI will generate a script suggesting products with a relaxing effect and display it on the smart glasses' display. The generative AI uses OpenAI GPT-4 to generate natural and appropriate suggestion messages.

[0520] 4. Implementing product proposals

[0521] The staff member checks the information displayed on the smart glasses and suggests individual products and services to the customer. The staff member then introduces specific products and encourages the customer to try them according to the generated suggestion message. For example, the staff member might say, "This aroma diffuser is a recommended product to help you relax."

[0522] 5. Feedback and Adjustments

[0523] After making a proposal, the system reanalyzes the customer's reactions (facial expressions and tone of voice) and adjusts the proposal if necessary, allowing the system to continue providing optimal service based on customer feedback.

[0524] Specific prompt examples

[0525] For example, in a brick-and-mortar scenario:

[0526] Prompt Sentence Examples

[0527] "The customer's facial expression is stressed and their voice tone is low. They are currently in the relaxation section of the store. The service they offer is product suggestions. The priority keywords are relaxation, healing, and stress relief."

[0528] In this way, the system of the present invention can provide personalized services in physical stores according to the emotional state of the user, thereby achieving high customer satisfaction.

[0529] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0530] Step 1:

[0531] The smart glasses' camera and emotion analysis sensor are used to capture the customer's facial expressions and tone of voice in real time, and the data is acquired. The input is the customer's image and voice data, and the output is the captured emotion data. The acquired data is sent to the server.

[0532] Step 2:

[0533] The server sends the acquired emotional data to the cloud and analyzes it using an emotion analysis engine (e.g., Microsoft Azure Emotion API). The input is the captured emotional data, and the output is the analyzed emotional state (e.g., "stressed," "enjoyed," etc.). The analysis results are passed to the generative AI.

[0534] Step 3:

[0535] The server uses a generative AI (e.g., OpenAI GPT-4) to generate an appropriate product recommendation script based on the analyzed emotional state. The input is the analyzed emotional state and a prompt, and the output is the generated recommendation script. For example, a recommendation script is generated that says, "This aroma diffuser is a recommended product to help you relax."

[0536] Step 4:

[0537] The server displays the generated proposal script on the display of the smart glasses. The input is the generated proposal script, and the output is the proposal message displayed on the smart glasses. The store staff checks the display and makes a proposal to the customer.

[0538] Step 5:

[0539] The staff introduces specific products to the customer and guides them to try them out, following the suggestion message displayed on the smart glasses. The input is the suggestion message displayed on the smart glasses, and the output is the product suggestion to the customer.

[0540] Step 6:

[0541] The customer's reaction is recaptured and sent to the server for feedback. The input is image and voice data of the customer after Yori's service is provided, and the output is the recaptured emotional data. Based on the feedback, the proposal content is adjusted as needed. The server re-analyzes this new emotional data and continues to make optimal proposals.

[0542] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0543] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0544] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0545] [Second embodiment]

[0546] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0547] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0548] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0549] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0550] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0551] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0552] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0553] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0554] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0555] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0556] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0557] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0558] The present invention provides a platform where anyone can register and use an AI account that uses a generating AI. Users can converse with the AI ​​account on this platform and seamlessly use multiple services. Specific embodiments of the present invention are described below.

[0559] Program Generation and Processing Description

[0560] server:

[0561] The server provides a platform environment that users and registered users can access. The main roles of the server are as follows:

[0562] The server registers and manages a list of APIs provided by registrants and their usage methods in a database. Registrants access the server through their terminals and enter information about the APIs they provide. This information includes the API name, endpoint, usage method, required input parameters, and output items.

[0563] The server also receives and analyzes natural language requests from users, using natural language processing technology to understand the service desired by the user and identify the corresponding API.

[0564] Based on the identified API, the server uses a generation AI to generate the appropriate script to call the API as desired by the user. The generated script is then executed by the server to call the specified API.

[0565] Subscriber Device:

[0566] The registrant terminal is a device that connects to the server and registers API information. The registrant accesses the server from the terminal and enters the API name, endpoint, usage method, input parameters, and output items. The registrant also provides additional use cases and prompts to explain in detail how to use the API.

[0567] User device:

[0568] The user terminal is a device that allows users to access the platform and converse with the AI ​​account. The user selects the AI ​​account they want to use from their terminal and inputs the desired service details in chat format. For example, if a user inputs "I would like to reserve a hotel and restaurant for 2 nights and 3 days in Tokyo," the server receives this request.

[0569] The server analyzes the request, identifies the corresponding accommodation and restaurant reservation API, uses the generative AI to generate the necessary script, and calls the API to make the reservation. After the reservation is completed, the server returns the result to the user.

[0570] Examples:

[0571] Example 1: Travel booking

[0572] User:

[0573] The user is planning a three-day, two-night trip to Tokyo and would like to make hotel and restaurant reservations.

[0574] Operation steps:

[0575] 1. User terminal:

[0576] The user chats with the AI ​​account and says, "I'd like to book a hotel and restaurant in Tokyo for 2 nights and 3 days."

[0577] 2. Server:

[0578] The server analyzes the request and identifies the accommodation and restaurant reservation APIs.

[0579] 3. Server:

[0580] The generating AI generates a script for reservations and embeds the necessary parameters (date of stay, number of people, budget, etc.).

[0581] 4. Server:

[0582] Execute the generated script and make an API call.

[0583] 5. Server:

[0584] Receives accommodation and restaurant reservation results and returns the results to the user.

[0585] Example 2: Linking multiple AI accounts

[0586] User:

[0587] Users want their AI assistant to help them shop online and check the weather at the same time.

[0588] Operation steps:

[0589] 1. User terminal:

[0590] Users can chat with the AI ​​account and ask, "Tell me today's shopping recommendations and weather information."

[0591] 2. Server:

[0592] The server identifies the shopping and weather APIs and generates the necessary scripts.

[0593] 3. Server:

[0594] If necessary, add other AI accounts to the conversation and call their respective APIs.

[0595] 4. Server:

[0596] It retrieves shopping and weather information and returns it to the user.

[0597] This system allows users to seamlessly link different services together, greatly improving convenience and efficiency.

[0598] The processing flow will be explained below.

[0599] Step 1:

[0600] Subscriber Device:

[0601] The registrant accesses the server from a terminal and opens the API registration screen. The registrant enters the API name, endpoint, usage method, required input parameters, and output items. In addition, the registrant enters API usage examples and prompts, providing a detailed explanation of how to use the API. After confirming the entered information, the registrant submits the registration to the server.

[0602] Step 2:

[0603] server:

[0604] The server analyzes the received API registration details and stores them in a database. It then verifies the API usage and sample code to ensure there are no problems. Once verification is complete, it returns a registration completion message to the registrant's terminal.

[0605] Step 3:

[0606] User device:

[0607] Users access the platform from their device and select the AI ​​account they want to use. On the selected chat screen, they enter the desired service details in natural language. For example, they could enter, "I would like to book a hotel and restaurant in Tokyo for 2 nights and 3 days."

[0608] Step 4:

[0609] server:

[0610] The server analyzes messages received through the chat system, uses natural language processing to analyze the user's wishes and identify the necessary API, and prepares data to automatically generate API parameters based on the analysis results.

[0611] Step 5:

[0612] server:

[0613] The server calls the generation AI engine and generates a script for calling the API. It provides the "identified API name" and "required parameters" to the generation AI and instructs it to generate the script. It then performs syntax checks and logic verification on the generated script.

[0614] Step 6:

[0615] server:

[0616] Add error handling logic to complete the final script. Run the completed script in the execution environment. Call the specified API and obtain the results in real time. Check that the obtained results follow the correct format.

[0617] Step 7:

[0618] server:

[0619] The acquired data is formatted based on the user's wishes. The formatted results are sent back to the user's device. For example, a message such as "Your hotel reservation for 2 nights and 3 days in Tokyo has been completed. Details are as follows:..." is sent.

[0620] Step 8:

[0621] User device:

[0622] The user can check the results on their device and select the next action. For example, they may send another message saying, "Please make a restaurant reservation as well."

[0623] Step 9:

[0624] server:

[0625] The server receives a new request from the user, prepares a new API call while preserving the existing conversation context, uses the generative AI again to generate the necessary scripts and make additional API calls, and then integrates all the results and returns the combined information to the user.

[0626] Step 10:

[0627] User device:

[0628] The user can check the final result on the device and send feedback if necessary, such as a message saying, "I have confirmed the reservation details. There are no problems."

[0629] Step 11:

[0630] server:

[0631] The server receives the feedback and stores it in a database for service improvement purposes. It then sends a message to the user to end the conversation and terminates the session.

[0632] In this way, multiple services that use generative AI can be used in seamless collaboration.

[0633] Example 1

[0634] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0635] Conventional systems make it difficult for users to seamlessly use a variety of services. In particular, they require the manual integration of multiple application programming interfaces (APIs), a complex and time-consuming process. Furthermore, there was a lack of automated means for analyzing natural language requests, selecting the appropriate API, and generating the corresponding program. As a result, users were required to have high operational skills, and the system was difficult to use.

[0636] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0637] In this invention, the server includes means for receiving a service request from a user terminal in natural language, means for registering and managing a list of application programming interfaces provided by registrants and usage methods in a data storage device, means for analyzing the user's request using natural language processing and identifying an appropriate application programming interface, means for generating a program for calling the application programming interface using generative artificial intelligence, means for executing the generated program and calling the application programming interface, and means for returning the call result to the user. This enables users to seamlessly link and use a variety of services through requests in natural language.

[0638] "User terminal" means an electronic device through which a user inputs service requests and interacts with the system.

[0639] "Service requests" refers to requests or requests made by users regarding the content of services provided through the system.

[0640] "Natural language" refers to the words and forms of expression used by humans on a daily basis, usually expressed in text form.

[0641] A "server" is a computer system that processes requests from user terminals and registrant terminals and provides data and services.

[0642] "Registrant" means a person or entity that provides and manages one or more application programming interfaces (APIs) to a system.

[0643] An "application programming interface (API)" is a set of definitions or protocols that allow software components to communicate.

[0644] "Information management" is the process of using a database to register, update, and delete API information and maintain how it is used.

[0645] "Natural language processing" refers to the use of computers to analyze, understand, and generate natural language, for example through the use of text analysis and language models.

[0646] "Generative AI" is a general term for AI techniques for performing generative tasks (such as text generation and code generation), using, for example, generative models and neural networks.

[0647] "Program generation" is the process of automatically creating code or scripts to perform a specific task.

[0648] An "API call" refers to the act of sending a request from software to an API to use its functions or data.

[0649] A "data storage device" is a hardware or software system for the long-term storage and management of data.

[0650] "Conversational" refers to interactions between users and systems, such as text responses via a chatbot.

[0651] The present invention relates to a system that uses a user terminal, a server, and a registered user terminal to utilize a generative AI model and provide a platform that allows seamless use of a variety of services.

[0652] server

[0653] The server performs the following main functions:

[0654] 1. Receiving a request from a user device

[0655] The server receives service requests from the user terminal in natural language using a communication protocol such as HTTP or WebSocket.

[0656] 2. API information registration and management

[0657] The server uses a data storage device to register and manage a list of application programming interfaces (APIs) provided by registrants and their usage methods in a database. Typically, a relational database such as MySQL or PostgreSQL is used.

[0658] 3. Natural Language Processing

[0659] The server analyzes user requests using natural language processing models such as BERT and GPT-3 to understand the service desired by the user and identify the appropriate API.

[0660] 4. Script generation using generative AI models

[0661] Based on the analysis results, the server uses a generative AI model (e.g., GPT-4) to generate a script for calling the API, including the appropriate endpoints and parameters.

[0662] 5. Calling the API

[0663] The server executes the generated script and calls the specified API, and the response from the API is received by the server and processed as needed.

[0664] 6. Return of results

[0665] Finally, the server returns the results of the API call to the user's device, which is a specific response based on the service content requested by the user.

[0666] Registered device

[0667] The Registrant Terminal performs the following functions:

[0668] 1. Enter API information

[0669] The registrant enters the API name, endpoint, usage method, input parameters, and output items into the server through a terminal, which can be done via a web interface.

[0670] 2. Providing use cases and prompts

[0671] Registrants may provide additional use cases and prompts to provide detailed instructions on how to use the API, thereby helping consumers to more easily use the API.

[0672] User terminal

[0673] The user terminal is responsible for the following functions:

[0674] 1. Enter your service request

[0675] Users input the desired service details in chat format, and the details of the request are sent to the server as a request in natural language.

[0676] 2. Receiving and displaying results

[0677] The results returned from the server are displayed on the user's terminal, and the user performs the next operation based on these results.

[0678] Specific examples

[0679] Example 1: Travel booking

[0680] When a user enters "I would like to book a hotel and restaurant in Tokyo for 2 nights and 3 days," the server analyzes the request and identifies the appropriate accommodation and restaurant reservation API. Using generative AI, it generates the necessary script and calls the API to make the reservation. The results are sent back to the user's device.

[0681] Example 2: Linking multiple AI accounts

[0682] When a user types "Tell me today's shopping recommendations and weather information," the server identifies the APIs for shopping and weather information, generates the necessary scripts, and, if necessary, invites other AI accounts into the conversation and calls their respective APIs. The results are then sent back to the user's device.

[0683] Prompt Sentence Examples

[0684] 1. "Create a 3-day, 2-night trip to Tokyo."

[0685] 2. "I'd like to book a hotel in Tokyo for two nights starting February 20th."

[0686] 3. "Tell me what to buy today and what the weather is like."

[0687] 4. "Provide current weather information along with online shopping."

[0688] As described above, the embodiment of the present invention allows users to seamlessly use multiple services through requests in natural language.

[0689] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0690] Step 1:

[0691] (User request input)

[0692] The user inputs their service request in natural language using the user terminal. For example, they input a request such as "I would like to reserve a hotel and restaurant in Tokyo for 2 nights and 3 days." This request is sent from the user terminal to the server.

[0693] input:

[0694] A user's natural language request (e.g., "I would like to book a hotel and restaurant for a 2-night, 3-day stay in Tokyo.")

[0695] output:

[0696] A natural language request is sent to the server

[0697] Specific behavior:

[0698] The user enters the desired service details in the chat window and clicks the send button. The user's terminal generates an HTTP request to send the entered details to the server.

[0699] Step 2:

[0700] (receive and analyze the request by the server)

[0701] The server receives a natural language request from the user's device. Next, it uses natural language processing technology (e.g., BERT or GPT-3 models) to analyze the request and understand the service desired by the user. From the analysis results, it extracts key information (e.g., location, duration, type of service, etc.).

[0702] input:

[0703] A natural language request received by the server

[0704] output:

[0705] Extracted key information (e.g., "Tokyo," "2 nights and 3 days," "hotel," "restaurant")

[0706] Specific behavior:

[0707] The server receives an HTTP request and feeds the text portion of the request into a natural language processing model, which analyzes the text and extracts important keywords and parameters.

[0708] Step 3:

[0709] (Server-specific API)

[0710] Based on the analysis, the server identifies the appropriate API, for example, "Hotel Reservation API" and "Restaurant Reservation API," which involves querying a database to select the appropriate API from a list of applicable APIs.

[0711] input:

[0712] Parsed key information

[0713] output:

[0714] List of identified APIs (e.g., "Hotel Booking API," "Restaurant Booking API")

[0715] Specific behavior:

[0716] The server queries the database to obtain API information that matches the analysis results, and then lists the endpoints and parameters of the obtained APIs.

[0717] Step 4:

[0718] (Script generation by generative AI)

[0719] The server uses a generative AI model (e.g., GPT-4) to generate a script to call the identified API. This script embeds the necessary parameters (e.g., location, period, number of people, etc.) and makes the API call.

[0720] input:

[0721] List of identified APIs and analyzed key information

[0722] output:

[0723] Generated API call script

[0724] Specific behavior:

[0725] The server inputs the API list and key information to the generation AI and requests it to generate a script. The generation AI generates the script and returns it to the server.

[0726] Step 5:

[0727] (API call by server)

[0728] The server executes the generated script and calls the specified API, the response from the API is received by the server and further processed as needed.

[0729] input:

[0730] Generated API call script

[0731] output:

[0732] Response data from the API (e.g., reservation confirmation information)

[0733] Specific behavior:

[0734] The server runs the generated script, calls the API via an HTTP request, receives the API response, and formats the data if necessary.

[0735] Step 6:

[0736] (Return of results)

[0737] The server returns the results of the API call to the user, who can then view the results on their device.

[0738] input:

[0739] Response data from the API

[0740] output:

[0741] Results displayed on the user's device

[0742] Specific behavior:

[0743] The server converts the received response data into a convenient format and sends it to the user's terminal as an HTTP response. The user can then check the results, such as a reservation completion notification, on their terminal.

[0744] (Application example 1)

[0745] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0746] Conventional online shopping systems have had difficulty seamlessly providing the services users need, such as detailed product information, stock availability, and purchasing procedures. In particular, centralizing multiple services and automatically linking them according to user requests has been a challenge.

[0747] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0748] In this invention, the server includes means for receiving service requests in natural language from a user terminal, means for registering and managing a list of APIs provided by registrants and usage methods in a database, means for analyzing the user's requests using natural language processing and identifying an appropriate API, means for generating a script for calling the API using a generation AI, means for executing the generated script and calling the API, means for returning the call results to the user, and means for the user to receive various services such as detailed product information, stock confirmation, and purchase procedures in the virtual store, thereby enabling the user to seamlessly complete a series of shopping procedures in the virtual store.

[0749] A "user terminal" is a device for inputting service requests in natural language and communicating with a server to receive information and results.

[0750] A "service request" is a request that specifies the specific service content that the user wants to receive.

[0751] "Natural language" is the language used by humans in everyday life, and includes informal words and sentences.

[0752] "API" stands for Application Programming Interface, an interface for exchanging functions between software programs.

[0753] A "database" is a system for systematically storing large amounts of data and efficiently searching, updating, and managing them.

[0754] "Natural language processing" is a technology that allows computers to understand, analyze, and generate human language.

[0755] "Generative AI" is an artificial intelligence technology that automatically generates new scripts and content based on input data.

[0756] A "script" is a program code that contains a set of instructions for performing a particular task or process.

[0757] A "virtual store" is a store operated on the Internet, a virtual space that offers products and services.

[0758] "Checking inventory" means checking how much of a particular product is currently available for purchase.

[0759] The "purchase procedure" refers to the process of all the steps required to purchase the selected product.

[0760] This invention is a system that provides AI accounts using generation AI in virtual stores, allowing users to provide detailed product information, check inventory, and complete purchase procedures seamlessly.

[0761] System Configuration

[0762] The system consists of the following main components:

[0763] 1. Server:

[0764] Technologies used: Python, Flask, TensorFlow

[0765] function:

[0766] A service request is received in natural language from a user terminal.

[0767] A database manages a list of APIs provided by registrants and their usage instructions.

[0768] It uses natural language processing to analyze user preferences and identify the appropriate API.

[0769] Use generative AI to generate scripts for API calls.

[0770] Run the generated script to call the API.

[0771] The call result is returned to the user.

[0772] 2. User terminal:

[0773] Technologies used: Kotlin or Swift (for smartphones), ARCore or ARKit (for smart glasses)

[0774] function:

[0775] The user inputs the service request in natural language and sends it to the server.

[0776] The information and results received from the server are displayed and read aloud.

[0777] Operation overview

[0778] User operation

[0779] Customers access the virtual store using an app on their smartphone or smart glasses, and then make requests by voice or text, such as "Please let me know if the Nike Air Max 2023 is in stock."

[0780] Server-side processing

[0781] 1. The server receives a natural language request from the user.

[0782] 2. Use natural language processing technology to analyze the request and identify the appropriate API.

[0783] 3. Use generative AI (using TensorFlow) to generate scripts for API calls corresponding to the request.

[0784] 4. Execute the generated script and call the product information API to retrieve the required data.

[0785] 5. The acquired information is sent back to the user's terminal.

[0786] Server example

[0787] Example prompt sentence:

[0788] A user wants more information about the following product: "Nike Air Max 2023". Get the required information from your API.

[0789] Specific examples of user terminals

[0790] When a user voice-inputs, "Please tell me the stock status of the Nike Air Max 2023," the request is sent to the server, which analyzes and processes it, sends the results back to the device, and displays a message such as "In stock: 10 left" on the smartphone screen.

[0791] Overall system flow

[0792] This system allows users to seamlessly obtain product information, check inventory, and complete purchase procedures within the virtual store, greatly improving the convenience of shopping. Users input their service requests in a chat format, and the server automatically calls the optimal API to provide the necessary information, making purchasing activities more efficient.

[0793] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0794] Step 1:

[0795] A user accesses a virtual store through a smartphone or smart glasses app and inputs their service request in natural language, for example, "Please tell me the stock status of the Nike Air Max 2023." This input is sent by the device to the server.

[0796] Step 2:

[0797] The server receives the input natural language request and analyzes the request using natural language processing technology. Specifically, it analyzes the input text and extracts intent such as "product information" or "inventory check." The input is a natural language request, and the output is the analyzed intent data.

[0798] Step 3:

[0799] The server identifies an appropriate API from a database based on the analyzed intent data. The database contains a list of multiple APIs and their usage methods. The analyzed intent data is the input, and the identified API is the output.

[0800] Step 4:

[0801] The server uses a generative AI model (using TensorFlow) to generate a script for calling the API. The user's request and the identified API information are input into the generative AI model, and the API call script is output. For example, a "script to call the Nike Air Max 2023 stock confirmation API" is generated.

[0802] Step 5:

[0803] The server executes the generated script and calls the specified API to obtain the required data. As a result of the API call, product information and stock status are obtained. The input is the API call script, and the output is the obtained product data.

[0804] Step 6:

[0805] The server processes the acquired data as necessary and converts it into an appropriate format before returning it to the user. For example, it converts inventory information into an easy-to-understand format such as "In stock: 10 remaining." The input is the acquired product data, and the output is the processed information data.

[0806] Step 7:

[0807] The terminal provides the information received from the server to the user by displaying it on the screen or outputting audio. For example, the display of a smartphone or smart glasses might show "In stock: 10 remaining." The input is the information data returned from the server, and the output is display data or audio data.

[0808] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0809] This invention provides a platform where anyone can use and register an AI account using generative AI. In particular, this invention combines an emotion engine that recognizes the user's emotions to provide more personalized services.

[0810] Program Generation and Processing Description

[0811] server:

[0812] The server provides a platform environment that users and registered users can access. The main roles of the server are as follows:

[0813] The server registers and manages a list of APIs provided by registrants and their usage methods in a database. Registrants access the server via their terminals and enter information about the APIs they provide. This includes the API name, endpoint, usage method, required input parameters, and output items.

[0814] Next, the server receives a request in natural language from the user. It uses natural language processing technology to understand the service desired by the user and identifies the corresponding API. Based on the identified API, it uses generative AI to generate an appropriate script. This script is used to call the API according to the user's request.

[0815] The server also has an emotion engine that analyzes the user's chat content to identify their emotional state. This emotional information is used to adjust the script content generated by the AI. For example, if the user is feeling stressed, the script output can be changed to a soothing message.

[0816] Subscriber Device:

[0817] The registrant terminal is a device that connects to the server and registers API information. The registrant accesses the server from the terminal and enters the API name, endpoint, usage method, input parameters, and output items. It also provides examples and prompts for users to use the API in practice, and provides detailed instructions on how to use the API.

[0818] User device:

[0819] The user terminal is a device that allows users to access the platform and converse with the AI ​​account. The user selects the AI ​​account they want to use from their terminal and inputs the desired service details in chat format. For example, if a user inputs "I would like to reserve a hotel and restaurant for 2 nights and 3 days in Tokyo," the server receives this request.

[0820] The server analyzes the request, identifies the corresponding accommodation and restaurant reservation APIs, uses generative AI to generate the necessary scripts, and calls the APIs to make the reservation. Once the reservation is complete, the server returns the results to the user.

[0821] An emotion engine is also integrated into this process, analyzing the user's emotional state in real time. For example, if the system detects that the user is tired, it can send an additional message along with the booking result, such as "We recommend a relaxing hotel."

[0822] Examples:

[0823] Example 1: Travel booking

[0824] User:

[0825] The user is planning a three-day, two-night trip to Tokyo and wants to make hotel and restaurant reservations. During the chat, the emotion engine detects that the user is feeling stressed.

[0826] Operation steps:

[0827] 1. User terminal:

[0828] The user chats with the AI ​​account and says, "I'd like to book a hotel and restaurant in Tokyo for 2 nights and 3 days."

[0829] 2. Server:

[0830] The server analyzes the request and identifies the accommodation and restaurant reservation APIs.

[0831] 3. Server:

[0832] The generating AI generates a script for reservations and embeds the necessary parameters (date of stay, number of people, budget, etc.).

[0833] The emotion engine detects the user's stress level and adjusts the content of the generated script.

[0834] 4. Server:

[0835] Execute the generated script and make an API call.

[0836] 5. Server:

[0837] It receives the results of accommodation and restaurant reservations and returns them to the user with a message such as, "We recommend a relaxing hotel."

[0838] Example 2: Linking multiple AI accounts

[0839] User:

[0840] Users want to shop online and check the weather at the same time. The emotion engine recognizes that users enjoy themselves.

[0841] Operation steps:

[0842] 1. User terminal:

[0843] Users tell the AI ​​account, "Tell me today's recommended shopping and weather information."

[0844] 2. Server:

[0845] The server identifies the shopping and weather APIs and generates the necessary scripts.

[0846] The emotion engine recognizes the user's enjoyment and adapts the generated script accordingly.

[0847] 3. Server:

[0848] If necessary, add other AI accounts to the conversation and call their respective APIs.

[0849] 4. Server:

[0850] It obtains shopping and weather information and returns it to the user along with a message such as, "Today's weather is sunny. Perfect day for shopping!"

[0851] In this way, by combining generative AI and an emotion engine, the present invention can provide optimal services according to the user's emotional state, further improving user satisfaction and convenience.

[0852] The processing flow will be explained below.

[0853] This invention provides a platform where anyone can use and register an AI account using generative AI. In particular, this invention combines an emotion engine that recognizes the user's emotions to provide more personalized services.

[0854] Program Generation and Processing Description

[0855] server:

[0856] Step 1:

[0857] server:

[0858] A list of APIs provided by registrants and their usage methods are registered and managed in a database. Registrants access the server from their terminal and enter information about the APIs they provide. This includes the API name, endpoint, usage method, required input parameters, and output items. This allows the API corresponding to the service desired by the user to be saved in the database.

[0859] Step 2:

[0860] server:

[0861] It receives requests in natural language from users. It uses natural language processing technology to understand the service desired by the user and identify the appropriate API. For example, if a request such as "I would like to book a hotel and restaurant in Tokyo for 2 nights and 3 days" is received via chat, it will be analyzed.

[0862] Step 3:

[0863] server:

[0864] The emotional engine analyzes the user's emotional state from the chat content, thereby identifying the stress or joy the user is feeling.

[0865] Step 4:

[0866] server:

[0867] It uses generative AI to generate scripts to call the identified APIs, and the generated scripts are tailored to the user's emotional state—for example, if they're feeling stressed, the script will be tailored to be relaxing.

[0868] Step 5:

[0869] server:

[0870] Execute the generated script and call the API. The specified API returns data in real time, and the server receives that data.

[0871] Step 6:

[0872] server:

[0873] It then formats the data it receives based on the user's preferences and generates additional messages as needed, such as "We recommend a hotel where you can relax."

[0874] Step 7:

[0875] server:

[0876] The formatted result is sent back to the user's terminal, where the user can receive the result and select the next action.

[0877] Examples:

[0878] Example 1: Travel booking

[0879] User:

[0880] The user is planning a three-day, two-night trip to Tokyo and wants to make hotel and restaurant reservations. During the chat, the emotion engine detects that the user is feeling stressed.

[0881] Operation steps:

[0882] Step 1:

[0883] User device:

[0884] The user chats with the AI ​​account and says, "I would like to book a hotel and restaurant in Tokyo for 2 nights and 3 days."

[0885] Step 2:

[0886] server:

[0887] The server analyzes the request and identifies the accommodation and restaurant reservation APIs.

[0888] Step 3:

[0889] server:

[0890] The generating AI generates a script for reservations and embeds the necessary parameters (date of stay, number of people, budget, etc.).

[0891] Step 4:

[0892] server:

[0893] The emotion engine detects the user's stress level and adjusts the content of the generated script.

[0894] Step 5:

[0895] server:

[0896] Execute the generated script and make an API call.

[0897] Step 6:

[0898] server:

[0899] It receives hotel and restaurant booking results and formats the results with a message such as "We recommend a relaxing hotel."

[0900] Step 7:

[0901] server:

[0902] The formatted result is sent back to the user's terminal.

[0903] Example 2: Linking multiple AI accounts

[0904] User:

[0905] Users want to shop online and check the weather at the same time. The emotion engine recognizes that users enjoy themselves.

[0906] Operation steps:

[0907] Step 1:

[0908] User device:

[0909] Users tell the AI ​​account, "Tell me today's recommended shopping and weather information."

[0910] Step 2:

[0911] server:

[0912] The server identifies the shopping and weather APIs and generates the necessary scripts.

[0913] Step 3:

[0914] server:

[0915] The emotion engine recognizes the user's enjoyment and adapts the generated script accordingly.

[0916] Step 4:

[0917] server:

[0918] If necessary, add other AI accounts to the conversation and call their respective APIs.

[0919] Step 5:

[0920] server:

[0921] Get shopping and weather information and format the results with a message like "Today's weather is sunny. Great day for shopping!"

[0922] Step 6:

[0923] server:

[0924] The formatted result is sent back to the user's terminal.

[0925] In this way, by combining generative AI and an emotion engine, the present invention can provide optimal services according to the user's emotional state, further improving user satisfaction and convenience.

[0926] Example 2

[0927] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0928] Modern information systems require users to receive a variety of services quickly and in a personalized manner. However, current systems often lack the functionality to analyze the user's emotional state and reflect the results in the content of the service provided. As a result, services provided without considering the user's emotional state may reduce user satisfaction. Furthermore, when providing complex services by linking multiple APIs, there is a lack of an efficient process for identifying APIs and generating scripts. This poses the challenge of requiring a great deal of effort and time to meet user requests.

[0929] The specification process by the specification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes: means for receiving a service request from a user terminal in natural language; means for registering and managing a list of APIs provided by registrants and usage methods in a database; means for analyzing the user's request and identifying an appropriate API using natural language processing; means for generating a script for calling the API using a generative AI model; means for executing the generated script to call the API; means for returning the call result to the user; means including an emotion engine for analyzing the user's emotional state; and means for adjusting the content of the generated script depending on the user's emotional state. This makes it possible to quickly provide personalized services according to the user's emotional state and efficiently link multiple APIs.

[0930] "User terminal" means a device through which a user accesses the system and inputs service requests.

[0931] A "registrant" is a person who registers information about the API that they provide in the system.

[0932] "API" stands for Application Programming Interface, an interface for providing specific functions and data to the outside world.

[0933] A "database" is a system that stores information in an organized manner and allows it to be retrieved when needed.

[0934] "Natural language processing" is a technology that allows computers to understand and process human language.

[0935] A "generative AI model" is an algorithm that uses artificial intelligence to generate scripts for specific tasks or purposes.

[0936] A "script" is program code for automatically executing a series of instructions.

[0937] An "emotion engine" is a technology that analyzes a user's emotional state from their text or input.

[0938] "Call result" refers to the response or data obtained after executing an API.

[0939] "Natural language" refers to the language used by humans on a daily basis, translated into a form that the system can understand.

[0940] "Fusion of multiple APIs" refers to the use of multiple APIs simultaneously or sequentially to provide one or more services.

[0941] This invention is a highly personalized service provision system that utilizes a generative AI model and an emotion engine. This system has the function of calling the optimal API based on the user's wishes and emotional state and returning the results to the user.

[0942] System Overview

[0943] The system includes the following major hardware and software components:

[0944] Server: Responsible for managing the API, parsing requests, generating scripts, running the emotion engine, and returning results.

[0945] User terminal: A device (PC, smartphone, etc.) that allows a user to access the system and input service requests.

[0946] Subscriber terminal: A device used to add API information to the system.

[0947] Database (e.g. MySQL, PostgreSQL): Stores and manages registered API information.

[0948] Natural language processing engines (e.g. Google NLP API, spaCy): used to analyze user preferences.

[0949] A generative AI model (e.g., OpenAI GPT-3) used to generate scripts for API calls.

[0950] Emotion engine (e.g., Microsoft Azure Emotion API): Used to analyze the user's emotional state.

[0951] Detailed System Operation

[0952] 1. Receiving Service Requests:

[0953] The user inputs their desired service request in natural language from their device and sends it to the server. For example, a user might input, "I would like to reserve a hotel and restaurant for 2 nights and 3 days in Tokyo."

[0954] 2. Parse the request:

[0955] The server uses a natural language processing engine to analyze the request and understand the user's wishes, extracting the keywords "hotel reservation" and "restaurant reservation."

[0956] 3. Identify APIs and generate scripts:

[0957] The server identifies the APIs corresponding to "hotel reservation" and "restaurant reservation" from the database. Then, it generates a script using the generative AI model. For example, it sends a prompt such as "Please generate a script using the APIs for hotel reservation and restaurant reservation."

[0958] 4. Applying the Emotion Engine:

[0959] The server analyzes the user's chat content using an emotion engine to identify the user's emotional state. For example, if the user is feeling stressed, the emotion engine conveys that information. The server then adjusts the content of the generated script to include a message such as, "We recommend a relaxing hotel."

[0960] 5. API call and return results:

[0961] The server runs the generated script, calls the appropriate API, receives the response from the API, and returns the results to the user, including the specific reservation information and an optional message (e.g., "We recommend a relaxing hotel").

[0962] Specific examples

[0963] Book a trip

[0964] User: Planning a trip, type in chat, "I'd like to book a hotel and restaurant in Tokyo for 2 nights and 3 days."

[0965] Server: Receives the request and analyzes it using a natural language processing engine. It generates an appropriate script using a generative AI model and analyzes the user's emotions using an emotion engine. If stress is detected, it adds a soothing message to the script.

[0966] Execution and return: Call the API to execute the reservation and return the reservation results to the user terminal.

[0967] Multiple API integrations

[0968] User: Wants to shop online and check the weather at the same time, so types in "Tell me today's recommended shopping and weather information."

[0969] Server: Identifies the shopping and weather APIs and generates the necessary scripts. If the emotion engine determines that the user is having fun, it includes a message that matches this excitement.

[0970] Execution and return: Calls multiple APIs and returns the obtained shopping information and weather information to the user along with a message such as "Today's weather is sunny. It's a great day for shopping!"

[0971] In this way, by combining a generative AI model and an emotion engine, the present invention can provide optimal services according to the user's emotional state, thereby improving user satisfaction and realizing efficient and personalized service provision.

[0972] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0973] Step 1: Enter and submit your service request on the user's device

[0974] Specific operation: The user uses the user terminal to input the desired service details in chat format. For example, the user might input "I would like to reserve a hotel and restaurant in Tokyo for 2 nights and 3 days."

[0975] Input: A message of service request written in natural language.

[0976] Output: The request message sent to the server.

[0977] Step 2: Server receives and analyzes the request

[0978] Specific operation: The server receives the request sent from the user terminal and analyzes the request using a natural language processing engine (e.g., Google NLP API, spaCy). Through the analysis, the keywords "hotel reservation" and "restaurant reservation" are extracted.

[0979] Input: A natural language request from the user.

[0980] Output: Parsed preference (e.g. "Hotel reservation", "Restaurant reservation").

[0981] Step 3: Identify the corresponding API

[0982] Specific operation: The server refers to the database and identifies the API corresponding to the analyzed request. For example, it retrieves the "Hotel Reservation" API and the "Restaurant Reservation" API.

[0983] Input: Parsed desired content.

[0984] Output: Identified API (e.g. "Hotel Booking" API, "Restaurant Booking" API).

[0985] Step 4: Generating scripts using generative AI models

[0986] Specific operation: The server uses a generative AI model (e.g., OpenAI GPT-3) to generate a script for API calls based on the user's request. The prompt text is "Please generate a script using the hotel reservation and restaurant reservation APIs."

[0987] Input: The API identified and what you want.

[0988] Data processing: Send prompt text to the generative AI model to generate a script for calling the API.

[0989] Output: The generated script.

[0990] Step 5: Emotional state analysis by the emotion engine

[0991] Specific operation: The server analyzes the user's chat content using an emotion engine (e.g., Microsoft Azure Emotion API) to identify the user's emotional state. For example, it detects whether the user is feeling stressed.

[0992] Input: User chat content.

[0993] Data Computation: Analyze emotional states using an emotion engine.

[0994] Output: Detected emotional state (e.g., stress).

[0995] Step 6: Adjust the script content

[0996] Specific behavior: The server adjusts the content of the generated script based on the detected emotional state. For example, if a user is feeling stressed, it might add a message such as "We recommend a relaxing hotel."

[0997] Input: Generated script and emotional state.

[0998] Data processing: The message part of the script is changed depending on the emotional state.

[0999] Output: The adjusted script.

[1000] Step 7: Making API calls

[1001] Specific operation: The server executes the tailored script, calls the specified API, and receives the response from the API.

[1002] Input: The adjusted script.

[1003] Output: The response data from the API.

[1004] Step 8: Returning the results

[1005] Specific operation: The server returns the response data from the API to the user's device and displays the result, for example, information that the reservation was successful and a message such as "We recommend a relaxing hotel."

[1006] Input: Response data from the API.

[1007] Data processing: Integrate response data and emotional responses.

[1008] Output: Results returned to the user's terminal.

[1009] (Application example 2)

[1010] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1011] Conventional automated service provision systems have difficulty in providing personalized responses that take into account the user's emotions, making it difficult to increase user satisfaction. In particular, when providing services in physical stores, flexible and appropriate responses that take into account the customer's psychological state are required.

[1012] The specification processing by the specification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for receiving a service request from a user terminal in natural language, means for registering and managing a list of APIs provided by registrants and their usage methods in a database, means for analyzing the user's request using natural language processing and identifying an appropriate API, means for generating a script for calling the API using a generation AI, means for executing the generated script and calling the API, means for returning the call result to the user, means for analyzing the user's emotional state, and means for adjusting the content of the generated script based on the emotional state. This makes it possible to provide personalized services according to the user's emotional state.

[1013] A "user terminal" is a device that allows a user to input service requests.

[1014] "Service Desired" refers to the specific service content that the user wishes to use.

[1015] "Natural language" refers to the language that humans use on a daily basis, not a specialized language such as a program language.

[1016] "API" stands for Application Programming Interface, an interface for exchanging functions and data between software programs.

[1017] A "database" is a system for efficiently storing and managing data.

[1018] "Natural language processing" is a technology that allows computers to understand and analyze natural language.

[1019] "Generative AI" is an artificial intelligence technology that generates appropriate responses and results based on input data.

[1020] A "script" is a program code that contains a series of instructions or commands.

[1021] "Emotional state" refers to the user's psychological state or mood, including emotions such as stress and joy.

[1022] "Emotion analysis" is a technology that determines a user's emotional state based on facial expressions, tone of voice, etc.

[1023] An "API call" is an operation that executes a specified API to obtain data or functionality.

[1024] This invention is a system that provides personalized customer service in brick-and-mortar stores, and is realized using smart glasses and emotion analysis sensor devices. The main components of the system are the user terminal, emotion analysis sensor, generative AI, natural language processing, and a server that manages and executes each API.

[1025] Major hardware and software used

[1026] Hardware: smart glasses (e.g., Google Glass), sensor devices for emotion analysis (e.g., Microsoft Azure Kinect)

[1027] software:

[1028] Natural language processing libraries (e.g. spaCy)

[1029] Sentiment analysis engine (e.g. Microsoft Azure Emotion API)

[1030] Generative AI systems (e.g., OpenAI GPT-4)

[1031] API management and databases (e.g. AWS, Firebase)

[1032] System Operation Overview

[1033] 1. Obtaining customer sentiment data

[1034] The smart glasses' camera and emotion-analysis sensors capture the customer's facial expressions and tone of voice in real time, allowing them to determine their emotional state on the spot. For example, if a customer is feeling stressed, that information will be detected.

[1035] 2. Emotional Data Analysis

[1036] The acquired emotional data is sent to the cloud and analyzed by an emotion analysis engine. For example, the Microsoft Azure Emotion API is used to identify emotional states such as "high stress level" or "enjoyed" based on facial expressions and voice characteristics.

[1037] 3. Proposal generation using generative AI

[1038] Based on the analysis results, the generative AI system will make optimal product suggestions. For example, if a customer is feeling stressed, the generative AI will generate a script suggesting products with a relaxing effect and display it on the smart glasses' display. The generative AI uses OpenAI GPT-4 to generate natural and appropriate suggestion messages.

[1039] 4. Implementing product proposals

[1040] The staff member checks the information displayed on the smart glasses and suggests individual products and services to the customer. The staff member then introduces specific products and encourages the customer to try them according to the generated suggestion message. For example, the staff member might say, "This aroma diffuser is a recommended product to help you relax."

[1041] 5. Feedback and Adjustments

[1042] After making a proposal, the system reanalyzes the customer's reactions (facial expressions and tone of voice) and adjusts the proposal if necessary, allowing the system to continue providing optimal service based on customer feedback.

[1043] Specific prompt examples

[1044] For example, in a brick-and-mortar scenario:

[1045] Prompt Sentence Examples

[1046] "The customer's facial expression is stressed and their voice tone is low. They are currently in the relaxation section of the store. The service they offer is product suggestions. The priority keywords are relaxation, healing, and stress relief."

[1047] In this way, the system of the present invention can provide personalized services in physical stores according to the emotional state of the user, thereby achieving high customer satisfaction.

[1048] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1049] Step 1:

[1050] The smart glasses' camera and emotion analysis sensor are used to capture the customer's facial expressions and tone of voice in real time, and the data is acquired. The input is the customer's image and voice data, and the output is the captured emotion data. The acquired data is sent to the server.

[1051] Step 2:

[1052] The server sends the acquired emotional data to the cloud and analyzes it using an emotion analysis engine (e.g., Microsoft Azure Emotion API). The input is the captured emotional data, and the output is the analyzed emotional state (e.g., "stressed," "enjoyed," etc.). The analysis results are passed to the generative AI.

[1053] Step 3:

[1054] The server uses a generative AI (e.g., OpenAI GPT-4) to generate an appropriate product recommendation script based on the analyzed emotional state. The input is the analyzed emotional state and a prompt, and the output is the generated recommendation script. For example, a recommendation script is generated that says, "This aroma diffuser is a recommended product to help you relax."

[1055] Step 4:

[1056] The server displays the generated proposal script on the display of the smart glasses. The input is the generated proposal script, and the output is the proposal message displayed on the smart glasses. The store staff checks the display and makes a proposal to the customer.

[1057] Step 5:

[1058] The staff introduces specific products to the customer and guides them to try them out, following the suggestion message displayed on the smart glasses. The input is the suggestion message displayed on the smart glasses, and the output is the product suggestion to the customer.

[1059] Step 6:

[1060] The customer's reaction is recaptured and sent to the server for feedback. The input is image and voice data of the customer after Yori's service is provided, and the output is the recaptured emotional data. Based on the feedback, the proposal content is adjusted as needed. The server re-analyzes this new emotional data and continues to make optimal proposals.

[1061] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1062] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1063] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1064] [Third embodiment]

[1065] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1066] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[1067] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1068] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1069] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1070] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1071] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1072] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1073] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1074] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1075] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1076] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1077] The present invention provides a platform where anyone can register and use an AI account that uses a generating AI. Users can converse with the AI ​​account on this platform and seamlessly use multiple services. Specific embodiments of the present invention are described below.

[1078] Program Generation and Processing Description

[1079] server:

[1080] The server provides a platform environment that users and registered users can access. The main roles of the server are as follows:

[1081] The server registers and manages a list of APIs provided by registrants and their usage methods in a database. Registrants access the server through their terminals and enter information about the APIs they provide. This information includes the API name, endpoint, usage method, required input parameters, and output items.

[1082] The server also receives and analyzes natural language requests from users, using natural language processing technology to understand the service desired by the user and identify the corresponding API.

[1083] Based on the identified API, the server uses a generation AI to generate the appropriate script to call the API as desired by the user. The generated script is then executed by the server to call the specified API.

[1084] Subscriber Device:

[1085] The registrant terminal is a device that connects to the server and registers API information. The registrant accesses the server from the terminal and enters the API name, endpoint, usage method, input parameters, and output items. The registrant also provides additional use cases and prompts to explain in detail how to use the API.

[1086] User device:

[1087] The user terminal is a device that allows users to access the platform and converse with the AI ​​account. The user selects the AI ​​account they want to use from their terminal and inputs the desired service details in chat format. For example, if a user inputs "I would like to reserve a hotel and restaurant for 2 nights and 3 days in Tokyo," the server receives this request.

[1088] The server analyzes the request, identifies the corresponding accommodation and restaurant reservation API, uses the generative AI to generate the necessary script, and calls the API to make the reservation. After the reservation is completed, the server returns the result to the user.

[1089] Examples:

[1090] Example 1: Travel booking

[1091] User:

[1092] The user is planning a three-day, two-night trip to Tokyo and would like to make hotel and restaurant reservations.

[1093] Operation steps:

[1094] 1. User terminal:

[1095] The user chats with the AI ​​account and says, "I'd like to book a hotel and restaurant in Tokyo for 2 nights and 3 days."

[1096] 2. Server:

[1097] The server analyzes the request and identifies the accommodation and restaurant reservation APIs.

[1098] 3. Server:

[1099] The generating AI generates a script for reservations and embeds the necessary parameters (date of stay, number of people, budget, etc.).

[1100] 4. Server:

[1101] Execute the generated script and make an API call.

[1102] 5. Server:

[1103] Receives accommodation and restaurant reservation results and returns the results to the user.

[1104] Example 2: Linking multiple AI accounts

[1105] User:

[1106] Users want their AI assistant to help them shop online and check the weather at the same time.

[1107] Operation steps:

[1108] 1. User terminal:

[1109] Users can chat with the AI ​​account and ask, "Tell me today's shopping recommendations and weather information."

[1110] 2. Server:

[1111] The server identifies the shopping and weather APIs and generates the necessary scripts.

[1112] 3. Server:

[1113] If necessary, add other AI accounts to the conversation and call their respective APIs.

[1114] 4. Server:

[1115] It retrieves shopping and weather information and returns it to the user.

[1116] This system allows users to seamlessly link different services together, greatly improving convenience and efficiency.

[1117] The processing flow will be explained below.

[1118] Step 1:

[1119] Subscriber Device:

[1120] The registrant accesses the server from a terminal and opens the API registration screen. The registrant enters the API name, endpoint, usage method, required input parameters, and output items. In addition, the registrant enters API usage examples and prompts, providing a detailed explanation of how to use the API. After confirming the entered information, the registrant submits the registration to the server.

[1121] Step 2:

[1122] server:

[1123] The server analyzes the received API registration details and stores them in a database. It then verifies the API usage and sample code to ensure there are no problems. Once verification is complete, it returns a registration completion message to the registrant's terminal.

[1124] Step 3:

[1125] User device:

[1126] Users access the platform from their device and select the AI ​​account they want to use. On the selected chat screen, they enter the desired service details in natural language. For example, they could enter, "I would like to book a hotel and restaurant in Tokyo for 2 nights and 3 days."

[1127] Step 4:

[1128] server:

[1129] The server analyzes messages received through the chat system, uses natural language processing to analyze the user's wishes and identify the necessary API, and prepares data to automatically generate API parameters based on the analysis results.

[1130] Step 5:

[1131] server:

[1132] The server calls the generation AI engine and generates a script for calling the API. It provides the "identified API name" and "required parameters" to the generation AI and instructs it to generate the script. It then performs syntax checks and logic verification on the generated script.

[1133] Step 6:

[1134] server:

[1135] Add error handling logic to complete the final script. Run the completed script in the execution environment. Call the specified API and obtain the results in real time. Check that the obtained results follow the correct format.

[1136] Step 7:

[1137] server:

[1138] The acquired data is formatted based on the user's wishes. The formatted results are sent back to the user's device. For example, a message such as "Your hotel reservation for 2 nights and 3 days in Tokyo has been completed. Details are as follows:..." is sent.

[1139] Step 8:

[1140] User device:

[1141] The user can check the results on their device and select the next action. For example, they may send another message saying, "Please make a restaurant reservation as well."

[1142] Step 9:

[1143] server:

[1144] The server receives a new request from the user, prepares a new API call while preserving the existing conversation context, uses the generative AI again to generate the necessary scripts and make additional API calls, and then integrates all the results and returns the combined information to the user.

[1145] Step 10:

[1146] User device:

[1147] The user can check the final result on the device and send feedback if necessary, such as a message saying, "I have confirmed the reservation details. There are no problems."

[1148] Step 11:

[1149] server:

[1150] The server receives the feedback and stores it in a database for service improvement purposes. It then sends a message to the user to end the conversation and terminates the session.

[1151] In this way, multiple services that use generative AI can be used in seamless collaboration.

[1152] Example 1

[1153] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1154] Conventional systems make it difficult for users to seamlessly use a variety of services. In particular, they require the manual integration of multiple application programming interfaces (APIs), a complex and time-consuming process. Furthermore, there was a lack of automated means for analyzing natural language requests, selecting the appropriate API, and generating the corresponding program. As a result, users were required to have high operational skills, and the system was difficult to use.

[1155] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1156] In this invention, the server includes means for receiving a service request from a user terminal in natural language, means for registering and managing a list of application programming interfaces provided by registrants and usage methods in a data storage device, means for analyzing the user's request using natural language processing and identifying an appropriate application programming interface, means for generating a program for calling the application programming interface using generative artificial intelligence, means for executing the generated program and calling the application programming interface, and means for returning the call result to the user. This enables users to seamlessly link and use a variety of services through requests in natural language.

[1157] "User terminal" means an electronic device through which a user inputs service requests and interacts with the system.

[1158] "Service requests" refers to requests or requests made by users regarding the content of services provided through the system.

[1159] "Natural language" refers to the words and forms of expression used by humans on a daily basis, usually expressed in text form.

[1160] A "server" is a computer system that processes requests from user terminals and registrant terminals and provides data and services.

[1161] "Registrant" means a person or entity that provides and manages one or more application programming interfaces (APIs) to a system.

[1162] An "application programming interface (API)" is a set of definitions or protocols that allow software components to communicate.

[1163] "Information management" is the process of using a database to register, update, and delete API information and maintain how it is used.

[1164] "Natural language processing" refers to the use of computers to analyze, understand, and generate natural language, for example through the use of text analysis and language models.

[1165] "Generative AI" is a general term for AI techniques for performing generative tasks (such as text generation and code generation), using, for example, generative models and neural networks.

[1166] "Program generation" is the process of automatically creating code or scripts to perform a specific task.

[1167] An "API call" refers to the act of sending a request from software to an API to use its functions or data.

[1168] A "data storage device" is a hardware or software system for the long-term storage and management of data.

[1169] "Conversational" refers to interactions between users and systems, such as text responses via a chatbot.

[1170] The present invention relates to a system that uses a user terminal, a server, and a registered user terminal to utilize a generative AI model and provide a platform that allows seamless use of a variety of services.

[1171] server

[1172] The server performs the following main functions:

[1173] 1. Receiving a request from a user device

[1174] The server receives service requests from the user terminal in natural language using a communication protocol such as HTTP or WebSocket.

[1175] 2. API information registration and management

[1176] The server uses a data storage device to register and manage a list of application programming interfaces (APIs) provided by registrants and their usage methods in a database. Typically, a relational database such as MySQL or PostgreSQL is used.

[1177] 3. Natural Language Processing

[1178] The server analyzes user requests using natural language processing models such as BERT and GPT-3 to understand the service desired by the user and identify the appropriate API.

[1179] 4. Script generation using generative AI models

[1180] Based on the analysis results, the server uses a generative AI model (e.g., GPT-4) to generate a script for calling the API, including the appropriate endpoints and parameters.

[1181] 5. Calling the API

[1182] The server executes the generated script and calls the specified API, and the response from the API is received by the server and processed as needed.

[1183] 6. Return of results

[1184] Finally, the server returns the results of the API call to the user's device, which is a specific response based on the service content requested by the user.

[1185] Registered device

[1186] The Registrant Terminal performs the following functions:

[1187] 1. Enter API information

[1188] The registrant enters the API name, endpoint, usage method, input parameters, and output items into the server through a terminal, which can be done via a web interface.

[1189] 2. Providing use cases and prompts

[1190] Registrants may provide additional use cases and prompts to provide detailed instructions on how to use the API, thereby helping consumers to more easily use the API.

[1191] User terminal

[1192] The user terminal is responsible for the following functions:

[1193] 1. Enter your service request

[1194] Users input the desired service details in chat format, and the details of the request are sent to the server as a request in natural language.

[1195] 2. Receiving and displaying results

[1196] The results returned from the server are displayed on the user's terminal, and the user performs the next operation based on these results.

[1197] Specific examples

[1198] Example 1: Travel booking

[1199] When a user enters "I would like to book a hotel and restaurant in Tokyo for 2 nights and 3 days," the server analyzes the request and identifies the appropriate accommodation and restaurant reservation API. Using generative AI, it generates the necessary script and calls the API to make the reservation. The results are sent back to the user's device.

[1200] Example 2: Linking multiple AI accounts

[1201] When a user types "Tell me today's shopping recommendations and weather information," the server identifies the APIs for shopping and weather information, generates the necessary scripts, and, if necessary, invites other AI accounts into the conversation and calls their respective APIs. The results are then sent back to the user's device.

[1202] Prompt Sentence Examples

[1203] 1. "Create a 3-day, 2-night trip to Tokyo."

[1204] 2. "I'd like to book a hotel in Tokyo for two nights starting February 20th."

[1205] 3. "Tell me what to buy today and what the weather is like."

[1206] 4. "Provide current weather information along with online shopping."

[1207] As described above, the embodiment of the present invention allows users to seamlessly use multiple services through requests in natural language.

[1208] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1209] Step 1:

[1210] (User request input)

[1211] The user inputs their service request in natural language using the user terminal. For example, they input a request such as "I would like to reserve a hotel and restaurant in Tokyo for 2 nights and 3 days." This request is sent from the user terminal to the server.

[1212] input:

[1213] A user's natural language request (e.g., "I would like to book a hotel and restaurant for a 2-night, 3-day stay in Tokyo.")

[1214] output:

[1215] A natural language request is sent to the server

[1216] Specific behavior:

[1217] The user enters the desired service details in the chat window and clicks the send button. The user's terminal generates an HTTP request to send the entered details to the server.

[1218] Step 2:

[1219] (receive and analyze the request by the server)

[1220] The server receives a natural language request from the user's device. Next, it uses natural language processing technology (e.g., BERT or GPT-3 models) to analyze the request and understand the service desired by the user. From the analysis results, it extracts key information (e.g., location, duration, type of service, etc.).

[1221] input:

[1222] A natural language request received by the server

[1223] output:

[1224] Extracted key information (e.g., "Tokyo," "2 nights and 3 days," "hotel," "restaurant")

[1225] Specific behavior:

[1226] The server receives an HTTP request and feeds the text portion of the request into a natural language processing model, which analyzes the text and extracts important keywords and parameters.

[1227] Step 3:

[1228] (Server-specific API)

[1229] Based on the analysis, the server identifies the appropriate API, for example, "Hotel Reservation API" and "Restaurant Reservation API," which involves querying a database to select the appropriate API from a list of applicable APIs.

[1230] input:

[1231] Parsed key information

[1232] output:

[1233] List of identified APIs (e.g., "Hotel Booking API," "Restaurant Booking API")

[1234] Specific behavior:

[1235] The server queries the database to obtain API information that matches the analysis results, and then lists the endpoints and parameters of the obtained APIs.

[1236] Step 4:

[1237] (Script generation by generative AI)

[1238] The server uses a generative AI model (e.g., GPT-4) to generate a script to call the identified API. This script embeds the necessary parameters (e.g., location, period, number of people, etc.) and makes the API call.

[1239] input:

[1240] List of identified APIs and analyzed key information

[1241] output:

[1242] Generated API call script

[1243] Specific behavior:

[1244] The server inputs the API list and key information to the generation AI and requests it to generate a script. The generation AI generates the script and returns it to the server.

[1245] Step 5:

[1246] (API call by server)

[1247] The server executes the generated script and calls the specified API, the response from the API is received by the server and further processed as needed.

[1248] input:

[1249] Generated API call script

[1250] output:

[1251] Response data from the API (e.g., reservation confirmation information)

[1252] Specific behavior:

[1253] The server runs the generated script, calls the API via an HTTP request, receives the API response, and formats the data if necessary.

[1254] Step 6:

[1255] (Return of results)

[1256] The server returns the results of the API call to the user, who can then view the results on their device.

[1257] input:

[1258] Response data from the API

[1259] output:

[1260] Results displayed on the user's device

[1261] Specific behavior:

[1262] The server converts the received response data into a convenient format and sends it to the user's terminal as an HTTP response. The user can then check the results, such as a reservation completion notification, on their terminal.

[1263] (Application example 1)

[1264] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1265] Conventional online shopping systems have had difficulty seamlessly providing the services users need, such as detailed product information, stock availability, and purchasing procedures. In particular, centralizing multiple services and automatically linking them according to user requests has been a challenge.

[1266] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1267] In this invention, the server includes means for receiving service requests in natural language from a user terminal, means for registering and managing a list of APIs provided by registrants and usage methods in a database, means for analyzing the user's requests using natural language processing and identifying an appropriate API, means for generating a script for calling the API using a generation AI, means for executing the generated script and calling the API, means for returning the call results to the user, and means for the user to receive various services such as detailed product information, stock confirmation, and purchase procedures in the virtual store, thereby enabling the user to seamlessly complete a series of shopping procedures in the virtual store.

[1268] A "user terminal" is a device for inputting service requests in natural language and communicating with a server to receive information and results.

[1269] A "service request" is a request that specifies the specific service content that the user wants to receive.

[1270] "Natural language" is the language used by humans in everyday life, and includes informal words and sentences.

[1271] "API" stands for Application Programming Interface, an interface for exchanging functions between software programs.

[1272] A "database" is a system for systematically storing large amounts of data and efficiently searching, updating, and managing them.

[1273] "Natural language processing" is a technology that allows computers to understand, analyze, and generate human language.

[1274] "Generative AI" is an artificial intelligence technology that automatically generates new scripts and content based on input data.

[1275] A "script" is a program code that contains a set of instructions for performing a particular task or process.

[1276] A "virtual store" is a store operated on the Internet, a virtual space that offers products and services.

[1277] "Checking inventory" means checking how much of a particular product is currently available for purchase.

[1278] The "purchase procedure" refers to the process of all the steps required to purchase the selected product.

[1279] This invention is a system that provides AI accounts using generation AI in virtual stores, allowing users to provide detailed product information, check inventory, and complete purchase procedures seamlessly.

[1280] System Configuration

[1281] The system consists of the following main components:

[1282] 1. Server:

[1283] Technologies used: Python, Flask, TensorFlow

[1284] function:

[1285] A service request is received in natural language from a user terminal.

[1286] A database manages a list of APIs provided by registrants and their usage instructions.

[1287] It uses natural language processing to analyze user preferences and identify the appropriate API.

[1288] Use generative AI to generate scripts for API calls.

[1289] Run the generated script to call the API.

[1290] The call result is returned to the user.

[1291] 2. User terminal:

[1292] Technologies used: Kotlin or Swift (for smartphones), ARCore or ARKit (for smart glasses)

[1293] function:

[1294] The user inputs the service request in natural language and sends it to the server.

[1295] The information and results received from the server are displayed and read aloud.

[1296] Operation overview

[1297] User operation

[1298] Customers access the virtual store using an app on their smartphone or smart glasses, and then make requests by voice or text, such as "Please let me know if the Nike Air Max 2023 is in stock."

[1299] Server-side processing

[1300] 1. The server receives a natural language request from the user.

[1301] 2. Use natural language processing technology to analyze the request and identify the appropriate API.

[1302] 3. Use generative AI (using TensorFlow) to generate scripts for API calls corresponding to the request.

[1303] 4. Execute the generated script and call the product information API to retrieve the required data.

[1304] 5. The acquired information is sent back to the user's terminal.

[1305] Server example

[1306] Example prompt sentence:

[1307] A user wants more information about the following product: "Nike Air Max 2023". Get the required information from your API.

[1308] Specific examples of user terminals

[1309] When a user voice-inputs, "Please tell me the stock status of the Nike Air Max 2023," the request is sent to the server, which analyzes and processes it, sends the results back to the device, and displays a message such as "In stock: 10 left" on the smartphone screen.

[1310] Overall system flow

[1311] This system allows users to seamlessly obtain product information, check inventory, and complete purchase procedures within the virtual store, greatly improving the convenience of shopping. Users input their service requests in a chat format, and the server automatically calls the optimal API to provide the necessary information, making purchasing activities more efficient.

[1312] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1313] Step 1:

[1314] A user accesses a virtual store through a smartphone or smart glasses app and inputs their service request in natural language, for example, "Please tell me the stock status of the Nike Air Max 2023." This input is sent by the device to the server.

[1315] Step 2:

[1316] The server receives the input natural language request and analyzes the request using natural language processing technology. Specifically, it analyzes the input text and extracts intent such as "product information" or "inventory check." The input is a natural language request, and the output is the analyzed intent data.

[1317] Step 3:

[1318] The server identifies an appropriate API from a database based on the analyzed intent data. The database contains a list of multiple APIs and their usage methods. The analyzed intent data is the input, and the identified API is the output.

[1319] Step 4:

[1320] The server uses a generative AI model (using TensorFlow) to generate a script for calling the API. The user's request and the identified API information are input into the generative AI model, and the API call script is output. For example, a "script to call the Nike Air Max 2023 stock confirmation API" is generated.

[1321] Step 5:

[1322] The server executes the generated script and calls the specified API to obtain the required data. As a result of the API call, product information and stock status are obtained. The input is the API call script, and the output is the obtained product data.

[1323] Step 6:

[1324] The server processes the acquired data as necessary and converts it into an appropriate format before returning it to the user. For example, it converts inventory information into an easy-to-understand format such as "In stock: 10 remaining." The input is the acquired product data, and the output is the processed information data.

[1325] Step 7:

[1326] The terminal provides the information received from the server to the user by displaying it on the screen or outputting audio. For example, the display of a smartphone or smart glasses might show "In stock: 10 remaining." The input is the information data returned from the server, and the output is display data or audio data.

[1327] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1328] This invention provides a platform where anyone can use and register an AI account using generative AI. In particular, this invention combines an emotion engine that recognizes the user's emotions to provide more personalized services.

[1329] Program Generation and Processing Description

[1330] server:

[1331] The server provides a platform environment that users and registered users can access. The main roles of the server are as follows:

[1332] The server registers and manages a list of APIs provided by registrants and their usage methods in a database. Registrants access the server via their terminals and enter information about the APIs they provide. This includes the API name, endpoint, usage method, required input parameters, and output items.

[1333] Next, the server receives a request in natural language from the user. It uses natural language processing technology to understand the service desired by the user and identifies the corresponding API. Based on the identified API, it uses generative AI to generate an appropriate script. This script is used to call the API according to the user's request.

[1334] The server also has an emotion engine that analyzes the user's chat content to identify their emotional state. This emotional information is used to adjust the script content generated by the AI. For example, if the user is feeling stressed, the script output can be changed to a soothing message.

[1335] Subscriber Device:

[1336] The registrant terminal is a device that connects to the server and registers API information. The registrant accesses the server from the terminal and enters the API name, endpoint, usage method, input parameters, and output items. It also provides examples and prompts for users to use the API in practice, and provides detailed instructions on how to use the API.

[1337] User device:

[1338] The user terminal is a device that allows users to access the platform and converse with the AI ​​account. The user selects the AI ​​account they want to use from their terminal and inputs the desired service details in chat format. For example, if a user inputs "I would like to reserve a hotel and restaurant for 2 nights and 3 days in Tokyo," the server receives this request.

[1339] The server analyzes the request, identifies the corresponding accommodation and restaurant reservation APIs, uses generative AI to generate the necessary scripts, and calls the APIs to make the reservation. Once the reservation is complete, the server returns the results to the user.

[1340] An emotion engine is also integrated into this process, analyzing the user's emotional state in real time. For example, if the system detects that the user is tired, it can send an additional message along with the booking result, such as "We recommend a relaxing hotel."

[1341] Examples:

[1342] Example 1: Travel booking

[1343] User:

[1344] The user is planning a three-day, two-night trip to Tokyo and wants to make hotel and restaurant reservations. During the chat, the emotion engine detects that the user is feeling stressed.

[1345] Operation steps:

[1346] 1. User terminal:

[1347] The user chats with the AI ​​account and says, "I'd like to book a hotel and restaurant in Tokyo for 2 nights and 3 days."

[1348] 2. Server:

[1349] The server analyzes the request and identifies the accommodation and restaurant reservation APIs.

[1350] 3. Server:

[1351] The generating AI generates a script for reservations and embeds the necessary parameters (date of stay, number of people, budget, etc.).

[1352] The emotion engine detects the user's stress level and adjusts the content of the generated script.

[1353] 4. Server:

[1354] Execute the generated script and make an API call.

[1355] 5. Server:

[1356] It receives the results of accommodation and restaurant reservations and returns them to the user with a message such as, "We recommend a relaxing hotel."

[1357] Example 2: Linking multiple AI accounts

[1358] User:

[1359] Users want to shop online and check the weather at the same time. The emotion engine recognizes that users enjoy themselves.

[1360] Operation steps:

[1361] 1. User terminal:

[1362] Users tell the AI ​​account, "Tell me today's recommended shopping and weather information."

[1363] 2. Server:

[1364] The server identifies the shopping and weather APIs and generates the necessary scripts.

[1365] The emotion engine recognizes the user's enjoyment and adapts the generated script accordingly.

[1366] 3. Server:

[1367] If necessary, add other AI accounts to the conversation and call their respective APIs.

[1368] 4. Server:

[1369] It obtains shopping and weather information and returns it to the user along with a message such as, "Today's weather is sunny. Perfect day for shopping!"

[1370] In this way, by combining generative AI and an emotion engine, the present invention can provide optimal services according to the user's emotional state, further improving user satisfaction and convenience.

[1371] The processing flow will be explained below.

[1372] This invention provides a platform where anyone can use and register an AI account using generative AI. In particular, this invention combines an emotion engine that recognizes the user's emotions to provide more personalized services.

[1373] Program Generation and Processing Description

[1374] server:

[1375] Step 1:

[1376] server:

[1377] A list of APIs provided by registrants and their usage methods are registered and managed in a database. Registrants access the server from their terminal and enter information about the APIs they provide. This includes the API name, endpoint, usage method, required input parameters, and output items. This allows the API corresponding to the service desired by the user to be saved in the database.

[1378] Step 2:

[1379] server:

[1380] It receives requests in natural language from users. It uses natural language processing technology to understand the service desired by the user and identify the appropriate API. For example, if a request such as "I would like to book a hotel and restaurant in Tokyo for 2 nights and 3 days" is received via chat, it will be analyzed.

[1381] Step 3:

[1382] server:

[1383] The emotional engine analyzes the user's emotional state from the chat content, thereby identifying the stress or joy the user is feeling.

[1384] Step 4:

[1385] server:

[1386] It uses generative AI to generate scripts to call the identified APIs, and the generated scripts are tailored to the user's emotional state—for example, if they're feeling stressed, the script will be tailored to be relaxing.

[1387] Step 5:

[1388] server:

[1389] Execute the generated script and call the API. The specified API returns data in real time, and the server receives that data.

[1390] Step 6:

[1391] server:

[1392] It then formats the data it receives based on the user's preferences and generates additional messages as needed, such as "We recommend a hotel where you can relax."

[1393] Step 7:

[1394] server:

[1395] The formatted result is sent back to the user's terminal, where the user can receive the result and select the next action.

[1396] Examples:

[1397] Example 1: Travel booking

[1398] User:

[1399] The user is planning a three-day, two-night trip to Tokyo and wants to make hotel and restaurant reservations. During the chat, the emotion engine detects that the user is feeling stressed.

[1400] Operation steps:

[1401] Step 1:

[1402] User device:

[1403] The user chats with the AI ​​account and says, "I would like to book a hotel and restaurant in Tokyo for 2 nights and 3 days."

[1404] Step 2:

[1405] server:

[1406] The server analyzes the request and identifies the accommodation and restaurant reservation APIs.

[1407] Step 3:

[1408] server:

[1409] The generating AI generates a script for reservations and embeds the necessary parameters (date of stay, number of people, budget, etc.).

[1410] Step 4:

[1411] server:

[1412] The emotion engine detects the user's stress level and adjusts the content of the generated script.

[1413] Step 5:

[1414] server:

[1415] Execute the generated script and make an API call.

[1416] Step 6:

[1417] server:

[1418] It receives hotel and restaurant booking results and formats the results with a message such as "We recommend a relaxing hotel."

[1419] Step 7:

[1420] server:

[1421] The formatted result is sent back to the user's terminal.

[1422] Example 2: Linking multiple AI accounts

[1423] User:

[1424] Users want to shop online and check the weather at the same time. The emotion engine recognizes that users enjoy themselves.

[1425] Operation steps:

[1426] Step 1:

[1427] User device:

[1428] Users tell the AI ​​account, "Tell me today's recommended shopping and weather information."

[1429] Step 2:

[1430] server:

[1431] The server identifies the shopping and weather APIs and generates the necessary scripts.

[1432] Step 3:

[1433] server:

[1434] The emotion engine recognizes the user's enjoyment and adapts the generated script accordingly.

[1435] Step 4:

[1436] server:

[1437] If necessary, add other AI accounts to the conversation and call their respective APIs.

[1438] Step 5:

[1439] server:

[1440] Get shopping and weather information and format the results with a message like "Today's weather is sunny. Great day for shopping!"

[1441] Step 6:

[1442] server:

[1443] The formatted result is sent back to the user's terminal.

[1444] In this way, by combining generative AI and an emotion engine, the present invention can provide optimal services according to the user's emotional state, further improving user satisfaction and convenience.

[1445] Example 2

[1446] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1447] Modern information systems require users to receive a variety of services quickly and in a personalized manner. However, current systems often lack the functionality to analyze the user's emotional state and reflect the results in the content of the service provided. As a result, services provided without considering the user's emotional state may reduce user satisfaction. Furthermore, when providing complex services by linking multiple APIs, there is a lack of an efficient process for identifying APIs and generating scripts. This poses the challenge of requiring a great deal of effort and time to meet user requests.

[1448] The specification process by the specification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes: means for receiving a service request from a user terminal in natural language; means for registering and managing a list of APIs provided by registrants and usage methods in a database; means for analyzing the user's request and identifying an appropriate API using natural language processing; means for generating a script for calling the API using a generative AI model; means for executing the generated script to call the API; means for returning the call result to the user; means including an emotion engine for analyzing the user's emotional state; and means for adjusting the content of the generated script depending on the user's emotional state. This makes it possible to quickly provide personalized services according to the user's emotional state and efficiently link multiple APIs.

[1449] "User terminal" means a device through which a user accesses the system and inputs service requests.

[1450] A "registrant" is a person who registers information about the API that they provide in the system.

[1451] "API" stands for Application Programming Interface, an interface for providing specific functions and data to the outside world.

[1452] A "database" is a system that stores information in an organized manner and allows it to be retrieved when needed.

[1453] "Natural language processing" is a technology that allows computers to understand and process human language.

[1454] A "generative AI model" is an algorithm that uses artificial intelligence to generate scripts for specific tasks or purposes.

[1455] A "script" is program code for automatically executing a series of instructions.

[1456] An "emotion engine" is a technology that analyzes a user's emotional state from their text or input.

[1457] "Call result" refers to the response or data obtained after executing an API.

[1458] "Natural language" refers to the language used by humans on a daily basis, translated into a form that the system can understand.

[1459] "Fusion of multiple APIs" refers to the use of multiple APIs simultaneously or sequentially to provide one or more services.

[1460] This invention is a highly personalized service provision system that utilizes a generative AI model and an emotion engine. This system has the function of calling the optimal API based on the user's wishes and emotional state and returning the results to the user.

[1461] System Overview

[1462] The system includes the following major hardware and software components:

[1463] Server: Responsible for managing the API, parsing requests, generating scripts, running the emotion engine, and returning results.

[1464] User terminal: A device (PC, smartphone, etc.) that allows a user to access the system and input service requests.

[1465] Subscriber terminal: A device used to add API information to the system.

[1466] Database (e.g. MySQL, PostgreSQL): Stores and manages registered API information.

[1467] Natural language processing engines (e.g. Google NLP API, spaCy): used to analyze user preferences.

[1468] A generative AI model (e.g., OpenAI GPT-3) used to generate scripts for API calls.

[1469] Emotion engine (e.g., Microsoft Azure Emotion API): Used to analyze the user's emotional state.

[1470] Detailed System Operation

[1471] 1. Receiving Service Requests:

[1472] The user inputs their desired service request in natural language from their device and sends it to the server. For example, a user might input, "I would like to reserve a hotel and restaurant for 2 nights and 3 days in Tokyo."

[1473] 2. Parse the request:

[1474] The server uses a natural language processing engine to analyze the request and understand the user's wishes, extracting the keywords "hotel reservation" and "restaurant reservation."

[1475] 3. Identify APIs and generate scripts:

[1476] The server identifies the APIs corresponding to "hotel reservation" and "restaurant reservation" from the database. Then, it generates a script using the generative AI model. For example, it sends a prompt such as "Please generate a script using the APIs for hotel reservation and restaurant reservation."

[1477] 4. Applying the Emotion Engine:

[1478] The server analyzes the user's chat content using an emotion engine to identify the user's emotional state. For example, if the user is feeling stressed, the emotion engine conveys that information. The server then adjusts the content of the generated script to include a message such as, "We recommend a relaxing hotel."

[1479] 5. API call and return results:

[1480] The server runs the generated script, calls the appropriate API, receives the response from the API, and returns the results to the user, including the specific reservation information and an optional message (e.g., "We recommend a relaxing hotel").

[1481] Specific examples

[1482] Book a trip

[1483] User: Planning a trip, type in chat, "I'd like to book a hotel and restaurant in Tokyo for 2 nights and 3 days."

[1484] Server: Receives the request and analyzes it using a natural language processing engine. It generates an appropriate script using a generative AI model and analyzes the user's emotions using an emotion engine. If stress is detected, it adds a soothing message to the script.

[1485] Execution and return: Call the API to execute the reservation and return the reservation results to the user terminal.

[1486] Multiple API integrations

[1487] User: Wants to shop online and check the weather at the same time, so types in "Tell me today's recommended shopping and weather information."

[1488] Server: Identifies the shopping and weather APIs and generates the necessary scripts. If the emotion engine determines that the user is having fun, it includes a message that matches this excitement.

[1489] Execution and return: Calls multiple APIs and returns the obtained shopping information and weather information to the user along with a message such as "Today's weather is sunny. It's a great day for shopping!"

[1490] In this way, by combining a generative AI model and an emotion engine, the present invention can provide optimal services according to the user's emotional state, thereby improving user satisfaction and realizing efficient and personalized service provision.

[1491] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1492] Step 1: Enter and submit your service request on the user's device

[1493] Specific operation: The user uses the user terminal to input the desired service details in chat format. For example, the user might input "I would like to reserve a hotel and restaurant in Tokyo for 2 nights and 3 days."

[1494] Input: A message of service request written in natural language.

[1495] Output: The request message sent to the server.

[1496] Step 2: Server receives and analyzes the request

[1497] Specific operation: The server receives the request sent from the user terminal and analyzes the request using a natural language processing engine (e.g., Google NLP API, spaCy). Through the analysis, the keywords "hotel reservation" and "restaurant reservation" are extracted.

[1498] Input: A natural language request from the user.

[1499] Output: Parsed preference (e.g. "Hotel reservation", "Restaurant reservation").

[1500] Step 3: Identify the corresponding API

[1501] Specific operation: The server refers to the database and identifies the API corresponding to the analyzed request. For example, it retrieves the "Hotel Reservation" API and the "Restaurant Reservation" API.

[1502] Input: Parsed desired content.

[1503] Output: Identified API (e.g. "Hotel Booking" API, "Restaurant Booking" API).

[1504] Step 4: Generating scripts using generative AI models

[1505] Specific operation: The server uses a generative AI model (e.g., OpenAI GPT-3) to generate a script for API calls based on the user's request. The prompt text is "Please generate a script using the hotel reservation and restaurant reservation APIs."

[1506] Input: The API identified and what you want.

[1507] Data processing: Send prompt text to the generative AI model to generate a script for calling the API.

[1508] Output: The generated script.

[1509] Step 5: Emotional state analysis by the emotion engine

[1510] Specific operation: The server analyzes the user's chat content using an emotion engine (e.g., Microsoft Azure Emotion API) to identify the user's emotional state. For example, it detects whether the user is feeling stressed.

[1511] Input: User chat content.

[1512] Data Computation: Analyze emotional states using an emotion engine.

[1513] Output: Detected emotional state (e.g., stress).

[1514] Step 6: Adjust the script content

[1515] Specific behavior: The server adjusts the content of the generated script based on the detected emotional state. For example, if a user is feeling stressed, it might add a message such as "We recommend a relaxing hotel."

[1516] Input: Generated script and emotional state.

[1517] Data processing: The message part of the script is changed depending on the emotional state.

[1518] Output: The adjusted script.

[1519] Step 7: Making API calls

[1520] Specific operation: The server executes the tailored script, calls the specified API, and receives the response from the API.

[1521] Input: The adjusted script.

[1522] Output: The response data from the API.

[1523] Step 8: Returning the results

[1524] Specific operation: The server returns the response data from the API to the user's device and displays the result, for example, information that the reservation was successful and a message such as "We recommend a relaxing hotel."

[1525] Input: Response data from the API.

[1526] Data processing: Integrate response data and emotional responses.

[1527] Output: Results returned to the user's terminal.

[1528] (Application example 2)

[1529] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1530] Conventional automated service provision systems have difficulty in providing personalized responses that take into account the user's emotions, making it difficult to increase user satisfaction. In particular, when providing services in physical stores, flexible and appropriate responses that take into account the customer's psychological state are required.

[1531] The specification processing by the specification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for receiving a service request from a user terminal in natural language, means for registering and managing a list of APIs provided by registrants and their usage methods in a database, means for analyzing the user's request using natural language processing and identifying an appropriate API, means for generating a script for calling the API using a generation AI, means for executing the generated script and calling the API, means for returning the call result to the user, means for analyzing the user's emotional state, and means for adjusting the content of the generated script based on the emotional state. This makes it possible to provide personalized services according to the user's emotional state.

[1532] A "user terminal" is a device that allows a user to input service requests.

[1533] "Service Desired" refers to the specific service content that the user wishes to use.

[1534] "Natural language" refers to the language that humans use on a daily basis, not a specialized language such as a program language.

[1535] "API" stands for Application Programming Interface, an interface for exchanging functions and data between software programs.

[1536] A "database" is a system for efficiently storing and managing data.

[1537] "Natural language processing" is a technology that allows computers to understand and analyze natural language.

[1538] "Generative AI" is an artificial intelligence technology that generates appropriate responses and results based on input data.

[1539] A "script" is a program code that contains a series of instructions or commands.

[1540] "Emotional state" refers to the user's psychological state or mood, including emotions such as stress and joy.

[1541] "Emotion analysis" is a technology that determines a user's emotional state based on facial expressions, tone of voice, etc.

[1542] An "API call" is an operation that executes a specified API to obtain data or functionality.

[1543] This invention is a system that provides personalized customer service in brick-and-mortar stores, and is realized using smart glasses and emotion analysis sensor devices. The main components of the system are the user terminal, emotion analysis sensor, generative AI, natural language processing, and a server that manages and executes each API.

[1544] Major hardware and software used

[1545] Hardware: smart glasses (e.g., Google Glass), sensor devices for emotion analysis (e.g., Microsoft Azure Kinect)

[1546] software:

[1547] Natural language processing libraries (e.g. spaCy)

[1548] Sentiment analysis engine (e.g. Microsoft Azure Emotion API)

[1549] Generative AI systems (e.g., OpenAI GPT-4)

[1550] API management and databases (e.g. AWS, Firebase)

[1551] System Operation Overview

[1552] 1. Obtaining customer sentiment data

[1553] The smart glasses' camera and emotion-analysis sensors capture the customer's facial expressions and tone of voice in real time, allowing them to determine their emotional state on the spot. For example, if a customer is feeling stressed, that information will be detected.

[1554] 2. Emotional Data Analysis

[1555] The acquired emotional data is sent to the cloud and analyzed by an emotion analysis engine. For example, the Microsoft Azure Emotion API is used to identify emotional states such as "high stress level" or "enjoyed" based on facial expressions and voice characteristics.

[1556] 3. Proposal generation using generative AI

[1557] Based on the analysis results, the generative AI system will make optimal product suggestions. For example, if a customer is feeling stressed, the generative AI will generate a script suggesting products with a relaxing effect and display it on the smart glasses' display. The generative AI uses OpenAI GPT-4 to generate natural and appropriate suggestion messages.

[1558] 4. Implementing product proposals

[1559] The staff member checks the information displayed on the smart glasses and suggests individual products and services to the customer. The staff member then introduces specific products and encourages the customer to try them according to the generated suggestion message. For example, the staff member might say, "This aroma diffuser is a recommended product to help you relax."

[1560] 5. Feedback and Adjustments

[1561] After making a proposal, the system reanalyzes the customer's reactions (facial expressions and tone of voice) and adjusts the proposal if necessary, allowing the system to continue providing optimal service based on customer feedback.

[1562] Specific prompt examples

[1563] For example, in a brick-and-mortar scenario:

[1564] Prompt Sentence Examples

[1565] "The customer's facial expression is stressed and their voice tone is low. They are currently in the relaxation section of the store. The service they offer is product suggestions. The priority keywords are relaxation, healing, and stress relief."

[1566] In this way, the system of the present invention can provide personalized services in physical stores according to the emotional state of the user, thereby achieving high customer satisfaction.

[1567] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1568] Step 1:

[1569] The smart glasses' camera and emotion analysis sensor are used to capture the customer's facial expressions and tone of voice in real time, and the data is acquired. The input is the customer's image and voice data, and the output is the captured emotion data. The acquired data is sent to the server.

[1570] Step 2:

[1571] The server sends the acquired emotional data to the cloud and analyzes it using an emotion analysis engine (e.g., Microsoft Azure Emotion API). The input is the captured emotional data, and the output is the analyzed emotional state (e.g., "stressed," "enjoyed," etc.). The analysis results are passed to the generative AI.

[1572] Step 3:

[1573] The server uses a generative AI (e.g., OpenAI GPT-4) to generate an appropriate product recommendation script based on the analyzed emotional state. The input is the analyzed emotional state and a prompt, and the output is the generated recommendation script. For example, a recommendation script is generated that says, "This aroma diffuser is a recommended product to help you relax."

[1574] Step 4:

[1575] The server displays the generated proposal script on the display of the smart glasses. The input is the generated proposal script, and the output is the proposal message displayed on the smart glasses. The store staff checks the display and makes a proposal to the customer.

[1576] Step 5:

[1577] The staff introduces specific products to the customer and guides them to try them out, following the suggestion message displayed on the smart glasses. The input is the suggestion message displayed on the smart glasses, and the output is the product suggestion to the customer.

[1578] Step 6:

[1579] The customer's reaction is recaptured and sent to the server for feedback. The input is image and voice data of the customer after Yori's service is provided, and the output is the recaptured emotional data. Based on the feedback, the proposal content is adjusted as needed. The server re-analyzes this new emotional data and continues to make optimal proposals.

[1580] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1581] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1582] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1583] [Fourth embodiment]

[1584] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1585] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1586] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1587] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1588] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1589] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1590] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1591] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1592] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1593] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1594] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1595] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1596] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1597] The present invention provides a platform where anyone can register and use an AI account that uses a generating AI. Users can converse with the AI ​​account on this platform and seamlessly use multiple services. Specific embodiments of the present invention are described below.

[1598] Program Generation and Processing Description

[1599] server:

[1600] The server provides a platform environment that users and registered users can access. The main roles of the server are as follows:

[1601] The server registers and manages a list of APIs provided by registrants and their usage methods in a database. Registrants access the server through their terminals and enter information about the APIs they provide. This information includes the API name, endpoint, usage method, required input parameters, and output items.

[1602] The server also receives and analyzes natural language requests from users, using natural language processing technology to understand the service desired by the user and identify the corresponding API.

[1603] Based on the identified API, the server uses a generation AI to generate the appropriate script to call the API as desired by the user. The generated script is then executed by the server to call the specified API.

[1604] Subscriber Device:

[1605] The registrant terminal is a device that connects to the server and registers API information. The registrant accesses the server from the terminal and enters the API name, endpoint, usage method, input parameters, and output items. The registrant also provides additional use cases and prompts to explain in detail how to use the API.

[1606] User device:

[1607] The user terminal is a device that allows users to access the platform and converse with the AI ​​account. The user selects the AI ​​account they want to use from their terminal and inputs the desired service details in chat format. For example, if a user inputs "I would like to reserve a hotel and restaurant for 2 nights and 3 days in Tokyo," the server receives this request.

[1608] The server analyzes the request, identifies the corresponding accommodation and restaurant reservation API, uses the generative AI to generate the necessary script, and calls the API to make the reservation. After the reservation is completed, the server returns the result to the user.

[1609] Examples:

[1610] Example 1: Travel booking

[1611] User:

[1612] The user is planning a three-day, two-night trip to Tokyo and would like to make hotel and restaurant reservations.

[1613] Operation steps:

[1614] 1. User terminal:

[1615] The user chats with the AI ​​account and says, "I'd like to book a hotel and restaurant in Tokyo for 2 nights and 3 days."

[1616] 2. Server:

[1617] The server analyzes the request and identifies the accommodation and restaurant reservation APIs.

[1618] 3. Server:

[1619] The generating AI generates a script for reservations and embeds the necessary parameters (date of stay, number of people, budget, etc.).

[1620] 4. Server:

[1621] Execute the generated script and make an API call.

[1622] 5. Server:

[1623] Receives accommodation and restaurant reservation results and returns the results to the user.

[1624] Example 2: Linking multiple AI accounts

[1625] User:

[1626] Users want their AI assistant to help them shop online and check the weather at the same time.

[1627] Operation steps:

[1628] 1. User terminal:

[1629] Users can chat with the AI ​​account and ask, "Tell me today's shopping recommendations and weather information."

[1630] 2. Server:

[1631] The server identifies the shopping and weather APIs and generates the necessary scripts.

[1632] 3. Server:

[1633] If necessary, add other AI accounts to the conversation and call their respective APIs.

[1634] 4. Server:

[1635] It retrieves shopping and weather information and returns it to the user.

[1636] This system allows users to seamlessly link different services together, greatly improving convenience and efficiency.

[1637] The processing flow will be explained below.

[1638] Step 1:

[1639] Subscriber Device:

[1640] The registrant accesses the server from a terminal and opens the API registration screen. The registrant enters the API name, endpoint, usage method, required input parameters, and output items. In addition, the registrant enters API usage examples and prompts, providing a detailed explanation of how to use the API. After confirming the entered information, the registrant submits the registration to the server.

[1641] Step 2:

[1642] server:

[1643] The server analyzes the received API registration details and stores them in a database. It then verifies the API usage and sample code to ensure there are no problems. Once verification is complete, it returns a registration completion message to the registrant's terminal.

[1644] Step 3:

[1645] User device:

[1646] Users access the platform from their device and select the AI ​​account they want to use. On the selected chat screen, they enter the desired service details in natural language. For example, they could enter, "I would like to book a hotel and restaurant in Tokyo for 2 nights and 3 days."

[1647] Step 4:

[1648] server:

[1649] The server analyzes messages received through the chat system, uses natural language processing to analyze the user's wishes and identify the necessary API, and prepares data to automatically generate API parameters based on the analysis results.

[1650] Step 5:

[1651] server:

[1652] The server calls the generation AI engine and generates a script for calling the API. It provides the "identified API name" and "required parameters" to the generation AI and instructs it to generate the script. It then performs syntax checks and logic verification on the generated script.

[1653] Step 6:

[1654] server:

[1655] Add error handling logic to complete the final script. Run the completed script in the execution environment. Call the specified API and obtain the results in real time. Check that the obtained results follow the correct format.

[1656] Step 7:

[1657] server:

[1658] The acquired data is formatted based on the user's wishes. The formatted results are sent back to the user's device. For example, a message such as "Your hotel reservation for 2 nights and 3 days in Tokyo has been completed. Details are as follows:..." is sent.

[1659] Step 8:

[1660] User device:

[1661] The user can check the results on their device and select the next action. For example, they may send another message saying, "Please make a restaurant reservation as well."

[1662] Step 9:

[1663] server:

[1664] The server receives a new request from the user, prepares a new API call while preserving the existing conversation context, uses the generative AI again to generate the necessary scripts and make additional API calls, and then integrates all the results and returns the combined information to the user.

[1665] Step 10:

[1666] User device:

[1667] The user can check the final result on the device and send feedback if necessary, such as a message saying, "I have confirmed the reservation details. There are no problems."

[1668] Step 11:

[1669] server:

[1670] The server receives the feedback and stores it in a database for service improvement purposes. It then sends a message to the user to end the conversation and terminates the session.

[1671] In this way, multiple services that use generative AI can be used in seamless collaboration.

[1672] Example 1

[1673] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1674] Conventional systems make it difficult for users to seamlessly use a variety of services. In particular, they require the manual integration of multiple application programming interfaces (APIs), a complex and time-consuming process. Furthermore, there was a lack of automated means for analyzing natural language requests, selecting the appropriate API, and generating the corresponding program. As a result, users were required to have high operational skills, and the system was difficult to use.

[1675] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1676] In this invention, the server includes means for receiving a service request from a user terminal in natural language, means for registering and managing a list of application programming interfaces provided by registrants and usage methods in a data storage device, means for analyzing the user's request using natural language processing and identifying an appropriate application programming interface, means for generating a program for calling the application programming interface using generative artificial intelligence, means for executing the generated program and calling the application programming interface, and means for returning the call result to the user. This enables users to seamlessly link and use a variety of services through requests in natural language.

[1677] "User terminal" means an electronic device through which a user inputs service requests and interacts with the system.

[1678] "Service requests" refers to requests or requests made by users regarding the content of services provided through the system.

[1679] "Natural language" refers to the words and forms of expression used by humans on a daily basis, usually expressed in text form.

[1680] A "server" is a computer system that processes requests from user terminals and registrant terminals and provides data and services.

[1681] "Registrant" means a person or entity that provides and manages one or more application programming interfaces (APIs) to a system.

[1682] An "application programming interface (API)" is a set of definitions or protocols that allow software components to communicate.

[1683] "Information management" is the process of using a database to register, update, and delete API information and maintain how it is used.

[1684] "Natural language processing" refers to the use of computers to analyze, understand, and generate natural language, for example through the use of text analysis and language models.

[1685] "Generative AI" is a general term for AI techniques for performing generative tasks (such as text generation and code generation), using, for example, generative models and neural networks.

[1686] "Program generation" is the process of automatically creating code or scripts to perform a specific task.

[1687] An "API call" refers to the act of sending a request from software to an API to use its functions or data.

[1688] A "data storage device" is a hardware or software system for the long-term storage and management of data.

[1689] "Conversational" refers to interactions between users and systems, such as text responses via a chatbot.

[1690] The present invention relates to a system that uses a user terminal, a server, and a registered user terminal to utilize a generative AI model and provide a platform that allows seamless use of a variety of services.

[1691] server

[1692] The server performs the following main functions:

[1693] 1. Receiving a request from a user device

[1694] The server receives service requests from the user terminal in natural language using a communication protocol such as HTTP or WebSocket.

[1695] 2. API information registration and management

[1696] The server uses a data storage device to register and manage a list of application programming interfaces (APIs) provided by registrants and their usage methods in a database. Typically, a relational database such as MySQL or PostgreSQL is used.

[1697] 3. Natural Language Processing

[1698] The server analyzes user requests using natural language processing models such as BERT and GPT-3 to understand the service desired by the user and identify the appropriate API.

[1699] 4. Script generation using generative AI models

[1700] Based on the analysis results, the server uses a generative AI model (e.g., GPT-4) to generate a script for calling the API, including the appropriate endpoints and parameters.

[1701] 5. Calling the API

[1702] The server executes the generated script and calls the specified API, and the response from the API is received by the server and processed as needed.

[1703] 6. Return of results

[1704] Finally, the server returns the results of the API call to the user's device, which is a specific response based on the service content requested by the user.

[1705] Registered device

[1706] The Registrant Terminal performs the following functions:

[1707] 1. Enter API information

[1708] The registrant enters the API name, endpoint, usage method, input parameters, and output items into the server through a terminal, which can be done via a web interface.

[1709] 2. Providing use cases and prompts

[1710] Registrants may provide additional use cases and prompts to provide detailed instructions on how to use the API, thereby helping consumers to more easily use the API.

[1711] User terminal

[1712] The user terminal is responsible for the following functions:

[1713] 1. Enter your service request

[1714] Users input the desired service details in chat format, and the details of the request are sent to the server as a request in natural language.

[1715] 2. Receiving and displaying results

[1716] The results returned from the server are displayed on the user's terminal, and the user performs the next operation based on these results.

[1717] Specific examples

[1718] Example 1: Travel booking

[1719] When a user enters "I would like to book a hotel and restaurant in Tokyo for 2 nights and 3 days," the server analyzes the request and identifies the appropriate accommodation and restaurant reservation API. Using generative AI, it generates the necessary script and calls the API to make the reservation. The results are sent back to the user's device.

[1720] Example 2: Linking multiple AI accounts

[1721] When a user types "Tell me today's shopping recommendations and weather information," the server identifies the APIs for shopping and weather information, generates the necessary scripts, and, if necessary, invites other AI accounts into the conversation and calls their respective APIs. The results are then sent back to the user's device.

[1722] Prompt Sentence Examples

[1723] 1. "Create a 3-day, 2-night trip to Tokyo."

[1724] 2. "I'd like to book a hotel in Tokyo for two nights starting February 20th."

[1725] 3. "Tell me what to buy today and what the weather is like."

[1726] 4. "Provide current weather information along with online shopping."

[1727] As described above, the embodiment of the present invention allows users to seamlessly use multiple services through requests in natural language.

[1728] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1729] Step 1:

[1730] (User request input)

[1731] The user inputs their service request in natural language using the user terminal. For example, they input a request such as "I would like to reserve a hotel and restaurant in Tokyo for 2 nights and 3 days." This request is sent from the user terminal to the server.

[1732] input:

[1733] A user's natural language request (e.g., "I would like to book a hotel and restaurant for a 2-night, 3-day stay in Tokyo.")

[1734] output:

[1735] A natural language request is sent to the server

[1736] Specific behavior:

[1737] The user enters the desired service details in the chat window and clicks the send button. The user's terminal generates an HTTP request to send the entered details to the server.

[1738] Step 2:

[1739] (receive and analyze the request by the server)

[1740] The server receives a natural language request from the user's device. Next, it uses natural language processing technology (e.g., BERT or GPT-3 models) to analyze the request and understand the service desired by the user. From the analysis results, it extracts key information (e.g., location, duration, type of service, etc.).

[1741] input:

[1742] A natural language request received by the server

[1743] output:

[1744] Extracted key information (e.g., "Tokyo," "2 nights and 3 days," "hotel," "restaurant")

[1745] Specific behavior:

[1746] The server receives an HTTP request and feeds the text portion of the request into a natural language processing model, which analyzes the text and extracts important keywords and parameters.

[1747] Step 3:

[1748] (Server-specific API)

[1749] Based on the analysis, the server identifies the appropriate API, for example, "Hotel Reservation API" and "Restaurant Reservation API," which involves querying a database to select the appropriate API from a list of applicable APIs.

[1750] input:

[1751] Parsed key information

[1752] output:

[1753] List of identified APIs (e.g., "Hotel Booking API," "Restaurant Booking API")

[1754] Specific behavior:

[1755] The server queries the database to obtain API information that matches the analysis results, and then lists the endpoints and parameters of the obtained APIs.

[1756] Step 4:

[1757] (Script generation by generative AI)

[1758] The server uses a generative AI model (e.g., GPT-4) to generate a script to call the identified API. This script embeds the necessary parameters (e.g., location, period, number of people, etc.) and makes the API call.

[1759] input:

[1760] List of identified APIs and analyzed key information

[1761] output:

[1762] Generated API call script

[1763] Specific behavior:

[1764] The server inputs the API list and key information to the generation AI and requests it to generate a script. The generation AI generates the script and returns it to the server.

[1765] Step 5:

[1766] (API call by server)

[1767] The server executes the generated script and calls the specified API, the response from the API is received by the server and further processed as needed.

[1768] input:

[1769] Generated API call script

[1770] output:

[1771] Response data from the API (e.g., reservation confirmation information)

[1772] Specific behavior:

[1773] The server runs the generated script, calls the API via an HTTP request, receives the API response, and formats the data if necessary.

[1774] Step 6:

[1775] (Return of results)

[1776] The server returns the results of the API call to the user, who can then view the results on their device.

[1777] input:

[1778] Response data from the API

[1779] output:

[1780] Results displayed on the user's device

[1781] Specific behavior:

[1782] The server converts the received response data into a convenient format and sends it to the user's terminal as an HTTP response. The user can then check the results, such as a reservation completion notification, on their terminal.

[1783] (Application example 1)

[1784] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1785] Conventional online shopping systems have had difficulty seamlessly providing the services users need, such as detailed product information, stock availability, and purchasing procedures. In particular, centralizing multiple services and automatically linking them according to user requests has been a challenge.

[1786] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1787] In this invention, the server includes means for receiving service requests in natural language from a user terminal, means for registering and managing a list of APIs provided by registrants and usage methods in a database, means for analyzing the user's requests using natural language processing and identifying an appropriate API, means for generating a script for calling the API using a generation AI, means for executing the generated script and calling the API, means for returning the call results to the user, and means for the user to receive various services such as detailed product information, stock confirmation, and purchase procedures in the virtual store, thereby enabling the user to seamlessly complete a series of shopping procedures in the virtual store.

[1788] A "user terminal" is a device for inputting service requests in natural language and communicating with a server to receive information and results.

[1789] A "service request" is a request that specifies the specific service content that the user wants to receive.

[1790] "Natural language" is the language used by humans in everyday life, and includes informal words and sentences.

[1791] "API" stands for Application Programming Interface, an interface for exchanging functions between software programs.

[1792] A "database" is a system for systematically storing large amounts of data and efficiently searching, updating, and managing them.

[1793] "Natural language processing" is a technology that allows computers to understand, analyze, and generate human language.

[1794] "Generative AI" is an artificial intelligence technology that automatically generates new scripts and content based on input data.

[1795] A "script" is a program code that contains a set of instructions for performing a particular task or process.

[1796] A "virtual store" is a store operated on the Internet, a virtual space that offers products and services.

[1797] "Checking inventory" means checking how much of a particular product is currently available for purchase.

[1798] The "purchase procedure" refers to the process of all the steps required to purchase the selected product.

[1799] This invention is a system that provides AI accounts using generation AI in virtual stores, allowing users to provide detailed product information, check inventory, and complete purchase procedures seamlessly.

[1800] System Configuration

[1801] The system consists of the following main components:

[1802] 1. Server:

[1803] Technologies used: Python, Flask, TensorFlow

[1804] function:

[1805] A service request is received in natural language from a user terminal.

[1806] A database manages a list of APIs provided by registrants and their usage instructions.

[1807] It uses natural language processing to analyze user preferences and identify the appropriate API.

[1808] Use generative AI to generate scripts for API calls.

[1809] Run the generated script to call the API.

[1810] The call result is returned to the user.

[1811] 2. User terminal:

[1812] Technologies used: Kotlin or Swift (for smartphones), ARCore or ARKit (for smart glasses)

[1813] function:

[1814] The user inputs the service request in natural language and sends it to the server.

[1815] The information and results received from the server are displayed and read aloud.

[1816] Operation overview

[1817] User operation

[1818] Customers access the virtual store using an app on their smartphone or smart glasses, and then make requests by voice or text, such as "Please let me know if the Nike Air Max 2023 is in stock."

[1819] Server-side processing

[1820] 1. The server receives a natural language request from the user.

[1821] 2. Use natural language processing technology to analyze the request and identify the appropriate API.

[1822] 3. Use generative AI (using TensorFlow) to generate scripts for API calls corresponding to the request.

[1823] 4. Execute the generated script and call the product information API to retrieve the required data.

[1824] 5. The acquired information is sent back to the user's terminal.

[1825] Server example

[1826] Example prompt sentence:

[1827] A user wants more information about the following product: "Nike Air Max 2023". Get the required information from your API.

[1828] Specific examples of user terminals

[1829] When a user voice-inputs, "Please tell me the stock status of the Nike Air Max 2023," the request is sent to the server, which analyzes and processes it, sends the results back to the device, and displays a message such as "In stock: 10 left" on the smartphone screen.

[1830] Overall system flow

[1831] This system allows users to seamlessly obtain product information, check inventory, and complete purchase procedures within the virtual store, greatly improving the convenience of shopping. Users input their service requests in a chat format, and the server automatically calls the optimal API to provide the necessary information, making purchasing activities more efficient.

[1832] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1833] Step 1:

[1834] A user accesses a virtual store through a smartphone or smart glasses app and inputs their service request in natural language, for example, "Please tell me the stock status of the Nike Air Max 2023." This input is sent by the device to the server.

[1835] Step 2:

[1836] The server receives the input natural language request and analyzes the request using natural language processing technology. Specifically, it analyzes the input text and extracts intent such as "product information" or "inventory check." The input is a natural language request, and the output is the analyzed intent data.

[1837] Step 3:

[1838] The server identifies an appropriate API from a database based on the analyzed intent data. The database contains a list of multiple APIs and their usage methods. The analyzed intent data is the input, and the identified API is the output.

[1839] Step 4:

[1840] The server uses a generative AI model (using TensorFlow) to generate a script for calling the API. The user's request and the identified API information are input into the generative AI model, and the API call script is output. For example, a "script to call the Nike Air Max 2023 stock confirmation API" is generated.

[1841] Step 5:

[1842] The server executes the generated script and calls the specified API to obtain the required data. As a result of the API call, product information and stock status are obtained. The input is the API call script, and the output is the obtained product data.

[1843] Step 6:

[1844] The server processes the acquired data as necessary and converts it into an appropriate format before returning it to the user. For example, it converts inventory information into an easy-to-understand format such as "In stock: 10 remaining." The input is the acquired product data, and the output is the processed information data.

[1845] Step 7:

[1846] The terminal provides the information received from the server to the user by displaying it on the screen or outputting audio. For example, the display of a smartphone or smart glasses might show "In stock: 10 remaining." The input is the information data returned from the server, and the output is display data or audio data.

[1847] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1848] This invention provides a platform where anyone can use and register an AI account using generative AI. In particular, this invention combines an emotion engine that recognizes the user's emotions to provide more personalized services.

[1849] Program Generation and Processing Description

[1850] server:

[1851] The server provides a platform environment that users and registered users can access. The main roles of the server are as follows:

[1852] The server registers and manages a list of APIs provided by registrants and their usage methods in a database. Registrants access the server via their terminals and enter information about the APIs they provide. This includes the API name, endpoint, usage method, required input parameters, and output items.

[1853] Next, the server receives a request in natural language from the user. It uses natural language processing technology to understand the service desired by the user and identifies the corresponding API. Based on the identified API, it uses generative AI to generate an appropriate script. This script is used to call the API according to the user's request.

[1854] The server also has an emotion engine that analyzes the user's chat content to identify their emotional state. This emotional information is used to adjust the script content generated by the AI. For example, if the user is feeling stressed, the script output can be changed to a soothing message.

[1855] Subscriber Device:

[1856] The registrant terminal is a device that connects to the server and registers API information. The registrant accesses the server from the terminal and enters the API name, endpoint, usage method, input parameters, and output items. It also provides examples and prompts for users to use the API in practice, and provides detailed instructions on how to use the API.

[1857] User device:

[1858] The user terminal is a device that allows users to access the platform and converse with the AI ​​account. The user selects the AI ​​account they want to use from their terminal and inputs the desired service details in chat format. For example, if a user inputs "I would like to reserve a hotel and restaurant for 2 nights and 3 days in Tokyo," the server receives this request.

[1859] The server analyzes the request, identifies the corresponding accommodation and restaurant reservation APIs, uses generative AI to generate the necessary scripts, and calls the APIs to make the reservation. Once the reservation is complete, the server returns the results to the user.

[1860] An emotion engine is also integrated into this process, analyzing the user's emotional state in real time. For example, if the system detects that the user is tired, it can send an additional message along with the booking result, such as "We recommend a relaxing hotel."

[1861] Examples:

[1862] Example 1: Travel booking

[1863] User:

[1864] The user is planning a three-day, two-night trip to Tokyo and wants to make hotel and restaurant reservations. During the chat, the emotion engine detects that the user is feeling stressed.

[1865] Operation steps:

[1866] 1. User terminal:

[1867] The user chats with the AI ​​account and says, "I'd like to book a hotel and restaurant in Tokyo for 2 nights and 3 days."

[1868] 2. Server:

[1869] The server analyzes the request and identifies the accommodation and restaurant reservation APIs.

[1870] 3. Server:

[1871] The generating AI generates a script for reservations and embeds the necessary parameters (date of stay, number of people, budget, etc.).

[1872] The emotion engine detects the user's stress level and adjusts the content of the generated script.

[1873] 4. Server:

[1874] Execute the generated script and make an API call.

[1875] 5. Server:

[1876] It receives the results of accommodation and restaurant reservations and returns them to the user with a message such as, "We recommend a relaxing hotel."

[1877] Example 2: Linking multiple AI accounts

[1878] User:

[1879] Users want to shop online and check the weather at the same time. The emotion engine recognizes that users enjoy themselves.

[1880] Operation steps:

[1881] 1. User terminal:

[1882] Users tell the AI ​​account, "Tell me today's recommended shopping and weather information."

[1883] 2. Server:

[1884] The server identifies the shopping and weather APIs and generates the necessary scripts.

[1885] The emotion engine recognizes the user's enjoyment and adapts the generated script accordingly.

[1886] 3. Server:

[1887] If necessary, add other AI accounts to the conversation and call their respective APIs.

[1888] 4. Server:

[1889] It obtains shopping and weather information and returns it to the user along with a message such as, "Today's weather is sunny. Perfect day for shopping!"

[1890] In this way, by combining generative AI and an emotion engine, the present invention can provide optimal services according to the user's emotional state, further improving user satisfaction and convenience.

[1891] The processing flow will be explained below.

[1892] This invention provides a platform where anyone can use and register an AI account using generative AI. In particular, this invention combines an emotion engine that recognizes the user's emotions to provide more personalized services.

[1893] Program Generation and Processing Description

[1894] server:

[1895] Step 1:

[1896] server:

[1897] A list of APIs provided by registrants and their usage methods are registered and managed in a database. Registrants access the server from their terminal and enter information about the APIs they provide. This includes the API name, endpoint, usage method, required input parameters, and output items. This allows the API corresponding to the service desired by the user to be saved in the database.

[1898] Step 2:

[1899] server:

[1900] It receives requests in natural language from users. It uses natural language processing technology to understand the service desired by the user and identify the appropriate API. For example, if a request such as "I would like to book a hotel and restaurant in Tokyo for 2 nights and 3 days" is received via chat, it will be analyzed.

[1901] Step 3:

[1902] server:

[1903] The emotional engine analyzes the user's emotional state from the chat content, thereby identifying the stress or joy the user is feeling.

[1904] Step 4:

[1905] server:

[1906] It uses generative AI to generate scripts to call the identified APIs, and the generated scripts are tailored to the user's emotional state—for example, if they're feeling stressed, the script will be tailored to be relaxing.

[1907] Step 5:

[1908] server:

[1909] Execute the generated script and call the API. The specified API returns data in real time, and the server receives that data.

[1910] Step 6:

[1911] server:

[1912] It then formats the data it receives based on the user's preferences and generates additional messages as needed, such as "We recommend a hotel where you can relax."

[1913] Step 7:

[1914] server:

[1915] The formatted result is sent back to the user's terminal, where the user can receive the result and select the next action.

[1916] Examples:

[1917] Example 1: Travel booking

[1918] User:

[1919] The user is planning a three-day, two-night trip to Tokyo and wants to make hotel and restaurant reservations. During the chat, the emotion engine detects that the user is feeling stressed.

[1920] Operation steps:

[1921] Step 1:

[1922] User device:

[1923] The user chats with the AI ​​account and says, "I would like to book a hotel and restaurant in Tokyo for 2 nights and 3 days."

[1924] Step 2:

[1925] server:

[1926] The server analyzes the request and identifies the accommodation and restaurant reservation APIs.

[1927] Step 3:

[1928] server:

[1929] The generating AI generates a script for reservations and embeds the necessary parameters (date of stay, number of people, budget, etc.).

[1930] Step 4:

[1931] server:

[1932] The emotion engine detects the user's stress level and adjusts the content of the generated script.

[1933] Step 5:

[1934] server:

[1935] Execute the generated script and make an API call.

[1936] Step 6:

[1937] server:

[1938] It receives hotel and restaurant booking results and formats the results with a message such as "We recommend a relaxing hotel."

[1939] Step 7:

[1940] server:

[1941] The formatted result is sent back to the user's terminal.

[1942] Example 2: Linking multiple AI accounts

[1943] User:

[1944] Users want to shop online and check the weather at the same time. The emotion engine recognizes that users enjoy themselves.

[1945] Operation steps:

[1946] Step 1:

[1947] User device:

[1948] Users tell the AI ​​account, "Tell me today's recommended shopping and weather information."

[1949] Step 2:

[1950] server:

[1951] The server identifies the shopping and weather APIs and generates the necessary scripts.

[1952] Step 3:

[1953] server:

[1954] The emotion engine recognizes the user's enjoyment and adapts the generated script accordingly.

[1955] Step 4:

[1956] server:

[1957] If necessary, add other AI accounts to the conversation and call their respective APIs.

[1958] Step 5:

[1959] server:

[1960] Get shopping and weather information and format the results with a message like "Today's weather is sunny. Great day for shopping!"

[1961] Step 6:

[1962] server:

[1963] The formatted result is sent back to the user's terminal.

[1964] In this way, by combining generative AI and an emotion engine, the present invention can provide optimal services according to the user's emotional state, further improving user satisfaction and convenience.

[1965] Example 2

[1966] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1967] Modern information systems require users to receive a variety of services quickly and in a personalized manner. However, current systems often lack the functionality to analyze the user's emotional state and reflect the results in the content of the service provided. As a result, services provided without considering the user's emotional state may reduce user satisfaction. Furthermore, when providing complex services by linking multiple APIs, there is a lack of an efficient process for identifying APIs and generating scripts. This poses the challenge of requiring a great deal of effort and time to meet user requests.

[1968] The specification process by the specification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes: means for receiving a service request from a user terminal in natural language; means for registering and managing a list of APIs provided by registrants and usage methods in a database; means for analyzing the user's request and identifying an appropriate API using natural language processing; means for generating a script for calling the API using a generative AI model; means for executing the generated script to call the API; means for returning the call result to the user; means including an emotion engine for analyzing the user's emotional state; and means for adjusting the content of the generated script depending on the user's emotional state. This makes it possible to quickly provide personalized services according to the user's emotional state and efficiently link multiple APIs.

[1969] "User terminal" means a device through which a user accesses the system and inputs service requests.

[1970] A "registrant" is a person who registers information about the API that they provide in the system.

[1971] "API" stands for Application Programming Interface, an interface for providing specific functions and data to the outside world.

[1972] A "database" is a system that stores information in an organized manner and allows it to be retrieved when needed.

[1973] "Natural language processing" is a technology that allows computers to understand and process human language.

[1974] A "generative AI model" is an algorithm that uses artificial intelligence to generate scripts for specific tasks or purposes.

[1975] A "script" is program code for automatically executing a series of instructions.

[1976] An "emotion engine" is a technology that analyzes a user's emotional state from their text or input.

[1977] "Call result" refers to the response or data obtained after executing an API.

[1978] "Natural language" refers to the language used by humans on a daily basis, translated into a form that the system can understand.

[1979] "Fusion of multiple APIs" refers to the use of multiple APIs simultaneously or sequentially to provide one or more services.

[1980] This invention is a highly personalized service provision system that utilizes a generative AI model and an emotion engine. This system has the function of calling the optimal API based on the user's wishes and emotional state and returning the results to the user.

[1981] System Overview

[1982] The system includes the following major hardware and software components:

[1983] Server: Responsible for managing the API, parsing requests, generating scripts, running the emotion engine, and returning results.

[1984] User terminal: A device (PC, smartphone, etc.) that allows a user to access the system and input service requests.

[1985] Subscriber terminal: A device used to add API information to the system.

[1986] Database (e.g. MySQL, PostgreSQL): Stores and manages registered API information.

[1987] Natural language processing engines (e.g. Google NLP API, spaCy): used to analyze user preferences.

[1988] A generative AI model (e.g., OpenAI GPT-3) used to generate scripts for API calls.

[1989] Emotion engine (e.g., Microsoft Azure Emotion API): Used to analyze the user's emotional state.

[1990] Detailed System Operation

[1991] 1. Receiving Service Requests:

[1992] The user inputs their desired service request in natural language from their device and sends it to the server. For example, a user might input, "I would like to reserve a hotel and restaurant for 2 nights and 3 days in Tokyo."

[1993] 2. Parse the request:

[1994] The server uses a natural language processing engine to analyze the request and understand the user's wishes, extracting the keywords "hotel reservation" and "restaurant reservation."

[1995] 3. Identify APIs and generate scripts:

[1996] The server identifies the APIs corresponding to "hotel reservation" and "restaurant reservation" from the database. Then, it generates a script using the generative AI model. For example, it sends a prompt such as "Please generate a script using the APIs for hotel reservation and restaurant reservation."

[1997] 4. Applying the Emotion Engine:

[1998] The server analyzes the user's chat content using an emotion engine to identify the user's emotional state. For example, if the user is feeling stressed, the emotion engine conveys that information. The server then adjusts the content of the generated script to include a message such as, "We recommend a relaxing hotel."

[1999] 5. API call and return results:

[2000] The server runs the generated script, calls the appropriate API, receives the response from the API, and returns the results to the user, including the specific reservation information and an optional message (e.g., "We recommend a relaxing hotel").

[2001] Specific examples

[2002] Book a trip

[2003] User: Planning a trip, type in chat, "I'd like to book a hotel and restaurant in Tokyo for 2 nights and 3 days."

[2004] Server: Receives the request and analyzes it using a natural language processing engine. It generates an appropriate script using a generative AI model and analyzes the user's emotions using an emotion engine. If stress is detected, it adds a soothing message to the script.

[2005] Execution and return: Call the API to execute the reservation and return the reservation results to the user terminal.

[2006] Multiple API integrations

[2007] User: Wants to shop online and check the weather at the same time, so types in "Tell me today's recommended shopping and weather information."

[2008] Server: Identifies the shopping and weather APIs and generates the necessary scripts. If the emotion engine determines that the user is having fun, it includes a message that matches this excitement.

[2009] Execution and return: Calls multiple APIs and returns the obtained shopping information and weather information to the user along with a message such as "Today's weather is sunny. It's a great day for shopping!"

[2010] In this way, by combining a generative AI model and an emotion engine, the present invention can provide optimal services according to the user's emotional state, thereby improving user satisfaction and realizing efficient and personalized service provision.

[2011] The flow of the identification process in the second embodiment will be described with reference to FIG.

[2012] Step 1: Enter and submit your service request on the user's device

[2013] Specific operation: The user uses the user terminal to input the desired service details in chat format. For example, the user might input "I would like to reserve a hotel and restaurant in Tokyo for 2 nights and 3 days."

[2014] Input: A message of service request written in natural language.

[2015] Output: The request message sent to the server.

[2016] Step 2: Server receives and analyzes the request

[2017] Specific operation: The server receives the request sent from the user terminal and analyzes the request using a natural language processing engine (e.g., Google NLP API, spaCy). Through the analysis, the keywords "hotel reservation" and "restaurant reservation" are extracted.

[2018] Input: A natural language request from the user.

[2019] Output: Parsed preference (e.g. "Hotel reservation", "Restaurant reservation").

[2020] Step 3: Identify the corresponding API

[2021] Specific operation: The server refers to the database and identifies the API corresponding to the analyzed request. For example, it retrieves the "Hotel Reservation" API and the "Restaurant Reservation" API.

[2022] Input: Parsed desired content.

[2023] Output: Identified API (e.g. "Hotel Booking" API, "Restaurant Booking" API).

[2024] Step 4: Generating scripts using generative AI models

[2025] Specific operation: The server uses a generative AI model (e.g., OpenAI GPT-3) to generate a script for API calls based on the user's request. The prompt text is "Please generate a script using the hotel reservation and restaurant reservation APIs."

[2026] Input: The API identified and what you want.

[2027] Data processing: Send prompt text to the generative AI model to generate a script for calling the API.

[2028] Output: The generated script.

[2029] Step 5: Emotional state analysis by the emotion engine

[2030] Specific operation: The server analyzes the user's chat content using an emotion engine (e.g., Microsoft Azure Emotion API) to identify the user's emotional state. For example, it detects whether the user is feeling stressed.

[2031] Input: User chat content.

[2032] Data Computation: Analyze emotional states using an emotion engine.

[2033] Output: Detected emotional state (e.g., stress).

[2034] Step 6: Adjust the script content

[2035] Specific behavior: The server adjusts the content of the generated script based on the detected emotional state. For example, if a user is feeling stressed, it might add a message such as "We recommend a relaxing hotel."

[2036] Input: Generated script and emotional state.

[2037] Data processing: The message part of the script is changed depending on the emotional state.

[2038] Output: The adjusted script.

[2039] Step 7: Making API calls

[2040] Specific operation: The server executes the tailored script, calls the specified API, and receives the response from the API.

[2041] Input: The adjusted script.

[2042] Output: The response data from the API.

[2043] Step 8: Returning the results

[2044] Specific operation: The server returns the response data from the API to the user's device and displays the result, for example, information that the reservation was successful and a message such as "We recommend a relaxing hotel."

[2045] Input: Response data from the API.

[2046] Data processing: Integrate response data and emotional responses.

[2047] Output: Results returned to the user's terminal.

[2048] (Application example 2)

[2049] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2050] Conventional automated service provision systems have difficulty in providing personalized responses that take into account the user's emotions, making it difficult to increase user satisfaction. In particular, when providing services in physical stores, flexible and appropriate responses that take into account the customer's psychological state are required.

[2051] The specification processing by the specification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for receiving a service request from a user terminal in natural language, means for registering and managing a list of APIs provided by registrants and their usage methods in a database, means for analyzing the user's request using natural language processing and identifying an appropriate API, means for generating a script for calling the API using a generation AI, means for executing the generated script and calling the API, means for returning the call result to the user, means for analyzing the user's emotional state, and means for adjusting the content of the generated script based on the emotional state. This makes it possible to provide personalized services according to the user's emotional state.

[2052] A "user terminal" is a device that allows a user to input service requests.

[2053] "Service Desired" refers to the specific service content that the user wishes to use.

[2054] "Natural language" refers to the language that humans use on a daily basis, not a specialized language such as a program language.

[2055] "API" stands for Application Programming Interface, an interface for exchanging functions and data between software programs.

[2056] A "database" is a system for efficiently storing and managing data.

[2057] "Natural language processing" is a technology that allows computers to understand and analyze natural language.

[2058] "Generative AI" is an artificial intelligence technology that generates appropriate responses and results based on input data.

[2059] A "script" is a program code that contains a series of instructions or commands.

[2060] "Emotional state" refers to the user's psychological state or mood, including emotions such as stress and joy.

[2061] "Emotion analysis" is a technology that determines a user's emotional state based on facial expressions, tone of voice, etc.

[2062] An "API call" is an operation that executes a specified API to obtain data or functionality.

[2063] This invention is a system that provides personalized customer service in brick-and-mortar stores, and is realized using smart glasses and emotion analysis sensor devices. The main components of the system are the user terminal, emotion analysis sensor, generative AI, natural language processing, and a server that manages and executes each API.

[2064] Major hardware and software used

[2065] Hardware: smart glasses (e.g., Google Glass), sensor devices for emotion analysis (e.g., Microsoft Azure Kinect)

[2066] software:

[2067] Natural language processing libraries (e.g. spaCy)

[2068] Sentiment analysis engine (e.g. Microsoft Azure Emotion API)

[2069] Generative AI systems (e.g., OpenAI GPT-4)

[2070] API management and databases (e.g. AWS, Firebase)

[2071] System Operation Overview

[2072] 1. Obtaining customer sentiment data

[2073] The smart glasses' camera and emotion-analysis sensors capture the customer's facial expressions and tone of voice in real time, allowing them to determine their emotional state on the spot. For example, if a customer is feeling stressed, that information will be detected.

[2074] 2. Emotional Data Analysis

[2075] The acquired emotional data is sent to the cloud and analyzed by an emotion analysis engine. For example, the Microsoft Azure Emotion API is used to identify emotional states such as "high stress level" or "enjoyed" based on facial expressions and voice characteristics.

[2076] 3. Proposal generation using generative AI

[2077] Based on the analysis results, the generative AI system will make optimal product suggestions. For example, if a customer is feeling stressed, the generative AI will generate a script suggesting products with a relaxing effect and display it on the smart glasses' display. The generative AI uses OpenAI GPT-4 to generate natural and appropriate suggestion messages.

[2078] 4. Implementing product proposals

[2079] The staff member checks the information displayed on the smart glasses and suggests individual products and services to the customer. The staff member then introduces specific products and encourages the customer to try them according to the generated suggestion message. For example, the staff member might say, "This aroma diffuser is a recommended product to help you relax."

[2080] 5. Feedback and Adjustments

[2081] After making a proposal, the system reanalyzes the customer's reactions (facial expressions and tone of voice) and adjusts the proposal if necessary, allowing the system to continue providing optimal service based on customer feedback.

[2082] Specific prompt examples

[2083] For example, in a brick-and-mortar scenario:

[2084] Prompt Sentence Examples

[2085] "The customer's facial expression is stressed and their voice tone is low. They are currently in the relaxation section of the store. The service they offer is product suggestions. The priority keywords are relaxation, healing, and stress relief."

[2086] In this way, the system of the present invention can provide personalized services in physical stores according to the emotional state of the user, thereby achieving high customer satisfaction.

[2087] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[2088] Step 1:

[2089] The smart glasses' camera and emotion analysis sensor are used to capture the customer's facial expressions and tone of voice in real time, and the data is acquired. The input is the customer's image and voice data, and the output is the captured emotion data. The acquired data is sent to the server.

[2090] Step 2:

[2091] The server sends the acquired emotional data to the cloud and analyzes it using an emotion analysis engine (e.g., Microsoft Azure Emotion API). The input is the captured emotional data, and the output is the analyzed emotional state (e.g., "stressed," "enjoyed," etc.). The analysis results are passed to the generative AI.

[2092] Step 3:

[2093] The server uses a generative AI (e.g., OpenAI GPT-4) to generate an appropriate product recommendation script based on the analyzed emotional state. The input is the analyzed emotional state and a prompt, and the output is the generated recommendation script. For example, a recommendation script is generated that says, "This aroma diffuser is a recommended product to help you relax."

[2094] Step 4:

[2095] The server displays the generated proposal script on the display of the smart glasses. The input is the generated proposal script, and the output is the proposal message displayed on the smart glasses. The store staff checks the display and makes a proposal to the customer.

[2096] Step 5:

[2097] The staff introduces specific products to the customer and guides them to try them out, following the suggestion message displayed on the smart glasses. The input is the suggestion message displayed on the smart glasses, and the output is the product suggestion to the customer.

[2098] Step 6:

[2099] The customer's reaction is recaptured and sent to the server for feedback. The input is image and voice data of the customer after Yori's service is provided, and the output is the recaptured emotional data. Based on the feedback, the proposal content is adjusted as needed. The server re-analyzes this new emotional data and continues to make optimal proposals.

[2100] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[2101] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[2102] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[2103] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[2104] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[2105] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[2106] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[2107] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[2108] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[2109] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[2110] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[2111] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[2112] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[2113] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[2114] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[2115] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[2116] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[2117] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[2118] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[2119] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[2120] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[2121] The following is further disclosed regarding the above embodiment.

[2122] (Claim 1)

[2123] means for receiving a service request in natural language from a user terminal;

[2124] A means to register and manage the list of APIs provided by registrants and their usage methods in a database;

[2125] A means of analyzing user preferences using natural language processing and identifying appropriate APIs;

[2126] A means for generating a script for calling the API using a generation AI;

[2127] A means to run the generated script and call the API;

[2128] means for returning the call result to the user;

[2129] A system including:

[2130] (Claim 2)

[2131] 10. The system according to claim 1, further comprising means for linking a plurality of APIs in accordance with a service desired by a user.

[2132] (Claim 3)

[2133] 10. The system of claim 1, further comprising means for calling the API and returning results through chat-style interaction with the user.

[2134] "Example 1"

[2135] (Claim 1)

[2136] means for receiving a service request in natural language from a user terminal;

[2137] A means for registering and managing a list of application programming interfaces provided by the registrant and usage methods in a data storage device;

[2138] means for analyzing user preferences using natural language processing and identifying an appropriate application programming interface;

[2139] means for generating a program for calling the application programming interface using a generative artificial intelligence;

[2140] means for executing the generated program and calling the application programming interface;

[2141] means for returning the call result to the user;

[2142] A system including:

[2143] (Claim 2)

[2144] 10. The system according to claim 1, further comprising means for coordinating a plurality of application programming interfaces in accordance with a service desired by a user.

[2145] (Claim 3)

[2146] 10. The system of claim 1, further comprising means for invoking the application programming interface and returning results through interactive interaction with a user.

[2147] "Application Example 1"

[2148] (Claim 1)

[2149] means for receiving a service request in natural language from a user terminal;

[2150] A means to register and manage the list of APIs provided by registrants and their usage methods in a database;

[2151] A means of analyzing user preferences using natural language processing and identifying appropriate APIs;

[2152] A means for generating a script for calling the API using a generation AI;

[2153] A means to run the generated script and call the API;

[2154] means for returning the call result to the user;

[2155] A means for users to receive various services such as detailed product information, inventory checks, and purchase procedures in the virtual store;

[2156] A system including:

[2157] (Claim 2)

[2158] 10. The system according to claim 1, further comprising means for linking a plurality of APIs in accordance with a service desired by a user.

[2159] (Claim 3)

[2160] 10. The system of claim 1, further comprising means for calling the API and returning results through chat-style interaction with the user.

[2161] "Example 2: Combining Emotion Engines"

[2162] (Claim 1)

[2163] means for receiving a service request in natural language from a user terminal;

[2164] A means to register and manage the list of APIs provided by registrants and their usage methods in a database;

[2165] A means of analyzing user preferences using natural language processing and identifying appropriate APIs;

[2166] A means for generating a script for calling the API using the generative AI model;

[2167] A means to run the generated script and call the API;

[2168] means for returning the call result to the user;

[2169] means including an emotion engine for analyzing an emotional state of a user;

[2170] a means for adjusting the content of the generated script in response to the emotional state of the user;

[2171] A system including:

[2172] (Claim 2)

[2173] 10. The system according to claim 1, further comprising means for linking a plurality of APIs in accordance with a service desired by a user.

[2174] (Claim 3)

[2175] 10. The system of claim 1, further comprising means for calling the API and returning results through chat-style interaction with the user.

[2176] "Application example 2 when combining emotion engines"

[2177] New Claims

[2178] (Claim 1)

[2179] means for receiving a service request in natural language from a user terminal;

[2180] A means to register and manage the list of APIs provided by registrants and their usage methods in a database;

[2181] A means of analyzing user preferences using natural language processing and identifying appropriate APIs;

[2182] A means for generating a script for calling the API using a generation AI;

[2183] A means to run the generated script and call the API;

[2184] means for returning the call result to the user;

[2185] means for analyzing the emotional state of a user;

[2186] means for adjusting the content of the generated script based on the emotional state;

[2187] A system including:

[2188] (Claim 2)

[2189] 10. The system according to claim 1, further comprising means for linking a plurality of APIs in accordance with a service desired by a user.

[2190] (Claim 3)

[2191] 10. The system of claim 1, further comprising means for calling the API and returning results through chat-style interaction with the user. [Explanation of symbols]

[2192] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. means for receiving a service request in natural language from a user terminal; A means to register and manage the list of APIs provided by registrants and their usage methods in a database, A means of analyzing user preferences using natural language processing and identifying appropriate APIs; A means for generating a script for calling the API using a generation AI; A means to run the generated script and call the API; means for returning the call result to the user; A system including:

2. 2. The system according to claim 1, further comprising means for linking a plurality of APIs in accordance with a service desired by a user.

3. The system according to claim 1, further comprising means for calling the API and returning the results through chat-style interaction with the user.

Images