System

Abstract

To provide a system.SOLUTION: A system includes: first means for collecting data on a purchase history and taste of a user and storing the data in a database; second means for generating a first prompt sentence for outputting a proposal of a product or service to the user on the basis of an analysis result of the data on the purchase history and taste of the user, which is stored in the database; third means for inputting the generated first prompt sentence into a generative AI model to generate the proposal of the product or service to the user; and fourth means for transmitting the generated proposal of the product or service to the user to a terminal of the user to present it to the user.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Now that in-store installation is no longer required, a large number of robots are returning, but there is a need for ways to utilize these robots effectively and generate revenue from usage fees. There is also a need for new ways to use robots, such as changing from touch panel-based operation to a more intuitive and natural conversational UI, automating customer service / proposals, and assisting in programming classes. [Means for solving the problem]

[0005] By utilizing the API of generative AI, we provide a means to change from touch panel-based operation to a conversational UI based on generative AI. We also provide a means to propose products and services based on user needs as a means of customer service / proposal. Furthermore, by providing a means to provide assistance for programming classes, we will achieve effective utilization of robots and increase usage fee revenue. [Brief explanation of the drawings]

[0006] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 2 is a sequence diagram showing a flow of processing in the data processing system according to the first embodiment of the first form example. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1 of Embodiment 1. [Figure 13]FIG. 10 is a sequence diagram showing a processing flow of a data processing system in a second embodiment of the second form example. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 of Embodiment Example 2. [Figure 15] FIG. 10 is a sequence diagram showing the flow of processing in a data processing system according to a third embodiment of the third embodiment. [Figure 16] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 3 of Embodiment 3. [Figure 17] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in the first embodiment of the first form example when an emotion engine is combined. [Figure 18] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1 of Form Example 1 when an emotion engine is combined. [Figure 19] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in the second embodiment of the second form example when an emotion engine is combined. [Figure 20] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 of Form Example 2 when an emotion engine is combined. [Figure 21] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in the third embodiment of the third form example when an emotion engine is combined. [Figure 22] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 3 of Form Example 3 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0007] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0008] First, the terms used in the following description will be explained.

[0009] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, the processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), or a TPU (TENSOR PROCESSING UNIT (registered trademark)).

[0010] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0011] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0012] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0013] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0014] [First embodiment]

[0015] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0016] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0017] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0018] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0019] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0020] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0021] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0022] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0023] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0024] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0025] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0026] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[0027] "Example 1"

[0028] As one embodiment of the present invention, a system utilizing a generative AI API is provided. This system enables a shift from touch panel-based operation to a conversational UI based on generative AI. Specifically, when a user issues an operation command in natural language without using a touch panel, the generative AI analyzes it and performs the appropriate operation. For example, in response to an instruction to "search for a product," the system searches a product database and presents the results to the user.

[0029] "Example 2"

[0030] In addition, as one embodiment of the present invention, a customer service / suggestion system is provided. This system proposes products and services based on the user's needs. Specifically, the system analyzes the user's purchasing history and preferences and proposes products and services based on those. For example, if the user has previously purchased sporting goods, the system proposes new sporting goods and related products.

[0031] "Example 3"

[0032] Furthermore, as one embodiment of the present invention, a system for providing assistance in programming classes is provided. This system has a function for supporting programming learning. Specifically, the system presents programming problems and checks the user's answers. The system also provides hints when the user is having trouble with a problem. For example, if the user is having trouble with a loop processing problem, the system presents the basic concept of loop processing and specific code examples.

[0033] The processing flow of each embodiment will be described below.

[0034] "Example 1"

[0035] Step 1: The user issues a command in natural language, such as "Search for a product."

[0036] Step 2: The generative AI analyzes the user's instructions. This analysis is performed using natural language processing technology.

[0037] Step 3: Based on the analysis results, the system takes appropriate action, specifically searching the product database and presenting the results to the user.

[0038] "Example 2"

[0039] Step 1: The system analyzes the user's purchasing history and preferences using machine learning techniques.

[0040] Step 2: Based on the analysis, the system will suggest products and services. For example, if the user has purchased sports equipment in the past, it will suggest new sports equipment and related products.

[0041] "Example 3"

[0042] Step 1: The system presents programming problems, which are chosen appropriately based on the user's learning progress and level of understanding.

[0043] Step 2: The user completes the assignment and submits their answer.

[0044] Step 3: The system checks the user's answer. If the answer is correct, the system moves on to the next task. If the answer is incorrect, the system provides feedback to the user.

[0045] Step 4: If the user is stuck on a task, the system will provide hints, such as explaining the basic concept of looping or providing concrete code examples.

[0046] Example 1

[0047] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0048] Conventional touch panel-based operation systems have the problem that it is difficult for users to operate intuitively, and operation is cumbersome, especially for elderly people and users who are unfamiliar with technology. In addition, when serving customers or making suggestions, it is necessary to respond quickly and accurately to user needs, but conventional systems have not been able to fully achieve this. Furthermore, even for assistance systems for programming classes, there was a need for an interface that allows users to issue operating instructions in natural language.

[0049] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0050] In this invention, the server includes: a means for the user to issue operation instructions in natural language; a means for the terminal to send the user's instructions to the generative AI's API; a means for the generative AI to analyze the user's instructions; a means for the server to receive instructions from the generative AI and search a database; a means for the server to return search results to the generative AI; a means for the generative AI to format the search results in natural language; a means for the terminal to present the search results to the user; a means for enabling a change from touch panel-based operation to a conversational UI based on the generative AI; a means for providing customer service / suggestions; and a means for providing assistance for programming classes. This allows the user to intuitively issue operation instructions in natural language, enabling quick and accurate responses. Furthermore, the accuracy of customer service and suggestions is improved, making it possible to effectively function as an assistance system for programming classes.

[0051] "User" refers to a person who uses the system to issue operational instructions.

[0052] "Terminal" refers to the device through which the user inputs operational instructions and sends them to the generative AI's API.

[0053] "Generative AI" refers to artificial intelligence that analyzes a user's instructions in natural language and generates appropriate responses and operations.

[0054] "API" stands for Application Programming Interface, and refers to an interface for exchanging functions and data between different software programs.

[0055] A "server" refers to a computer system whose role is to receive instructions from the generative AI, search a database, and return the results.

[0056] A "database" refers to a collection of data that is systematically organized and stored to facilitate easy searching and updating.

[0057] "Conversational UI" refers to a user interface in which the user issues instructions in natural language and the system responds accordingly.

[0058] "Customer service / suggestion" refers to the act of suggesting products or services based on the user's needs.

[0059] "Support for programming classes" refers to the provision of functions and services to support programming education.

[0060] This invention is a system that utilizes a generative AI API to enable a change from touch panel-based operation to a conversational UI based on generative AI. This system allows users to issue operation instructions in natural language, and the generative AI analyzes the instructions and performs appropriate responses and operations.

[0061] Hardware and software used

[0062] Hardware: Smartphones, tablets, PCs, servers

[0063] Software: Generative AI APIs (e.g., OpenAI's GPT-3), speech recognition software (e.g., Google's Speech-to-Text API), database management systems (e.g., MySQL)

[0064] System configuration

[0065] 1. The user issues instructions in natural language.

[0066] For example: "Search for a product"

[0067] 2. The device receives the user's instructions and sends them to the generative AI API.

[0068] The device converts voice input into text and sends that text to the generative AI API.

[0069] 3. The generative AI analyzes the user's instructions and determines the appropriate operation.

[0070] Generative AI analyzes the text it receives and understands the user's intent.

[0071] 4. The server receives instructions from the generative AI and searches the product database.

[0072] The server receives queries from the generative AI and searches the product database.

[0073] 5. The server sends the search results back to the generative AI.

[0074] The server sends the search results back to the generative AI.

[0075] 6. Generative AI formats the search results in natural language and returns them to the device.

[0076] Generative AI formats the results in natural language.

[0077] 7. The device presents the search results to the user.

[0078] The device presents the natural language results received from the generative AI to the user.

[0079] Specific examples

[0080] When a user commands the device to "search for products," the device converts the command into text using voice recognition software and sends it to the generative AI's API. The generative AI analyzes the command, generates a query to search the product database, and sends it to the server. The server searches the product database and returns the results to the generative AI. The generative AI formats the results in natural language and returns them to the device. Finally, the device presents the search results to the user.

[0081] Prompt Sentence Examples

[0082] "If a user instructs the AI ​​to 'search for a product,' please explain how the generative AI will analyze this and what operations it will perform."

[0083] This system allows users to issue operational instructions in natural language without using a touch panel, making operation more intuitive and efficient. It also improves the accuracy of customer service and suggestions, and can function effectively as an auxiliary system for programming classes.

[0084] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0085] Step 1:

[0086] The user issues an operation instruction in natural language.

[0087] Input: User's voice command (e.g. "Search for products")

[0088] Output: Audio data

[0089] Specific action: The user speaks into the smartphone's microphone.

[0090] Step 2:

[0091] The device sends the user's instructions to the generative AI API.

[0092] Input: Audio data

[0093] Output: Text data

[0094] Specific operation: The device uses voice recognition software (e.g., Google Speech-to-Text API) to convert voice data into text data, and then sends that text to the generative AI API as an HTTP request.

[0095] Step 3:

[0096] Generative AI analyzes user instructions.

[0097] Input: Text data (e.g., "Search for a product")

[0098] Output: Analysis results (e.g., search query)

[0099] How it works: Generative AI (e.g., OpenAI's GPT-3) analyzes text data, understands user intent, and generates search queries.

[0100] Step 4:

[0101] The server receives instructions from the generative AI and searches the product database.

[0102] Input: Search query

[0103] Output: Search result data

[0104] Specific operation: The server uses a database management system (e.g., MySQL) to execute an SQL query against the product database and retrieve search results.

[0105] Step 5:

[0106] The server sends the search results back to the generative AI.

[0107] Input: Search result data

[0108] Output: Search result data (sent to generative AI)

[0109] Specific operation: The server sends the search results in JSON format to the generative AI.

[0110] Step 6:

[0111] Generative AI formats search results in natural language.

[0112] Input: Search result data

[0113] Output: Natural language result text

[0114] Specific operation: The generative AI analyzes the search result data and formats the results in a format such as "The following products were found: Product A, Product B, Product C."

[0115] Step 7:

[0116] The terminal presents the search results to the user.

[0117] Input: Natural language result text

[0118] Output: Search results presented to the user

[0119] What happens: Your device displays search results on the screen or reads them out loud.

[0120] (Application example 1)

[0121] Next, a description will be given of Application Example 1 of Embodiment Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0122] Conventional touch panel-based operation systems have the problem that it is difficult for users to operate them intuitively, and that operation is cumbersome, especially for elderly people and users who are unfamiliar with technology. Furthermore, even systems that use voice recognition technology have difficulty accurately analyzing the user's natural language instructions and generating appropriate responses. Furthermore, there is a demand for improving the user experience by introducing a conversational UI that uses generative AI.

[0123] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0124] In this invention, the server includes means for using a generative AI API to enable a change from touch panel-based operation to a generative AI-based conversational UI, means for providing customer service / suggestions, means for providing assistance for programming education, means for converting a user's natural language instructions into text using speech recognition, means for analyzing the user's instructions and generating appropriate operations using the generative AI, and means for responding to the user with the generated operation results by voice using speech synthesis. This allows the user to issue operation instructions in natural language without using a touch panel, achieving intuitive and easy operation.

[0125] "Generative AI" is an artificial intelligence technology that analyzes user input and generates appropriate responses and operations.

[0126] "API" stands for Application Program Interface, an interface for exchanging functions and data between different software programs.

[0127] A "touch panel" is an input device that a user can directly operate using a finger or a stylus.

[0128] A "conversational UI" is a user interface that allows users and systems to interact using natural language.

[0129] "Customer service / proposals" refers to the act of proposing products and services based on the user's needs.

[0130] "Support for programming education" means providing functions and tools to support programming education.

[0131] "Speech recognition" is a technology that converts a user's voice into text.

[0132] "Natural language instructions" are instructions given to the system by the user in a language that they normally use.

[0133] "Speech synthesis" is a technology that converts text information into speech and outputs it.

[0134] A system for implementing this invention utilizes a generative AI API to enable a change from touch panel-based operation to a conversational UI based on generative AI. The system includes means for providing customer service / suggestions, means for providing assistance for programming education, means for converting a user's natural language instructions into text using speech recognition, means for analyzing the user's instructions and generating appropriate operations using generative AI, and means for responding to the user with the generated operation results by voice using speech synthesis.

[0135] Hardware and software used

[0136] Hardware: Smartphone, microphone

[0137] Software: OpenAI API, SpeechRecognition Library (registered trademark), pyttsx3 Library (registered trademark)

[0138] Data processing and calculation

[0139] Voice Recognition

[0140] The device uses speech recognition technology to convert the user's speech into text. Specifically, it uses the SpeechRecognition library to capture the user's voice input and converts it into text using Google's speech recognition service.

[0141] Use of generative AI

[0142] The server uses the API of generative AI to analyze the user's text instructions and generate appropriate operations. Specifically, it uses the OpenAI API to generate responses and operations based on the user's instructions.

[0143] Speech synthesis

[0144] The device uses speech synthesis technology to respond to the user with the generated operation results by voice. Specifically, it uses the pyttsx3 library to convert text information into speech and respond to the user.

[0145] Specific examples

[0146] When a user says "Find a new smartphone," the system works as follows:

[0147] 1. The device captures the user's speech and converts it to text using the SpeechRecognition library.

[0148] 2. The server uses the generative AI API to analyze the user's instructions and generate a list of appropriate smartphones.

[0149] 3. The device uses the pyttsx3 library to audibly respond to the user with the generated list of smartphones.

[0150] Prompt Sentence Examples

[0151] User Instructions: Find a new smartphone

[0152] Please take appropriate action.

[0153] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0154] Step 1:

[0155] The user issues a command by voice.

[0156] Input: User's voice commands

[0157] Output: Audio data

[0158] Specific action: The user speaks a natural language command such as "Find a new smartphone."

[0159] Step 2:

[0160] The device performs voice recognition and converts the voice data into text.

[0161] Input: Audio data

[0162] Output: Text data

[0163] How it works: The device uses the microphone to capture voice data and converts it into text through Google's speech recognition service using the SpeechRecognition library.

[0164] Step 3:

[0165] The server uses the generative AI API to analyze the text data and generate appropriate operations.

[0166] Input: Text data

[0167] Output: Generated operation results (e.g., a list of smartphones)

[0168] Specific operation: The server uses OpenAI's API to generate responses and operations based on the user's instructions. For example, in response to the instruction "Find a new smartphone," it generates a list of smartphones.

[0169] Step 4:

[0170] The terminal performs voice synthesis and responds to the user by voice with the generated operation results.

[0171] Input: Generated operation result (text data)

[0172] Output: Audio data

[0173] Specific operation: The device uses the pyttsx3 library to convert the generated list of smartphones into speech and respond to the user.

[0174] Step 5:

[0175] The user receives a voice response.

[0176] Input: Audio data

[0177] Output: User understanding and behavior

[0178] Specific actions: The user listens to the voice response from the device and decides on the next action, such as selecting from a list of smartphones presented.

[0179] Example 2

[0180] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0181] With conventional customer service / recommendation systems, it was difficult to effectively utilize a user's purchasing history and preferences to recommend products and services. Furthermore, it was not easy to switch from touch panel-based operation to a conversational UI based on generative AI, and there was a need to improve the user experience. Furthermore, there was a lack of support functions for programming classes.

[0182] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0183] In this invention, the server includes: means for utilizing a generative AI API to enable a change from touch panel-based operation to an interactive UI based on generative AI; means for collecting data on a user's purchase history and preferences and storing it in a database; means for analyzing data acquired from the database using a data analysis tool; means for generating product and service proposals based on the analysis results using a generative AI model; means for transmitting the generated proposals to a user terminal and presenting them to the user; and means for providing assistance for programming classes. This enables effective product and service proposals based on the user's purchase history and preferences, improving the user experience through an interactive UI and providing assistance functions for programming classes.

[0184] An "API for generative AI" is an application programming interface that allows external use of the functions of generative artificial intelligence.

[0185] "Conversational UI" is a user interface in which the user and the system communicate two-way using natural language.

[0186] "User purchase history" refers to a record of products and services purchased by a user in the past.

[0187] A "database" is a system for efficiently storing, managing, and retrieving data.

[0188] A "data analysis tool" is software used to analyze collected data and extract useful information.

[0189] A "generative AI model" is a model that uses artificial intelligence to generate text and suggestions.

[0190] A "user terminal" is a device such as a computer or smartphone that can be directly operated by a user.

[0191] "Support for programming classes" refers to functions and tools to support programming education.

[0192] This invention is a customer service / recommendation system that suggests products and services based on a user's purchasing history and preferences. The system utilizes a generative AI API, enabling the user to switch from a touch panel-based operation to a conversational UI based on generative AI. It also includes support functions for programming classes.

[0193] Hardware and software used

[0194] Hardware: Servers, user devices (PCs, smartphones, etc.)

[0195] Software: Database management systems (e.g., MySQL), generative AI models (e.g., GPT-4®), data analysis tools (e.g., Python® libraries)

[0196] Data collection

[0197] The server collects data about the user's purchasing history and preferences and stores it in a database. For example, if a user purchases running shoes, the server stores data such as "Running shoes, October 1, 2023, $100."

[0198] Data analysis

[0199] The server retrieves the user's purchasing history and preference data from the database and analyzes it using data analysis tools. Specifically, it uses a Python library to analyze the user's purchasing history and calculate the purchase frequency and trends of specific products. For example, the server analyzes the user's purchasing history over the past six months and determines that sporting goods are frequently purchased.

[0200] Proposal generation

[0201] The server uses a generative AI model (e.g., GPT-4) to generate product and service suggestions based on the analysis results. An example of a prompt to the generative AI model is, "Please suggest new sports equipment based on User A's purchasing history." An example of a generated suggestion is, "Dear User A, you have previously purchased running shoes and sportswear. We have new running shoes and the latest sportswear in stock, so please take a look."

[0202] Presenting the proposal

[0203] The server sends the generated suggestions to the user's device and presents them to the user. Specifically, the server sends the generated suggestions to the user's smartphone or PC via push notification or email. For example, the server displays a message on the user's smartphone saying, "New running shoes have arrived. Try them out!"

[0204] In this way, the system can suggest appropriate products and services based on the user's needs. The conversational UI based on generative AI also enhances the user experience. Furthermore, it also provides support functions for programming classes, making it suitable for use in educational settings.

[0205] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0206] Step 1: Data collection

[0207] The server collects data on users' purchasing history and preferences and stores it in a database.

[0208] Input: Information when the user purchases a product (product name, purchase date and time, price, etc.)

[0209] What happens: If a user buys running shoes, the server records data like "Running shoes, October 1, 2023, $100" in a database.

[0210] Output: Purchase history data stored in a database

[0211] Step 2: Data analysis

[0212] The server retrieves data on users' purchasing history and preferences from the database and analyzes it using data analysis tools.

[0213] Input: User purchase history data stored in a database

[0214] How it works: The server uses Python libraries to analyze a user's purchase history and calculate the frequency and trends of purchases of specific products. For example, it analyzes purchase history for the past six months and finds that sporting goods are frequently purchased.

[0215] Output: Analysis results (e.g., information that the user frequently purchases sports equipment)

[0216] Step 3: Proposal Generation

[0217] The server uses a generative AI model (e.g., GPT-4) to generate product and service suggestions based on the analysis results.

[0218] Input: Analysis results (user purchasing trends)

[0219] Specific operation: The server inputs a prompt sentence into the generative AI model and generates appropriate suggestions. An example of a prompt sentence is, "Please suggest new sporting goods based on user A's purchasing history."

[0220] Output: Generated suggestions (e.g. "User A, you have purchased running shoes and sportswear in the past. Please take a look at our new running shoes and latest sportswear.")

[0221] Step 4: Present your proposal

[0222] The server transmits the generated proposal to the user terminal and presents it to the user.

[0223] Input: Generated proposals

[0224] Specific operation: The server sends the generated suggestions to the user's smartphone or PC via push notification or email. For example, the user's smartphone displays a message saying, "New running shoes have arrived. Try them out!"

[0225] Output: The proposal message displayed on the user's terminal

[0226] In this way, the system can suggest appropriate products and services based on the user's needs.

[0227] (Application example 2)

[0228] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0229] Conventional customer service / recommendation systems were unable to fully utilize users' purchasing history and preferences, making it difficult to propose individually customized products and services. Furthermore, the accuracy of product recommendations based on user needs was low, making it difficult to improve user satisfaction. Furthermore, there was no established method for product recommendations based on similarity, making it difficult to propose appropriate products to users.

[0230] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0231] In this invention, the server includes a means for using the generative AI API to enable a change from touch panel-based operation to a generative AI-based interactive UI, a means for customer service / suggestion, a means for providing assistance for programming classes, a means for analyzing the user's purchase history and preferences and proposing individually customized products and services, and a means for proposing products based on similarity. This enables individually customized product suggestions based on the user's purchase history and preferences, which is expected to improve user satisfaction. Furthermore, product suggestions based on similarity enable more appropriate product suggestions.

[0232] An "API for generative AI" is an application programming interface (API) that allows external use of the functions of generative artificial intelligence (AI).

[0233] A "conversational UI" is a user interface (UI) that allows users and systems to communicate interactively using natural language.

[0234] "Means of customer service / suggestion" refers to the means of introducing products and services to users and encouraging them to purchase.

[0235] "Means for providing assistance to programming classes" are means for supporting programming education, enabling learners to learn programming efficiently.

[0236] "Means for analyzing user purchasing history and preferences" refers to the means for collecting and analyzing data on the history of products and services purchased by users in the past and their preferences.

[0237] "Means for proposing individually customized products and services" refers to means for proposing products and services optimized for each user based on the user's purchasing history and preferences.

[0238] "Means for suggesting products based on similarity" refers to a means of analyzing a user's purchasing history and preference data, calculating the similarity with other users, and suggesting products purchased by users with similar tastes.

[0239] A system for implementing this invention includes a means for utilizing a generative AI API to enable a change from touch panel-based operation to an interactive UI based on generative AI, a means for providing customer service / suggestions, a means for providing assistance in programming classes, a means for analyzing a user's purchasing history and preferences and suggesting individually customized products and services, and a means for suggesting products based on similarity.

[0240] System Program

[0241] The system is implemented using Python. Specifically, it manages user purchase history data, vectorizes the data, and calculates similarity. The part that interacts with users uses a generative AI API, employing a generative AI model such as OpenAI's GPT-3.

[0242] Program processing explanation

[0243] The server collects users' purchase history data and converts it into a data frame. Then, it combines the purchase history data as text and vectorizes it. This allows it to calculate the similarity of the purchase history between users using cosine similarity. Based on this similarity, it suggests products to specific users.

[0244] Specifically, based on the user's past purchase history of products and services, the system suggests products purchased by other users with similar tastes. For example, if a user has previously purchased "sports shoes" and "running wear," related products such as "yoga mats" and "swimming goggles" will be suggested based on the purchasing history of similar users.

[0245] Hardware and software used

[0246] Hardware: General PCs and servers

[0247] Software: Python, generative AI API (e.g., OpenAI's GPT-3)

[0248] Prompt Sentence Examples

[0249] Examples of prompts to input to a generative AI model include:

[0250] Suggest relevant products based on the user's purchasing history. For example, if the user has purchased sporting goods in the past, suggest new sporting goods and related products.

[0251] In this way, it becomes possible to make individually customized product suggestions based on the user's purchasing history and preferences, which is expected to improve user satisfaction.In addition, product suggestions based on similarity will enable more appropriate product suggestions.

[0252] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0253] Step 1:

[0254] The server collects user purchase history data. Specifically, it retrieves information such as user ID, product ID, and product name from the database and converts it into a data frame. The input is purchase history data from the database, and the output is purchase history data in data frame format.

[0255] Step 2:

[0256] The server combines the purchase history data as text. Specifically, it aggregates the purchase history data of the same user into a single text data. The input is purchase history data in data frame format, and the output is the purchase history text for each user.

[0257] Step 3:

[0258] The server vectorizes the purchase history text. Specifically, it converts the purchase history text into a TF-IDF (Term Frequency-Inverse Document Frequency) vector. The input is the purchase history text for each user, and the output is a TF-IDF vector.

[0259] Step 4:

[0260] The server calculates the cosine similarity. Specifically, it calculates the similarity of purchase histories between users using the cosine similarity. The input is a TF-IDF vector, and the output is the similarity score between users.

[0261] Step 5:

[0262] The server proposes products based on similarity. Specifically, for a specific user, it proposes products purchased by other users who are highly similar to the user. The input is the similarity score between users and their purchase history data, and the output is a list of suggested products.

[0263] Step 6:

[0264] The server uses the API of generative AI to interact with the user. Specifically, it uses a generative AI model (e.g., GPT-3) to generate an appropriate response to the user's input. The input is the user's input text and prompt sentence, and the output is the generated response text.

[0265] Step 7:

[0266] The terminal displays the suggested product list received from the server and the generated response text to the user. Specifically, the suggested product list and the dialogue content are displayed on the screen of a smartphone or PC. The input is the suggested product list from the server and the generated response text, and the output is the information displayed to the user.

[0267] Example 3

[0268] Next, a description will be given of a third embodiment of the third embodiment. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0269] In conventional programming classes, it is difficult for users to receive appropriate support when working on assignments, which often slows down learning progress, especially for beginners. Furthermore, there are limited ways for users to get instant hints when they are stuck on an assignment, which reduces learning efficiency. Furthermore, there is a lack of systems that can accurately check users' answers and provide appropriate feedback. To solve these issues, a support system that allows users to learn programming efficiently is needed.

[0270] The specific processing by the specific processing unit 290 of the data processing device 12 in the third embodiment is realized by the following means.

[0271] In this invention, the server includes a means for a user to log in, a means for the server to issue a question, a means for the user to work on the question, a means for the user to submit an answer, a means for the server to check the answer, a means for the server to provide feedback, a means for the user to request a hint, and a means for the server to provide the hint, thereby enabling a support system that allows users to efficiently learn programming.

[0272] An "API for generative AI" is an application programming interface that allows external use of the functions of generative artificial intelligence.

[0273] "Conversational UI" is a user interface that allows users and systems to communicate two-way using natural language.

[0274] "Means of customer service / suggestion" is a function for suggesting products and services based on the user's needs.

[0275] "Means for providing assistance for programming classes" refers to a function for supporting programming learning.

[0276] "Means for users to log in" refers to the function by which users access the system and perform authentication.

[0277] "Means for the server to provide assignments" refers to the function that enables the server to select programming assignments from the database and present them to the user.

[0278] "Means for users to tackle the assignment" refers to a function that allows users to create a program for the assignment presented to them.

[0279] "Means for users to submit answers" is a function that allows users to send the programs they have created to the server.

[0280] "Means for the server to check answers" refers to the function that the server uses to analyze the accuracy of the programs submitted by users.

[0281] The "means for the server to provide feedback" is a function that allows the server to provide appropriate feedback to the user based on the analysis results of the answer.

[0282] "Means for users to request hints" is a function that allows users to request hints from the server when they are having trouble with a task.

[0283] "Means for the server to provide hints" refers to a function that allows the server to use generative AI to provide appropriate hints to users.

[0284] This invention is an auxiliary system for programming classes that helps users to learn programming efficiently. The system operates in cooperation with three parties: a server, a terminal, and a user.

[0285] The server provides a web interface for users to access. Through this interface, users can complete programming challenges. The server issues the challenges, checks the answers, and provides hints. Specifically, the server does the following:

[0286] 1. Assignment: The server presents the user with programming assignments stored in the database. For example, it assigns an assignment on loop processing in Python.

[0287] 2. Checking the answers: The server receives the code submitted by the user and checks its accuracy using a Python code analysis library.

[0288] 3. Providing Hints: When a user has difficulty with a task, the server generates a hint using a generative AI model (e.g., OpenAI's GPT-3). For example, the server inputs a prompt such as "Please tell me the basic usage of for loops" into the generative AI model and presents the obtained answer to the user.

[0289] A user accesses the system using a terminal and enters their username and password on the login screen. The server receives this and authenticates them by checking the database. If authentication is successful, the server retrieves the user's progress from the database and selects the appropriate task.

[0290] As users work on the assignment, the server automatically saves their code in real time, ensuring that progress isn't lost if the user accidentally closes their browser. Once the user submits their solution, the server instantly analyzes the code and generates feedback, such as "The code is correct. All numbers were printed correctly."

[0291] When a user requests a hint, the server sends a prompt to the generative AI model saying, "Please tell me the basic usage of for loops," and provides the hint within a few seconds. Specifically, the server provides the following hints:

[0292] A for loop is used to cycle through numbers in a specified range. In this example, we'll print the numbers 1 through 10.

[0293] In this way, the server, the terminal, and the user can cooperate to efficiently advance programming learning. The flow of the specific processing in the third embodiment will be described with reference to FIG.

[0294] Step 1:

[0295] A user logs in

[0296] A user accesses the system using a terminal and enters a username and password on the login screen. The server receives this and authenticates it by checking it against the database. The input is the username and password, and the output is the authentication result. If authentication is successful, the user's progress is retrieved from the database.

[0297] Step 2:

[0298] The server issues the assignment

[0299] The server selects a programming task from the database and presents it to the user. The input is the user's progress, and the output is the selected task. For example, the task might be "Write a program that uses a Python for loop to output the numbers from 1 to 10."

[0300] Step 3:

[0301] User tackles the problem

[0302] Users work on assignments on their devices and write programs using a code editor. The input is the assignment and the output is the code written by the user. For example, you might write code like this:

[0303] python

[0304] for i in range(1, 11):

[0305] print(i)

[0306] Step 4:

[0307] User submits answer

[0308] The user submits the answer to the server from their device. When they click the submit button, the code is sent to the server. The input is the code created by the user, and the output is the code sent to the server.

[0309] Step 5:

[0310] The server checks the answer

[0311] The server analyzes the received code and checks its accuracy. It uses a Python code analysis library to do this. The input is the code submitted by the user, and the output is the analysis result. For example, it checks whether the code is error-free and whether it produces the expected output.

[0312] Step 6:

[0313] Server provides feedback

[0314] The server generates feedback based on the check results and presents it to the user. The input is the analysis result, and the output is a feedback message. For example, it provides feedback such as "The code is correct. All numbers were output correctly."

[0315] Step 7:

[0316] The user requests a hint (if necessary)

[0317] When a user has difficulty with a task, they can request a hint from the server. For example, they can input "I don't understand the loop processing." The input is the user's hint request, and the output is a hint request message.

[0318] Step 8:

[0319] The server provides a hint

[0320] The server generates hints using a generative AI model (e.g., OpenAI's GPT-3). For example, the server inputs a prompt such as "Please tell me the basic usage of a for loop" into the generative AI model, and presents the resulting answer to the user. The input is the prompt, and the output is the generated hint. Specifically, the server provides the following hints:

[0321] A for loop is used to cycle through numbers in a specified range. In this example, we'll print the numbers 1 through 10.

[0322] (Application example 3)

[0323] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0324] Previous support systems for programming classes lacked the functionality to provide appropriate hints when users were struggling with a task, resulting in reduced learning efficiency. Furthermore, when learning to program factory robots, there was a lack of support systems that allowed users to learn while actually operating the robots. This made it difficult for users to efficiently acquire programming skills.

[0325] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 3 is realized by the following means.

[0326] In this invention, the server includes a means for using the API of the generative AI to enable a change from touch panel-based operation to an interactive UI based on the generative AI, a means for providing customer service / suggestions, a means for providing assistance in programming classes, a means for supporting factory robot programming learning, a means for checking the user's answers, and a means for providing hints when the user is having trouble with a task, thereby enabling the user to efficiently acquire programming skills.

[0327] An "API for generative AI" is an application programming interface that allows external use of the functions of generative artificial intelligence.

[0328] "Touch panel-based operations" refers to all operations performed by users using a touch panel.

[0329] "Interactive UI based on generative AI" is a user interface that interacts with the user based on generative artificial intelligence.

[0330] "Means of customer service / suggestion" refers to means that have the function of suggesting products and services based on the user's needs.

[0331] "Means for providing assistance for programming classes" are means that have the function of supporting learners in programming classes.

[0332] The "means for supporting learning programming for factory robots" is a means having a support function for learning programming for robots used in factories.

[0333] "Means for checking the user's answers" refers to means that has the function of checking whether the answers entered by the user are correct.

[0334] "Means for providing hints when a user is having trouble with a task" refers to means that has the function of providing hints for solving a task when a user is having difficulty with the task.

[0335] The system for implementing this invention utilizes a generative AI API and includes means for enabling a change from touch panel-based operation to an interactive UI based on generative AI, means for providing customer service / suggestions, means for providing assistance in programming classes, means for supporting learning programming for factory robots, means for checking the user's answers, and means for providing hints when the user is having trouble with a task.

[0336] System configuration

[0337] Hardware

[0338] Server: Hosts the generative AI API and generates appropriate responses to user input.

[0339] Device: A device that a user operates, such as a smartphone, tablet, or factory robot.

[0340] Factory robot: A robot that can be used to learn programming.

[0341] software

[0342] Generative AI API: An application programming interface for externally using the functions of generative artificial intelligence.

[0343] Conversational UI: A user interface that interacts with users based on generative AI.

[0344] Programming learning support application: An application to support learning factory robot programming.

[0345] Data processing and calculation

[0346] Assignment questions

[0347] The server uses the generative AI API to present users with programming challenges, such as "making a robot move forward."

[0348] Answer check

[0349] The server receives the answers entered by the user through the terminal and executes them on the factory robot, checks whether the robot operates correctly, and provides feedback on the results to the user.

[0350] Hints provided

[0351] If the user has difficulty with a task, the server uses the generative AI API to generate an appropriate hint and provide it to the user, for example, "Use the forward command."

[0352] Specific examples

[0353] For example, if a user answers "move forward" to the task "move the robot forward," the server sends the answer to the factory robot and checks whether the robot actually moves forward. If the answer is correct, the robot moves on to the next task; if the answer is incorrect, the server provides a hint: "Use the move forward command."

[0354] Prompt Sentence Examples

[0355] Challenge: Moving the robot forward

[0356] Enter your answer: Advance

[0357] In this way, users can efficiently acquire programming skills.

[0358] The flow of the specific processing in Application Example 3 will be described with reference to FIG.

[0359] Step 1:

[0360] The server uses a generative AI API to present programming challenges to users.

[0361] Input: User's learning progress data

[0362] Data processing: Selecting appropriate tasks based on learning progress data

[0363] Output: The text of the task (e.g., "Move the robot forward")

[0364] Specific operation: The server analyzes the user's past answer history and progress, and generates a challenge of appropriate difficulty. The generated challenge is sent to the device and displayed to the user.

[0365] Step 2:

[0366] Users input their answers to the questions through the terminal.

[0367] Input: Assignment text, user answer

[0368] Data processing: Receive user answers in text format

[0369] Output: User's answer data

[0370] Specific operation: The user enters the answer in the input field on the device and presses the send button. The answer data is sent to the server.

[0371] Step 3:

[0372] The server receives the user's answer and has the factory robot execute it.

[0373] Input: User's answer data

[0374] Data processing: Converting answer data into robot control commands

[0375] Output: Robot control commands

[0376] How it works: The server analyzes the user's answers and converts them into control commands that the robot can understand. The converted commands are then sent to the factory robot.

[0377] Step 4:

[0378] The factory robot operates based on the control commands it receives.

[0379] Input: Robot control command

[0380] Data processing: Execute control commands

[0381] Output: Robot movement results

[0382] Specific operation: The factory robot executes the control command received from the server and performs the specified operation (e.g., moving forward).

[0383] Step 5:

[0384] The server checks the results of the robot's actions and provides feedback to the user.

[0385] Input: Robot action result

[0386] Data processing: Evaluation of operational results

[0387] Output: A feedback message (e.g., "Correct!" or "Incorrect. Hint: Use the forward command")

[0388] Specific behavior: The server receives the robot's behavior results and evaluates whether they meet the requirements of the task. Based on the evaluation results, it generates a feedback message for the user and sends it to the device.

[0389] Step 6:

[0390] When a user has difficulty with a task, the server uses a generative AI API to generate appropriate hints and provide them to the user.

[0391] Input: User answer data, assignment text

[0392] Data processing: Generate hints based on answer data and assignment text

[0393] Output: Hint message (e.g. "Use forward command")

[0394] Specific operation: The server analyzes the user's answer data and the task text, and generates appropriate hints using the generative AI API. The generated hints are sent to the device and displayed to the user.

[0395] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0396] "Example 1"

[0397] One embodiment of the present invention is a system that includes an emotion engine in which a generative AI API recognizes a user's emotions. This system reads emotions from the user's tone of voice, facial expressions, and choice of words, and then engages in dialogue accordingly. For example, if the AI ​​senses that the user is angry, it chooses words to apologize, and if it senses that the user is happy, it chooses words to share that joy.

[0398] "Example 2"

[0399] Another embodiment of the present invention is a customer service / suggestion method using an emotion engine. This system reads the user's emotions and suggests products and services that match those emotions. For example, if the user feels depressed, the AI ​​will suggest products and services that will cheer them up. Conversely, if the user feels happy, the AI ​​will suggest products and services that will amplify that joy.

[0400] "Example 3"

[0401] Another embodiment of the present invention is a programming classroom assistant using an emotion engine. This system reads the user's emotions and provides appropriate learning materials based on those emotions. For example, if the user feels confused, the AI ​​will provide more basic learning materials. Conversely, if the user feels confident, the AI ​​will provide more advanced learning materials.

[0402] The processing flow of each embodiment will be described below.

[0403] "Example 1"

[0404] Step 1: Through dialogue with the user, read their emotions from their tone of voice, facial expressions, choice of words, etc.

[0405] Step 2: The AI ​​adjusts the content of the dialogue based on the emotions it reads.

[0406] Step 3: For example, if the user feels angry, the AI ​​will choose words to apologize, and if the user feels happy, the AI ​​will choose words to share that joy.

[0407] "Example 2"

[0408] Step 1: Interact with the user to understand their emotions.

[0409] Step 2: The AI ​​tailors product and service recommendations based on the emotions it detects.

[0410] Step 3: For example, if the user feels depressed, the AI ​​will suggest products and services that will cheer them up; if the user feels happy, the AI ​​will suggest products and services that will amplify that joy.

[0411] "Example 3"

[0412] Step 1: Interact with the user to understand their emotions.

[0413] Step 2: The AI ​​adjusts the educational material provided based on the emotions it reads.

[0414] Step 3: For example, if the user feels confused, the AI ​​will provide more basic learning materials, and if the user feels confident, the AI ​​will provide more advanced learning materials.

[0415] Example 1

[0416] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0417] Conventional touch panel-based operation systems have the problem that it is difficult for users to operate intuitively, and operation is cumbersome, especially for elderly people and users who are unfamiliar with technology. In addition, the user experience is limited because it is not possible to have a dialogue that takes into account the user's emotions. Furthermore, there is a lack of systems that utilize voice input and emotion recognition, and more natural interaction is required.

[0418] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0419] In this invention, the server includes: a means for utilizing a generative AI API to enable a change from touch panel-based operation to an interactive user interface based on generative AI; a means for providing customer service / suggestions; a means for providing assistance for programming education; a means for collecting user voice input and sending it to the server; a means for the server to convert the voice data into text and analyze it; a means for capturing the user's facial expressions and analyzing their emotions; a means for determining and executing appropriate operations based on the analysis results; a means for generating dialogue using generative AI; and a means for presenting the results to the user. This allows the user to issue operation instructions in natural language, and the system can engage in dialogue that corresponds to the user's emotions.

[0420] "API for generative AI" refers to the application programming interface for generative artificial intelligence, an interface that allows external software and systems to use the functions of generative AI.

[0421] An "interactive user interface" is an interface that allows a user to interact with a system using natural language and to perform operations and give instructions.

[0422] "Means of customer service / suggestion" refers to the function of suggesting products and services based on the user's needs and requests and providing appropriate responses.

[0423] "Means for providing assistance for programming education" refers to means for providing functions and tools to support programming education.

[0424] "Means for collecting voice input and transmitting to a server" refers to a function for collecting the user's voice as digital data and transmitting that data to a server.

[0425] "Means for converting voice data into text and analyzing it" refers to a function for converting collected voice data into text data and analyzing the text to understand the user's intent.

[0426] "Means for capturing facial expressions and analyzing emotions" refers to a function that captures the user's facial expressions with a camera and analyzes the facial expression data to identify the user's emotions.

[0427] "Means for determining and executing appropriate operations" refers to the function of determining the operation that the system should perform based on the analysis results and then executing that operation.

[0428] "Means for generating dialogue" refers to the function of using generative AI to generate dialogue based on user input and emotions.

[0429] "Means for presenting results" refers to the system's ability to visually or audibly present the analysis results and generated dialogue to the user.

[0430] This invention is a system that utilizes a generative AI API to enable a change from touch panel-based operation to an interactive user interface based on generative AI. This system allows users to issue operation instructions in natural language, and the system can engage in dialogue according to the user's emotions.

[0431] Hardware and software used

[0432] Hardware: microphones, cameras, servers, devices (PCs, smartphones, tablets, etc.)

[0433] Software: Generative AI API (e.g., OpenAI GPT-3), emotion engine

[0434] Data processing and calculation

[0435] 1. Acquiring voice input:

[0436] The user issues a command by voice through the microphone of the terminal.

[0437] The device collects the voice data and sends it to the server.

[0438] 2. Analysis of audio data:

[0439] The server receives the voice data and converts it into text using a generative AI API.

[0440] Generative AI analyzes text data and understands the user's intent.

[0441] 3. Emotion recognition:

[0442] The device's camera captures the user's facial expressions and sends them to the emotion engine.

[0443] The emotion engine reads emotions from the user's tone of voice, facial expressions, and choice of words.

[0444] 4. Take appropriate action:

[0445] The server determines the appropriate operation based on the analysis results of the generative AI and the emotional data from the emotion engine.

[0446] For example, in response to an instruction to "search for products," the server searches the product database and presents the results to the user.

[0447] 5. Dialogue generation:

[0448] If the emotion engine determines that the user is angry, the generative AI generates an apology.

[0449] If the emotion engine determines that the user is happy, the generative AI generates words to share that joy.

[0450] 6. Presentation of results:

[0451] The terminal displays the search results and the dialogue message received from the server to the user.

[0452] Specific examples

[0453] User instruction: "Find me a new phone"

[0454] System Action:

[0455] 1. The user says, "Find a new phone."

[0456] 2. The device's microphone collects the audio and sends it to the server.

[0457] 3. The server converts the speech to text using a generative AI API.

[0458] 4. The server analyzes the text data and understands the user's intent.

[0459] 5. The device camera captures the user's facial expressions and sends them to the emotion engine.

[0460] 6. The emotion engine analyzes the user's emotions.

[0461] 7. The server searches the product database and retrieves the results.

[0462] 8. The server uses generative AI to generate a dialogue such as, "I'm glad you found the product you were looking for."

[0463] 9. The terminal displays the search results and interactive messages to the user.

[0464] Prompt Sentence Examples

[0465] Prompt: "If the user indicates they are looking for a new smartphone, search your product database and present the results. Also, if the user is happy, generate a quote to share their joy."

[0466] In this way, the system of the present invention can analyze instructions given by the user in natural language, perform appropriate operations, and generate dialogue that corresponds to the user's emotions.

[0467] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0468] Step 1:

[0469] The user issues a command by voice.

[0470] Input: User's voice command (e.g., "Find a new phone")

[0471] Output: Audio data

[0472] Specific operation: The user speaks into the device's microphone, saying, "Find a new smartphone."

[0473] Step 2:

[0474] The device collects the voice data and sends it to the server.

[0475] Input: Audio data

[0476] Output: Audio data sent to the server

[0477] Specific operation: The device uses the built-in microphone to collect the user's voice as digital voice data and transmits the data to the server.

[0478] Step 3:

[0479] The server converts the audio data into text.

[0480] Input: Audio data sent to the server

[0481] Output: Text data (e.g. "Looking for a new smartphone")

[0482] Specific operation: The server sends the received voice data to a generative AI API (e.g., OpenAI GPT-3) and converts the voice to text.

[0483] Step 4:

[0484] The server analyzes the text data and understands the user's intent.

[0485] Input: Text data

[0486] Output: User intent (e.g., product search request)

[0487] Specific operation: The server analyzes the generated text data and determines that it needs to search the product database based on the instruction "Find a new smartphone."

[0488] Step 5:

[0489] The device captures the user's facial expressions and sends them to the emotion engine.

[0490] Input: User's facial expression data

[0491] Output: Facial expression data sent to the emotion engine

[0492] What it does: The device uses its built-in camera to capture the user's facial expressions and sends that data to the emotion engine.

[0493] Step 6:

[0494] The emotion engine analyzes the user's emotions.

[0495] Input: Facial expression data sent to the emotion engine

[0496] Output: User emotion data (e.g. happy, angry)

[0497] Specific operation: The emotion engine analyzes the received facial expression data and identifies the user's emotion. For example, if the user is smiling, it is judged to be "happy," and if the user has an angry expression, it is judged to be "angry."

[0498] Step 7:

[0499] The server determines and performs the appropriate action.

[0500] Input: User intent, user emotion data

[0501] Output: Operation result (e.g. product search results)

[0502] Specific operation: The server determines the appropriate operation based on the user's intention and emotional data. For example, it searches a product database and retrieves a list of applicable smartphones.

[0503] Step 8:

[0504] The server generates the dialogue using generative AI.

[0505] Input: Operation results, user emotion data

[0506] Output: Dialogue message (e.g. "I'm glad you found the product you were looking for.")

[0507] How it works: The server uses generative AI to generate dialogue based on the user's emotions. For example, if the user is happy, it generates a response such as, "I'm glad you found the product you were looking for."

[0508] Step 9:

[0509] The terminal presents the results to the user.

[0510] Input: Operation result, dialogue message

[0511] Output: Search results and dialogue messages presented to the user

[0512] Specific operation: The device displays the search results and interactive messages received from the server to the user. For example, the device displays a list of smartphones and a message such as "We're glad you found the product you were looking for."

[0513] (Application example 1)

[0514] Next, a description will be given of Application Example 1 of Embodiment Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0515] Conventional touch panel-based operation systems are difficult for users to operate intuitively, making them particularly difficult to use for elderly people and users unfamiliar with technology. Furthermore, conventional customer service systems are unable to recognize and respond to customer emotions, making it difficult to improve customer satisfaction. Furthermore, even interactive systems using smart glasses are unable to adequately analyze voice input or generate appropriate responses, limiting their use in brick-and-mortar stores.

[0516] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0517] In this invention, the server includes a means for using the API of generative AI to enable a change from touch panel-based operation to a conversational UI based on generative AI, a means for providing customer service / proposals, a means for providing assistance for programming classes, a means for analyzing customer voice input using smart glasses and generating appropriate responses, and a means for recognizing customer emotions and engaging in dialogue accordingly, thereby enabling users to issue operational instructions in natural language and enabling appropriate responses according to the customer's emotions.

[0518] "Generative AI" refers to artificial intelligence that generates new data and information based on user input.

[0519] "API" stands for Application Programming Interface, an interface for exchanging functions and data between different software programs.

[0520] A "touch panel" is a display that can be operated directly with a finger or a stylus.

[0521] An "interactive UI" is a user interface in which the user interacts with the system in natural language.

[0522] "Customer service" refers to the act of providing products and services to customers.

[0523] "Suggestion" refers to the act of recommending products or services based on a user's needs.

[0524] A "programming classroom" is an educational facility that teaches programming knowledge and skills.

[0525] "Assistance" refers to the act of supporting a specific task or activity.

[0526] "Smart glasses" are wearable devices equipped with functions such as a display, camera, and microphone.

[0527] "Voice input" refers to the act of a user giving voice instructions to a system through a microphone.

[0528] "Analysis" refers to the act of analyzing input data and understanding its meaning.

[0529] A "response" is a response that the system gives in response to a user's input.

[0530] "Emotion recognition" is a technology that reads emotions from a user's tone of voice, facial expressions, choice of words, etc.

[0531] "Dialogue" refers to the act of communication between a user and a system.

[0532] The system for implementing this invention utilizes a generative AI API, enabling a change from touch panel-based operation to a conversational UI based on generative AI. The system includes means for providing customer service and suggestions, means for providing assistance in programming classes, means for analyzing customer voice input using smart glasses and generating appropriate responses, and means for recognizing customer emotions and engaging in dialogue accordingly.

[0533] The server uses the generative AI API to analyze the user's voice input and generate an appropriate response. Specifically, when the user issues a command through the smart glasses, the voice data is sent to the server. The server uses speech recognition software to convert the voice data into text and inputs the text into the generative AI model. The generative AI model generates an appropriate response based on the input text and presents the response to the user.

[0534] The server also uses an emotion recognition engine to read emotions from the user's tone of voice and facial expressions. The emotion recognition engine analyzes video data captured by the camera and recognizes the user's emotions. The recognized emotional information is input into a generative AI model and taken into consideration when generating responses.

[0535] The hardware used includes smart glasses, a microphone, and a camera, while the software used includes the OpenAI API, an emotion recognition engine (EmotionRecognizer), voice recognition software, and video capture software (OpenCV®).

[0536] As a concrete example, the following prompt sentence is input to the generative AI model:

[0537] "Customer question: 'How do I use this product?'

[0538] Customer sentiment: "Curious"

[0539] Please generate an appropriate response."

[0540] Based on this prompt, the generative AI model generates the following response:

[0541] "This product is very easy to use. First, turn it on, then use the included remote control to operate it. For detailed instructions, please see the manual here."

[0542] In this way, the server can generate an appropriate response based on the user's voice input and emotional information and present it to the user through the smart glasses, allowing the user to issue operational instructions in natural language and respond appropriately according to the customer's emotions.

[0543] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0544] Step 1:

[0545] The user issues voice commands through the smart glasses.

[0546] Input: User's voice commands

[0547] Output: Audio data

[0548] How it works: The user speaks a question or command into the smart glasses' microphone, which then captures the voice and stores it as audio data.

[0549] Step 2:

[0550] The terminal transmits the voice data to the server.

[0551] Input: Audio data

[0552] Output: Audio data sent to the server

[0553] Specific operation: The smart glasses transmit the captured audio data to the server via wireless communication.

[0554] Step 3:

[0555] The server uses speech recognition software to convert the voice data into text.

[0556] Input: Audio data

[0557] Output: Text data

[0558] Specific operation: The server inputs the received voice data into voice recognition software (e.g., OpenAI API) and converts the voice data into text data.

[0559] Step 4:

[0560] The server analyzes the video data captured by the camera and recognizes the user's emotions.

[0561] Input: Video data

[0562] Output: Emotional information

[0563] How it works: The camera in the smart glasses captures the user's facial expressions and sends the video data to the server, which then uses an emotion recognition engine to analyze the video data and recognize the user's emotions.

[0564] Step 5:

[0565] The server inputs text data and emotional information into a generative AI model to generate an appropriate response.

[0566] Input: Text data, emotion information

[0567] Output: Response text

[0568] How it works: The server inputs text data and emotional information as a prompt into a generative AI model (e.g., OpenAI's GPT-3). The generative AI model generates an appropriate response based on the prompt and returns the response text to the server.

[0569] Step 6:

[0570] The server sends the response text to the smart glasses.

[0571] Input: Response text

[0572] Output: Response text sent to the smart glasses

[0573] Specific operation: The server sends the generated response text to the smart glasses, and the smart glasses receive the response text.

[0574] Step 7:

[0575] The smart glasses present the response text to the user.

[0576] Input: Response text

[0577] Output: The response presented to the user

[0578] Specific operation: The smart glasses display the received response text on the display and present it to the user, who can check the response through the smart glasses display.

[0579] Example 2

[0580] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0581] Conventional customer service / recommendation systems were unable to fully utilize users' purchasing history and preferences, and were unable to make recommendations based on the user's emotions, making it difficult to provide optimal products and services to users. Furthermore, it was not easy to switch from touch panel-based operation to a conversational UI based on generative AI, and they lacked support functions for programming education. This resulted in a poor user experience and limited the system's usefulness.

[0582] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0583] In this invention, the server includes a means for using the generative AI API to enable a change from touch panel-based operation to a generative AI-based interactive UI, a means for analyzing a user's purchasing history and preferences and suggesting products and services based on the results, a means for analyzing the user's emotions and suggesting products and services that match those emotions, and a means for providing assistance for programming education. This enables optimal product and service suggestions based on the user's needs and emotions, improving the user experience. It also facilitates the change to a generative AI-based interactive UI, enhancing assistance functions for programming education.

[0584] An "API for generative AI" is an application programming interface that allows external use of the functions of generative artificial intelligence.

[0585] "Conversational UI" is a user interface that allows users and systems to communicate two-way using natural language.

[0586] "Purchase history" is a record of products and services a user has purchased in the past.

[0587] "Preferences" refer to the preferences and interests a user has for a particular category or product.

[0588] "Sentiment analysis" is the process of identifying emotions from a user's text or voice data and assessing their emotional state.

[0589] "Support for programming education" refers to functions and tools to support the learning and teaching of programming.

[0590] "Product and service proposal" is the act of selecting and presenting appropriate products and services to users based on their needs and feelings.

[0591] This invention is a system that utilizes a generative AI API to enable a change from touch panel-based operation to a generative AI-based interactive UI. It also has the ability to analyze a user's purchasing history and preferences and suggest products and services based on those, as well as the ability to analyze a user's emotions and suggest products and services that match those emotions. It also includes support functions for programming education.

[0592] Hardware and software used

[0593] Server: Database server (e.g., MySQL), AI model server (e.g., TENSORFLOW (registered trademark))

[0594] Device: User interface (e.g. web browser, mobile app)

[0595] Software: Sentiment analysis engines (e.g., IBM Watson®), recommendation systems (e.g., Apache® Mahout)

[0596] Data processing and calculation

[0597] User Data Collection

[0598] Users input their purchasing history and preferences through the device. For example, they select products they have purchased in the past or categories they are interested in. The device then transmits the information they have entered to the server in real time. Specifically, data is collected through input fields in web forms and mobile apps.

[0599] Data storage

[0600] The server stores the user data sent from the device in a database, such as a MySQL database containing information like user ID, purchase history, and preferred categories. The server establishes a database connection and executes SQL queries to insert the data.

[0601] Analyzing the data

[0602] The server analyzes the stored user data. Specifically, it uses a recommendation system such as Apache Mahout to analyze the user's purchasing patterns and preferences. A sentiment analysis engine uses IBM Watson or similar to analyze emotions from the user's text and voice data. The server retrieves the user data from the database and inputs it into an AI model. The AI ​​model analyzes the user's purchasing history and emotions and returns the results.

[0603] Product and service suggestions

[0604] The server selects appropriate products and services based on the analysis results. For example, if the user has purchased sports equipment in the past, it will suggest new sports equipment products. The emotion engine suggests products and services that match the user's emotions. For example, if the user is feeling down, it will suggest products that will cheer them up. The server integrates the results of the recommendation system and the emotion analysis engine to generate optimal suggestions.

[0605] View Suggestions

[0606] The device displays the suggestions received from the server to the user, for example, by displaying a list of suggested products or services on a web page or mobile app interface. The device receives the response from the server and renders the data in the user interface.

[0607] Examples of specific examples and prompts

[0608] Specific examples

[0609] User purchase history: If the user has purchased sporting goods in the past, we will suggest new sporting goods and related products.

[0610] Emotion-based suggestions: If the emotion engine determines that a user is feeling down, it will suggest products or services that will cheer them up. Conversely, if a user is happy, it will suggest products or services that will increase that joy.

[0611] Prompt Sentence Examples

[0612] "Based on a user's purchase history, suggest products they're likely to buy next."

[0613] "Analyze user emotions and suggest products and services that match those emotions."

[0614] In this way, the server, terminal, and user work together to realize a system that proposes optimal products and services to users.

[0615] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0616] Step 1: Collect user data

[0617] User: The user inputs information about their purchasing history and preferences through the device, for example, selecting products they have purchased in the past and categories they are interested in.

[0618] Input: Purchase history and preference information entered by the user.

[0619] Device: The device transmits user-entered information to the server in real time, specifically collecting data through input fields in web forms and mobile apps.

[0620] Output: User data sent to the server.

[0621] Step 2: Save your data

[0622] Server: The server stores the user data sent from the device in a database. For example, it stores information such as user ID, purchase history, and favorite categories in a MySQL database.

[0623] Input: User data sent from the device.

[0624] What happens: The server establishes a database connection and executes an SQL query to insert data.

[0625] Output: User data stored in the database.

[0626] Step 3: Analyze the data

[0627] Server: The server analyzes the stored user data, specifically using a recommendation system such as Apache Mahout to analyze user purchasing patterns and preferences.

[0628] Input: User data retrieved from the database.

[0629] Sentiment Analysis Engine: The server uses the sentiment analysis engine to analyze emotions from the user's text and voice data.

[0630] How it works: The server retrieves user data from the database and inputs it into the AI ​​model. The AI ​​model analyzes the user's purchasing history and emotions and returns the results.

[0631] Output: User purchasing patterns and sentiment analysis results.

[0632] Step 4: Propose your product or service

[0633] Server: The server selects appropriate products and services based on the analysis results. For example, if the user has purchased sports equipment in the past, it will suggest new sports equipment.

[0634] Input: User purchasing patterns and sentiment analysis results.

[0635] Emotion engine: The server recommends products and services that match the user's emotions. For example, if the user is feeling down, it will recommend products that will cheer them up.

[0636] Specific operation: The server integrates the results of the recommendation system and the sentiment analysis engine to generate optimal suggestions.

[0637] Output: A list of suggested products and services.

[0638] Step 5: Viewing Proposals

[0639] Device: The device displays the suggestions received from the server to the user, for example, by displaying a list of suggested products or services on a web page or mobile app interface.

[0640] Input: Proposal data received from the server.

[0641] Specific operation: The device receives the response from the server and renders the data in the user interface.

[0642] Output: The suggested products or services that are displayed to the user.

[0643] (Application example 2)

[0644] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0645] While existing customer service and recommendation systems can take into account a user's purchasing history and preferences, they are unable to analyze user emotions in real time and recommend products and services based on those emotions. Furthermore, there is a lack of effective ways to improve the shopping experience in physical stores, making it difficult to increase user satisfaction.

[0646] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0647] In this invention, the server includes a means for using a generative AI API to enable a change from touch panel-based operation to a generative AI-based interactive UI, a means for customer service / suggestion, a means for providing assistance for programming classes, a means for analyzing a user's purchase history, preferences, and emotions and suggesting products and services based on the results, and a means for suggesting products and services to the user using smart glasses. This makes it possible to analyze a user's emotions in real time and suggest products and services based on the results. It also improves the shopping experience in physical stores and increases user satisfaction.

[0648] "Generative AI" is an artificial intelligence technology that generates appropriate responses and suggestions based on user input and data.

[0649] "API" stands for Application Programming Interface, an interface for exchanging functions and data between different software programs.

[0650] An "interactive UI" is a user interface in which the user and the system perform operations and exchange information through natural dialogue.

[0651] "Means of customer service / suggestion" is a function for suggesting products and services based on the user's needs and preferences.

[0652] "Means for providing assistance to programming classes" is a function for supporting programming education.

[0653] "Purchase history" is a record of products and services a user has purchased in the past.

[0654] "Preferences" refer to the tendency of users to particularly like certain products or services.

[0655] "Emotion" refers to the user's psychological state or mood.

[0656] "Smart glasses" are wearable devices equipped with functions such as a display and camera.

[0657] "Means for suggesting products and services" is a function for suggesting appropriate products and services based on user data and emotions.

[0658] The following system configuration will be described as an embodiment of the present invention.

[0659] System Configuration

[0660] This system utilizes a generative AI API and includes a means to change from touch panel-based operation to an interactive UI based on generative AI, a means to provide customer service / suggestions, a means to provide assistance for programming classes, a means to analyze users' purchasing history, preferences, and emotions and suggest products and services based on that, and a means to suggest products and services to users using smart glasses.

[0661] Hardware and Software

[0662] Hardware: smart glasses, camera, server

[0663] Software: OpenCV (image processing library), EmotionRecognizer (emotion recognition library), RecommendationEngine (suggestion engine)

[0664] Processing flow

[0665] 1. Camera setup: The camera built into the smart glasses is used to capture the user's face.

[0666] 2. Emotion Recognition: Recognize user emotions from captured images using EmotionRecognizer.

[0667] 3. Generate Recommendations: Use the Recommendation Engine to suggest products and services based on the recognized sentiment.

[0668] 4. Displaying suggestions: Proposed products and services are displayed on the smart glasses display.

[0669] Specific examples

[0670] For example, if a user is in a sporting goods store, the app will suggest new sports shoes and related accessories based on the data of sports equipment purchased in the past. If the app detects that a user is feeling depressed, it will suggest a relaxing scented candle or a refreshing drink.

[0671] Prompt Sentence Examples

[0672] "Please suggest new related products based on the product data of the user's past purchases."

[0673] "Analyze user sentiment in real time and suggest products and services that match those sentiments."

[0674] In this way, it is possible to analyze user sentiment in real time and suggest products and services based on that sentiment, improving the shopping experience in physical stores and increasing user satisfaction.

[0675] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0676] Step 1:

[0677] The server activates the camera built into the smart glasses and captures the user's face.

[0678] Input: Camera image

[0679] Output: Captured user face image

[0680] What happens: The camera detects the user's face and captures its image data.

[0681] Step 2:

[0682] The server inputs the captured facial image into EmotionRecognizer to recognize the user's emotions.

[0683] Input: A captured image of the user's face

[0684] Output: Recognized user emotion data (e.g., happy, sad, surprised, etc.)

[0685] Specific operation: EmotionRecognizer analyzes facial images and performs data calculations to identify emotions.

[0686] Step 3:

[0687] The server inputs the recognized emotional data into the Recommendation Engine, which then compares it with the user's purchasing history and preferences to suggest products and services.

[0688] Input: Recognized user emotion data, user purchase history, user preferences

[0689] Output: A list of suggested products and services

[0690] Specific operation: The Recommendation Engine performs data calculations based on emotional data, purchasing history, and preferences to select the most suitable products and services.

[0691] Step 4:

[0692] The server displays a list of suggested goods and services on the smart glasses' display.

[0693] Input: List of proposed products and services

[0694] Output: Product and service information displayed on the smart glasses display

[0695] Specific operation: Displays product and service information as text and images on the smart glasses display.

[0696] Step 5:

[0697] The user reviews the offers displayed on the smart glasses and makes selections and purchases as needed.

[0698] Input: Product or service information displayed on smart glasses

[0699] Output: User's choice or purchase action

[0700] Specific Action: The user interacts with the smart glasses interface to select or purchase the suggested product or service.

[0701] Example 3

[0702] Next, a description will be given of a third embodiment of the third embodiment. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0703] In conventional programming classes, when users find a task difficult, it takes a lot of time and effort to select the appropriate learning materials. It is also difficult to provide appropriate feedback and learning materials according to the user's emotions, which reduces learning efficiency. Furthermore, there is a lack of systems using interactive UIs, which limits the user's learning experience.

[0704] The specific processing by the specific processing unit 290 of the data processing device 12 in the third embodiment is realized by the following means.

[0705] In this invention, the server includes a means for using the API of the generative AI to enable a change from touch panel-based operation to an interactive UI based on the generative AI, a means for providing customer service / suggestions, a means for providing assistance for programming classes, a means for checking the user's answers, a means for providing hints when the user is struggling with an assignment, and a means for reading the user's emotions using an emotion engine and providing appropriate learning materials according to those emotions. This improves the user's learning efficiency and makes it possible to provide appropriate feedback and learning materials.

[0706] A "generative AI API" is an application programming interface for generating data using artificial intelligence.

[0707] An "interactive UI" is a user interface that allows two-way communication between the user and the system using natural language.

[0708] "Means for customer service / suggestions" is a function for suggesting products and services based on the user's needs.

[0709] "Means for providing assistance for programming classes" refers to a function for supporting programming learning.

[0710] The "means for checking the user's answers" is a function for analyzing the answers to the program entered by the user and evaluating their accuracy.

[0711] The "means of providing hints when the user is having trouble with a task" is a function for providing appropriate advice or examples when the user is having difficulty with a task.

[0712] An "emotion engine" is a technology that analyzes a user's facial expressions and voice to determine their emotional state.

[0713] "Means for providing appropriate learning materials" refers to a function for selecting and providing optimal learning materials according to the user's learning situation and emotional state.

[0714] This invention is a support system for programming classes, which uses a generative AI API to realize an interactive UI and support users' learning. This system operates among three parties: a server, a terminal, and a user.

[0715] Hardware and software used

[0716] Server: Cloud computing service (e.g., AWS (registered trademark))

[0717] Device: User's PC or tablet

[0718] Generative AI model: General-purpose generative AI (e.g., GPT-4)

[0719] Emotion Engine: Emotion Analysis Software

[0720] Specific operation of the system

[0721] Assignment questions

[0722] When a user logs in, the server presents a programming assignment. The assignment is randomly selected from a collection of problems stored in a database on the server. For example, the server might display, "Please solve the following assignment. Write a program that uses a 'for' loop to output the numbers from 1 to 10."

[0723] Enter and submit your answers

[0724] The user inputs the answer to the task on the terminal. The terminal sends the input answer to the server. For example, the user inputs "for (int i = 1; i <= 10; i++) { System.out.println(i);}".

[0725] Check your answers

[0726] The server sends the received answer to the generative AI model to check its accuracy. The generative AI model analyzes the answer and determines whether it is correct or incorrect. The server then sends the check results to the device, which displays them to the user. For example, the server might display "The answer is correct" or "There is an error in the answer."

[0727] Providing hints

[0728] When a user has difficulty with a task, the device sends a "hint request" request to the server. The server uses a generative AI model to generate an appropriate hint and sends it to the device. The device then displays the hint to the user. For example, the server might display "The basic syntax of a 'for' loop is as follows: for (initialization; condition; update) { / / processing}."

[0729] Reading emotions

[0730] The device analyzes the user's facial expressions and voice using an emotion engine and sends the emotion data to the server. The server analyzes the emotion data and determines whether the user is confused or confident. For example, the device sends the emotion data saying "the user is confused."

[0731] Provision of teaching materials

[0732] The server selects appropriate learning materials according to the user's emotions. The server sends the selected learning materials to the terminal, and the terminal displays the learning materials to the user. For example, the server displays, "We sense that the user is confused, so we will provide learning materials on basic loop processing."

[0733] Prompt Sentence Examples

[0734] Assignment questions

[0735] "Give the user the following programming assignment: Write a program that uses a 'for' loop to print the numbers 1 through 10."

[0736] Providing hints

[0737] "If a user is struggling with a looping problem, please provide them with basic concepts and concrete code examples."

[0738] Providing emotionally relevant learning materials

[0739] "If you feel the user is confused, provide basic learning materials. Conversely, if you feel the user is confident, provide more advanced learning materials." The flow of the identification process in Example 3 will be explained using Figure 21.

[0740] Step 1:

[0741] User Login

[0742] A user logs into the system using a terminal. The terminal sends the user's authentication information (username and password) to the server. The server checks the received authentication information against a database, and if authentication is successful, returns a login success message to the user. The input is the user's authentication information, and the output is a login success or failure message.

[0743] Step 2:

[0744] Assignment questions

[0745] The server presents programming assignments to logged-in users. The server randomly selects an assignment from the database and sends it to the terminal. The terminal then displays the received assignment to the user. The input is the user's login information, and the output is the content of the assignment. Specifically, the server displays the message, "Please solve the following assignment. Write a program that uses a 'for' loop to output the numbers from 1 to 10."

[0746] Step 3:

[0747] Enter and submit your answers

[0748] The user inputs the answer to the task on the terminal. The terminal sends the input answer to the server. The input is the user's answer, and the output is the sending of the answer to the server. Specifically, the user inputs "for (int i = 1; i <= 10; i++) { System.out.println(i);}".

[0749] Step 4:

[0750] Check your answers

[0751] The server sends the received answer to the generative AI model to check its accuracy. The generative AI model analyzes the answer and determines whether it is correct or incorrect. The server sends the check results to the terminal, which displays them to the user. The input is the user's answer, and the output is a judgment of whether the answer is correct or incorrect. Specifically, the server displays "The answer is correct" or "There is an error in the answer."

[0752] Step 5:

[0753] Providing hints

[0754] When a user has trouble with a task, the device sends a "hint request" request to the server. The server uses a generative AI model to generate an appropriate hint and sends it to the device. The device then displays the hint to the user. The input is the user's request, and the output is the content of the hint. In concrete terms, the server displays "The basic syntax of a 'for' loop is as follows: for (initialization; condition; update) { / / processing}".

[0755] Step 6:

[0756] Reading emotions

[0757] The device analyzes the user's facial expressions and voice using an emotion engine and sends the emotion data to the server. The server analyzes the emotion data and determines whether the user is confused or confident. The input is the user's facial expressions and voice, and the output is emotion data. Specifically, the device sends emotion data stating that "the user is confused."

[0758] Step 7:

[0759] Provision of teaching materials

[0760] The server selects appropriate learning materials according to the user's emotions. The server sends the selected learning materials to the terminal, which then displays the learning materials to the user. The input is emotional data, and the output is appropriate learning materials. Specifically, the server displays the message, "We sense that the user is confused, so we will provide learning materials on basic loop processing."

[0761] (Application example 3)

[0762] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0763] Conventional programming education systems have difficulty providing appropriate learning materials based on the user's emotions and level of understanding, making it difficult to meet individual needs, especially in specialized training for factory robot operators and engineers. Furthermore, touch panel-based operation makes it impossible to realize an intuitive, interactive user interface, preventing users from improving their learning efficiency. To solve these issues, a system combining a conversational UI utilizing generative AI with emotion analysis capabilities is needed.

[0764] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 3 is realized by the following means.

[0765] In this invention, the server includes a means for using the API of the generative AI to enable a change from touch panel-based operation to an interactive UI based on the generative AI, a means for providing customer service / suggestions, a means for providing assistance for programming classes, a means for analyzing the user's emotions and providing appropriate learning materials according to those emotions, and a means for supporting programming education for factory robot operators and engineers. This makes it possible to provide appropriate learning materials according to the user's emotions and level of understanding, thereby realizing effective learning support that meets the individual needs of factory robot operators and engineers, particularly in specialized education.

[0766] "Generative AI" is a system that uses artificial intelligence technology to generate new data and information.

[0767] "API" stands for Application Programming Interface, an interface for exchanging functions and data between different software programs.

[0768] An "interactive UI" is a user interface in which the user and the system perform operations and exchange information through natural dialogue.

[0769] "Means for customer service / suggestions" refers to a system that has the function of suggesting products and services based on the user's needs.

[0770] "Means for providing assistance for programming classes" refers to a system with functions to support learning programming.

[0771] "Means for analyzing a user's emotions and providing appropriate learning materials in accordance with those emotions" refers to a system that has the function of analyzing a user's emotions and providing appropriate learning materials based on the results.

[0772] "A means for supporting programming education for factory robot operators and engineers" is a system that has the function of supporting factory robot operators and engineers when they learn programming.

[0773] The system for implementing this invention includes a means for utilizing a generative AI API to enable a change from touch panel-based operation to an interactive UI based on generative AI, a means for providing customer service / suggestions, a means for providing assistance in programming classes, a means for analyzing user emotions and providing appropriate teaching materials according to those emotions, and a means for supporting programming education for factory robot operators and engineers.

[0774] The server uses a generative AI API to generate appropriate responses to user input. This allows for a shift from touchscreen-based operation to a conversational UI. Specifically, when a user is working on a programming challenge, the server analyzes the user's input and provides appropriate hints and learning materials.

[0775] The terminals are devices such as smartphones and head-mounted displays, allowing users to learn programming in the actual work environment of a factory. The terminals are equipped with cameras and microphones to analyze the user's emotions and send this data to a server.

[0776] To analyze user emotions, the system uses natural language processing (NLP) technology. Specifically, it uses Python and NLP libraries to analyze the user's input text and voice data. Based on the analysis results, the server classifies the user's emotions into categories such as "confused," "confident," or "neutral," and provides appropriate learning materials accordingly.

[0777] As a concrete example, consider a factory robot operator working on a loop processing problem. If the user types "I don't understand the loop processing," the server interprets the user's emotion as "confused" and provides the following hints:

[0778] Let's review some basic concepts of looping. For example, a for loop repeats a specific number of times. Here's a simple example:

[0779] python

[0780] for i in range(5):

[0781] print(i)

[0782] This code will output the numbers 0 through 4.

[0783] Example prompts to input to a generative AI model:

[0784] "When users are struggling with looping challenges, provide them with basic concepts and concrete code examples."

[0785] In this way, it becomes possible to provide appropriate learning materials according to the user's emotions and level of understanding, enabling effective learning support that meets individual needs, particularly in specialized education for factory robot operators and engineers.

[0786] The flow of the specific processing in Application Example 3 will be described with reference to FIG.

[0787] Step 1:

[0788] A user uses a terminal to work on a programming assignment.

[0789] Input: User input for the problem (e.g., "I don't understand loop processing")

[0790] Output: User input data

[0791] Specific actions: The user uses a smartphone or head-mounted display to input questions or difficulties regarding the programming task.

[0792] Step 2:

[0793] The terminal transmits the user's input data to the server.

[0794] Input: User-entered data

[0795] Output: User input data sent to the server

[0796] Specific operation: The terminal collects the user's input data and sends it to a server via the Internet.

[0797] Step 3:

[0798] The server receives the user's input data and analyzes it using a generative AI model.

[0799] Input: User-entered data

[0800] Output: Analysis result (e.g. "confused")

[0801] How it works: The server uses a generative AI model (e.g., a natural language processing library) to analyze the user's input data and classify the user's emotions into categories such as "confused," "confident," or "neutral."

[0802] Step 4:

[0803] The server generates appropriate teaching materials based on the analysis results.

[0804] Input: Analysis result (e.g. "confused")

[0805] Output: Appropriate teaching materials (e.g., basic concepts of loop processing and concrete code examples)

[0806] Specific operation: The server uses the generative AI model to generate learning materials appropriate for the user based on the analysis results. For example, if the user is analyzed as "confused," it will provide basic concepts and specific code examples.

[0807] Step 5:

[0808] The server sends the generated teaching materials to the terminal.

[0809] Input: Generated teaching materials

[0810] Output: The teaching materials sent to the device

[0811] Specific operation: The teaching materials generated by the server are sent to the terminal via the Internet.

[0812] Step 6:

[0813] The terminal displays the received educational material to the user.

[0814] Input: Teaching materials sent from the server

[0815] Output: The material displayed to the user

[0816] Specific operation: The terminal displays the received learning material to the user, allowing the user to study it.

[0817] Step 7:

[0818] The user takes on a new challenge or enters an additional question.

[0819] Input: New input data from the user

[0820] Output: New input data

[0821] Specific Action: The user attempts a new task or enters an additional question and returns to step 1.

[0822] The above processing steps make it possible to provide appropriate learning materials according to the user's emotions and level of understanding, thereby enabling effective learning support that meets individual needs, particularly in specialized education for factory robot operators and engineers.

[0823] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0824] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0825] Another example of generative AI is Gemini (registered trademark) (Internet search engine). <url: https: gemini.google.com ?hl="ja">) are mentioned.

[0826] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0827] [Second embodiment]

[0828] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0829] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0830] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0831] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0832] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0833] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0834] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0835] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0836] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0837] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0838] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0839] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[0840] "Example 1"

[0841] As one embodiment of the present invention, a system utilizing a generative AI API is provided. This system enables a shift from touch panel-based operation to a conversational UI based on generative AI. Specifically, when a user issues an operation command in natural language without using a touch panel, the generative AI analyzes it and performs the appropriate operation. For example, in response to an instruction to "search for a product," the system searches a product database and presents the results to the user.

[0842] "Example 2"

[0843] In addition, as one embodiment of the present invention, a customer service / suggestion system is provided. This system proposes products and services based on the user's needs. Specifically, the system analyzes the user's purchasing history and preferences and proposes products and services based on those. For example, if the user has previously purchased sporting goods, the system proposes new sporting goods and related products.

[0844] "Example 3"

[0845] Furthermore, as one embodiment of the present invention, a system for providing assistance in programming classes is provided. This system has a function for supporting programming learning. Specifically, the system presents programming problems and checks the user's answers. The system also provides hints when the user is having trouble with a problem. For example, if the user is having trouble with a loop processing problem, the system presents the basic concept of loop processing and specific code examples.

[0846] The processing flow of each embodiment will be described below.

[0847] "Example 1"

[0848] Step 1: The user issues a command in natural language, such as "Search for a product."

[0849] Step 2: The generative AI analyzes the user's instructions. This analysis is performed using natural language processing technology.

[0850] Step 3: Based on the analysis results, the system takes appropriate action, specifically searching the product database and presenting the results to the user.

[0851] "Example 2"

[0852] Step 1: The system analyzes the user's purchasing history and preferences using machine learning techniques.

[0853] Step 2: Based on the analysis, the system will suggest products and services. For example, if the user has purchased sports equipment in the past, it will suggest new sports equipment and related products.

[0854] "Example 3"

[0855] Step 1: The system presents programming problems, which are chosen appropriately based on the user's learning progress and level of understanding.

[0856] Step 2: The user completes the assignment and submits their answer.

[0857] Step 3: The system checks the user's answer. If the answer is correct, the system moves on to the next task. If the answer is incorrect, the system provides feedback to the user.

[0858] Step 4: If the user is stuck on a task, the system will provide hints, such as explaining the basic concept of looping or providing concrete code examples.

[0859] Example 1

[0860] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0861] Conventional touch panel-based operation systems have the problem that it is difficult for users to operate intuitively, and operation is cumbersome, especially for elderly people and users who are unfamiliar with technology. In addition, when serving customers or making suggestions, it is necessary to respond quickly and accurately to user needs, but conventional systems have not been able to fully achieve this. Furthermore, even for assistance systems for programming classes, there was a need for an interface that allows users to issue operating instructions in natural language.

[0862] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0863] In this invention, the server includes: a means for the user to issue operation instructions in natural language; a means for the terminal to send the user's instructions to the generative AI's API; a means for the generative AI to analyze the user's instructions; a means for the server to receive instructions from the generative AI and search a database; a means for the server to return search results to the generative AI; a means for the generative AI to format the search results in natural language; a means for the terminal to present the search results to the user; a means for enabling a change from touch panel-based operation to a conversational UI based on the generative AI; a means for providing customer service / suggestions; and a means for providing assistance for programming classes. This allows the user to intuitively issue operation instructions in natural language, enabling quick and accurate responses. Furthermore, the accuracy of customer service and suggestions is improved, making it possible to effectively function as an assistance system for programming classes.

[0864] "User" refers to a person who uses the system to issue operational instructions.

[0865] "Terminal" refers to the device through which the user inputs operational instructions and sends them to the generative AI's API.

[0866] "Generative AI" refers to artificial intelligence that analyzes a user's instructions in natural language and generates appropriate responses and operations.

[0867] "API" stands for Application Programming Interface, and refers to an interface for exchanging functions and data between different software programs.

[0868] A "server" refers to a computer system whose role is to receive instructions from the generative AI, search a database, and return the results.

[0869] A "database" refers to a collection of data that is systematically organized and stored to facilitate easy searching and updating.

[0870] "Conversational UI" refers to a user interface in which the user issues instructions in natural language and the system responds accordingly.

[0871] "Customer service / suggestion" refers to the act of suggesting products or services based on the user's needs.

[0872] "Support for programming classes" refers to the provision of functions and services to support programming education.

[0873] This invention is a system that utilizes a generative AI API to enable a change from touch panel-based operation to a conversational UI based on generative AI. This system allows users to issue operation instructions in natural language, and the generative AI analyzes the instructions and performs appropriate responses and operations.

[0874] Hardware and software used

[0875] Hardware: Smartphones, tablets, PCs, servers

[0876] Software: Generative AI APIs (e.g., OpenAI's GPT-3), speech recognition software (e.g., Google Speech-to-Text API), database management systems (e.g., MySQL)

[0877] System configuration

[0878] 1. The user issues instructions in natural language.

[0879] For example: "Search for a product"

[0880] 2. The device receives the user's instructions and sends them to the generative AI API.

[0881] The device converts voice input into text and sends that text to the generative AI API.

[0882] 3. The generative AI analyzes the user's instructions and determines the appropriate operation.

[0883] Generative AI analyzes the text it receives and understands the user's intent.

[0884] 4. The server receives instructions from the generative AI and searches the product database.

[0885] The server receives queries from the generative AI and searches the product database.

[0886] 5. The server sends the search results back to the generative AI.

[0887] The server sends the search results back to the generative AI.

[0888] 6. Generative AI formats the search results in natural language and returns them to the device.

[0889] Generative AI formats the results in natural language.

[0890] 7. The device presents the search results to the user.

[0891] The device presents the natural language results received from the generative AI to the user.

[0892] Specific examples

[0893] When a user commands the device to "search for products," the device converts the command into text using voice recognition software and sends it to the generative AI's API. The generative AI analyzes the command, generates a query to search the product database, and sends it to the server. The server searches the product database and returns the results to the generative AI. The generative AI formats the results in natural language and returns them to the device. Finally, the device presents the search results to the user.

[0894] Prompt Sentence Examples

[0895] "If a user instructs the AI ​​to 'search for a product,' please explain how the generative AI will analyze this and what operations it will perform."

[0896] This system allows users to issue operational instructions in natural language without using a touch panel, making operation more intuitive and efficient. It also improves the accuracy of customer service and suggestions, and can function effectively as an auxiliary system for programming classes.

[0897] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0898] Step 1:

[0899] The user issues an operation instruction in natural language.

[0900] Input: User's voice command (e.g. "Search for products")

[0901] Output: Audio data

[0902] Specific action: The user speaks into the smartphone's microphone.

[0903] Step 2:

[0904] The device sends the user's instructions to the generative AI API.

[0905] Input: Audio data

[0906] Output: Text data

[0907] Specific operation: The device uses voice recognition software (e.g., Google Speech-to-Text API) to convert voice data into text data, and then sends that text to the generative AI API as an HTTP request.

[0908] Step 3:

[0909] Generative AI analyzes user instructions.

[0910] Input: Text data (e.g., "Search for a product")

[0911] Output: Analysis results (e.g., search query)

[0912] How it works: Generative AI (e.g., OpenAI's GPT-3) analyzes text data, understands user intent, and generates search queries.

[0913] Step 4:

[0914] The server receives instructions from the generative AI and searches the product database.

[0915] Input: Search query

[0916] Output: Search result data

[0917] Specific operation: The server uses a database management system (e.g., MySQL) to execute an SQL query against the product database and retrieve search results.

[0918] Step 5:

[0919] The server sends the search results back to the generative AI.

[0920] Input: Search result data

[0921] Output: Search result data (sent to generative AI)

[0922] Specific operation: The server sends the search results in JSON format to the generative AI.

[0923] Step 6:

[0924] Generative AI formats search results in natural language.

[0925] Input: Search result data

[0926] Output: Natural language result text

[0927] Specific operation: The generative AI analyzes the search result data and formats the results in a format such as "The following products were found: Product A, Product B, Product C."

[0928] Step 7:

[0929] The terminal presents the search results to the user.

[0930] Input: Natural language result text

[0931] Output: Search results presented to the user

[0932] What happens: Your device displays search results on the screen or reads them out loud.

[0933] (Application example 1)

[0934] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0935] Conventional touch panel-based operation systems have the problem that it is difficult for users to operate them intuitively, and that operation is cumbersome, especially for elderly people and users who are unfamiliar with technology. Furthermore, even systems that use voice recognition technology have difficulty accurately analyzing the user's natural language instructions and generating appropriate responses. Furthermore, there is a demand for improving the user experience by introducing a conversational UI that uses generative AI.

[0936] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0937] In this invention, the server includes means for using a generative AI API to enable a change from touch panel-based operation to a generative AI-based conversational UI, means for providing customer service / suggestions, means for providing assistance for programming education, means for converting a user's natural language instructions into text using speech recognition, means for analyzing the user's instructions and generating appropriate operations using the generative AI, and means for responding to the user with the generated operation results by voice using speech synthesis. This allows the user to issue operation instructions in natural language without using a touch panel, achieving intuitive and easy operation.

[0938] "Generative AI" is an artificial intelligence technology that analyzes user input and generates appropriate responses and operations.

[0939] "API" stands for Application Program Interface, an interface for exchanging functions and data between different software programs.

[0940] A "touch panel" is an input device that a user can directly operate using a finger or a stylus.

[0941] A "conversational UI" is a user interface that allows users and systems to interact using natural language.

[0942] "Customer service / proposals" refers to the act of proposing products and services based on the user's needs.

[0943] "Support for programming education" means providing functions and tools to support programming education.

[0944] "Speech recognition" is a technology that converts a user's voice into text.

[0945] "Natural language instructions" are instructions given to the system by the user in a language that they normally use.

[0946] "Speech synthesis" is a technology that converts text information into speech and outputs it.

[0947] A system for implementing this invention utilizes a generative AI API to enable a change from touch panel-based operation to a conversational UI based on generative AI. The system includes means for providing customer service / suggestions, means for providing assistance for programming education, means for converting a user's natural language instructions into text using speech recognition, means for analyzing the user's instructions and generating appropriate operations using generative AI, and means for responding to the user with the generated operation results by voice using speech synthesis.

[0948] Hardware and software used

[0949] Hardware: Smartphone, microphone

[0950] Software: OpenAI API, SpeechRecognition library, pyttsx3 library

[0951] Data processing and calculation

[0952] Voice Recognition

[0953] The device uses speech recognition technology to convert the user's speech into text. Specifically, it uses the SpeechRecognition library to capture the user's voice input and converts it into text using Google's speech recognition service.

[0954] Use of generative AI

[0955] The server uses the API of generative AI to analyze the user's text instructions and generate appropriate operations. Specifically, it uses the OpenAI API to generate responses and operations based on the user's instructions.

[0956] Speech synthesis

[0957] The device uses speech synthesis technology to respond to the user with the generated operation results by voice. Specifically, it uses the pyttsx3 library to convert text information into speech and respond to the user.

[0958] Specific examples

[0959] When a user says "Find a new smartphone," the system works as follows:

[0960] 1. The device captures the user's speech and converts it to text using the SpeechRecognition library.

[0961] 2. The server uses the generative AI API to analyze the user's instructions and generate a list of appropriate smartphones.

[0962] 3. The device uses the pyttsx3 library to audibly respond to the user with the generated list of smartphones.

[0963] Prompt Sentence Examples

[0964] User Instructions: Find a new smartphone

[0965] Please take appropriate action.

[0966] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0967] Step 1:

[0968] The user issues a command by voice.

[0969] Input: User's voice commands

[0970] Output: Audio data

[0971] Specific action: The user speaks a natural language command such as "Find a new smartphone."

[0972] Step 2:

[0973] The device performs voice recognition and converts the voice data into text.

[0974] Input: Audio data

[0975] Output: Text data

[0976] How it works: The device uses the microphone to capture voice data and converts it into text through Google's speech recognition service using the SpeechRecognition library.

[0977] Step 3:

[0978] The server uses the generative AI API to analyze the text data and generate appropriate operations.

[0979] Input: Text data

[0980] Output: Generated operation results (e.g., a list of smartphones)

[0981] Specific operation: The server uses OpenAI's API to generate responses and operations based on the user's instructions. For example, in response to the instruction "Find a new smartphone," it generates a list of smartphones.

[0982] Step 4:

[0983] The terminal performs voice synthesis and responds to the user by voice with the generated operation results.

[0984] Input: Generated operation result (text data)

[0985] Output: Audio data

[0986] Specific operation: The device uses the pyttsx3 library to convert the generated list of smartphones into speech and respond to the user.

[0987] Step 5:

[0988] The user receives a voice response.

[0989] Input: Audio data

[0990] Output: User understanding and behavior

[0991] Specific actions: The user listens to the voice response from the device and decides on the next action, such as selecting from a list of smartphones presented.

[0992] Example 2

[0993] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0994] With conventional customer service / recommendation systems, it was difficult to effectively utilize a user's purchasing history and preferences to recommend products and services. Furthermore, it was not easy to switch from touch panel-based operation to a conversational UI based on generative AI, and there was a need to improve the user experience. Furthermore, there was a lack of support functions for programming classes.

[0995] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0996] In this invention, the server includes: means for utilizing a generative AI API to enable a change from touch panel-based operation to an interactive UI based on generative AI; means for collecting data on a user's purchase history and preferences and storing it in a database; means for analyzing data acquired from the database using a data analysis tool; means for generating product and service proposals based on the analysis results using a generative AI model; means for transmitting the generated proposals to a user terminal and presenting them to the user; and means for providing assistance for programming classes. This enables effective product and service proposals based on the user's purchase history and preferences, improving the user experience through an interactive UI and providing assistance functions for programming classes.

[0997] An "API for generative AI" is an application programming interface that allows external use of the functions of generative artificial intelligence.

[0998] "Conversational UI" is a user interface in which the user and the system communicate two-way using natural language.

[0999] "User purchase history" refers to a record of products and services purchased by a user in the past.

[1000] A "database" is a system for efficiently storing, managing, and retrieving data.

[1001] A "data analysis tool" is software used to analyze collected data and extract useful information.

[1002] A "generative AI model" is a model that uses artificial intelligence to generate text and suggestions.

[1003] A "user terminal" is a device such as a computer or smartphone that can be directly operated by a user.

[1004] "Support for programming classes" refers to functions and tools to support programming education.

[1005] This invention is a customer service / recommendation system that suggests products and services based on a user's purchasing history and preferences. The system utilizes a generative AI API, enabling the user to switch from a touch panel-based operation to a conversational UI based on generative AI. It also includes support functions for programming classes.

[1006] Hardware and software used

[1007] Hardware: Servers, user devices (PCs, smartphones, etc.)

[1008] Software: Database management systems (e.g., MySQL), generative AI models (e.g., GPT-4), data analysis tools (e.g., Python libraries)

[1009] Data collection

[1010] The server collects data about the user's purchasing history and preferences and stores it in a database. For example, if a user purchases running shoes, the server stores data such as "Running shoes, October 1, 2023, $100."

[1011] Data analysis

[1012] The server retrieves the user's purchasing history and preference data from the database and analyzes it using data analysis tools. Specifically, it uses a Python library to analyze the user's purchasing history and calculate the purchase frequency and trends of specific products. For example, the server analyzes the user's purchasing history over the past six months and determines that sporting goods are frequently purchased.

[1013] Proposal generation

[1014] The server uses a generative AI model (e.g., GPT-4) to generate product and service suggestions based on the analysis results. An example of a prompt to the generative AI model is, "Please suggest new sports equipment based on User A's purchasing history." An example of a generated suggestion is, "Dear User A, you have previously purchased running shoes and sportswear. We have new running shoes and the latest sportswear in stock, so please take a look."

[1015] Presenting the proposal

[1016] The server sends the generated suggestions to the user's device and presents them to the user. Specifically, the server sends the generated suggestions to the user's smartphone or PC via push notification or email. For example, the server displays a message on the user's smartphone saying, "New running shoes have arrived. Try them out!"

[1017] In this way, the system can suggest appropriate products and services based on the user's needs. The conversational UI based on generative AI also enhances the user experience. Furthermore, it also provides support functions for programming classes, making it suitable for use in educational settings.

[1018] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1019] Step 1: Data collection

[1020] The server collects data on users' purchasing history and preferences and stores it in a database.

[1021] Input: Information when the user purchases a product (product name, purchase date and time, price, etc.)

[1022] What happens: If a user buys running shoes, the server records data like "Running shoes, October 1, 2023, $100" in a database.

[1023] Output: Purchase history data stored in a database

[1024] Step 2: Data analysis

[1025] The server retrieves data on users' purchasing history and preferences from the database and analyzes it using data analysis tools.

[1026] Input: User purchase history data stored in a database

[1027] How it works: The server uses Python libraries to analyze a user's purchase history and calculate the frequency and trends of purchases of specific products. For example, it analyzes purchase history for the past six months and finds that sporting goods are frequently purchased.

[1028] Output: Analysis results (e.g., information that the user frequently purchases sports equipment)

[1029] Step 3: Proposal Generation

[1030] The server uses a generative AI model (e.g., GPT-4) to generate product and service suggestions based on the analysis results.

[1031] Input: Analysis results (user purchasing trends)

[1032] Specific operation: The server inputs a prompt sentence into the generative AI model and generates appropriate suggestions. An example of a prompt sentence is, "Please suggest new sporting goods based on user A's purchasing history."

[1033] Output: Generated suggestions (e.g. "User A, you have purchased running shoes and sportswear in the past. Please take a look at our new running shoes and latest sportswear.")

[1034] Step 4: Present your proposal

[1035] The server transmits the generated proposal to the user terminal and presents it to the user.

[1036] Input: Generated proposals

[1037] Specific operation: The server sends the generated suggestions to the user's smartphone or PC via push notification or email. For example, the user's smartphone displays a message saying, "New running shoes have arrived. Try them out!"

[1038] Output: The proposal message displayed on the user's terminal

[1039] In this way, the system can suggest appropriate products and services based on the user's needs.

[1040] (Application example 2)

[1041] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1042] Conventional customer service / recommendation systems were unable to fully utilize users' purchasing history and preferences, making it difficult to propose individually customized products and services. Furthermore, the accuracy of product recommendations based on user needs was low, making it difficult to improve user satisfaction. Furthermore, there was no established method for product recommendations based on similarity, making it difficult to propose appropriate products to users.

[1043] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1044] In this invention, the server includes a means for using the generative AI API to enable a change from touch panel-based operation to a generative AI-based interactive UI, a means for customer service / suggestion, a means for providing assistance for programming classes, a means for analyzing the user's purchase history and preferences and proposing individually customized products and services, and a means for proposing products based on similarity. This enables individually customized product suggestions based on the user's purchase history and preferences, which is expected to improve user satisfaction. Furthermore, product suggestions based on similarity enable more appropriate product suggestions.

[1045] An "API for generative AI" is an application programming interface (API) that allows external use of the functions of generative artificial intelligence (AI).

[1046] A "conversational UI" is a user interface (UI) that allows users and systems to communicate interactively using natural language.

[1047] "Means of customer service / suggestion" refers to the means of introducing products and services to users and encouraging them to purchase.

[1048] "Means for providing assistance to programming classes" are means for supporting programming education, enabling learners to learn programming efficiently.

[1049] "Means for analyzing user purchasing history and preferences" refers to the means for collecting and analyzing data on the history of products and services purchased by users in the past and their preferences.

[1050] "Means for proposing individually customized products and services" refers to means for proposing products and services optimized for each user based on the user's purchasing history and preferences.

[1051] "Means for suggesting products based on similarity" refers to a means of analyzing a user's purchasing history and preference data, calculating the similarity with other users, and suggesting products purchased by users with similar tastes.

[1052] A system for implementing this invention includes a means for utilizing a generative AI API to enable a change from touch panel-based operation to an interactive UI based on generative AI, a means for providing customer service / suggestions, a means for providing assistance in programming classes, a means for analyzing a user's purchasing history and preferences and suggesting individually customized products and services, and a means for suggesting products based on similarity.

[1053] System Program

[1054] The system is implemented using Python. Specifically, it manages user purchase history data, vectorizes the data, and calculates similarity. The part that interacts with users uses a generative AI API, employing a generative AI model such as OpenAI's GPT-3.

[1055] Program processing explanation

[1056] The server collects users' purchase history data and converts it into a data frame. Then, it combines the purchase history data as text and vectorizes it. This allows it to calculate the similarity of the purchase history between users using cosine similarity. Based on this similarity, it suggests products to specific users.

[1057] Specifically, based on the user's past purchase history of products and services, the system suggests products purchased by other users with similar tastes. For example, if a user has previously purchased "sports shoes" and "running wear," related products such as "yoga mats" and "swimming goggles" will be suggested based on the purchasing history of similar users.

[1058] Hardware and software used

[1059] Hardware: General PCs and servers

[1060] Software: Python, generative AI API (e.g., OpenAI's GPT-3)

[1061] Prompt Sentence Examples

[1062] Examples of prompts to input to a generative AI model include:

[1063] Suggest relevant products based on the user's purchasing history. For example, if the user has purchased sporting goods in the past, suggest new sporting goods and related products.

[1064] In this way, it becomes possible to make individually customized product suggestions based on the user's purchasing history and preferences, which is expected to improve user satisfaction.In addition, product suggestions based on similarity will enable more appropriate product suggestions.

[1065] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1066] Step 1:

[1067] The server collects user purchase history data. Specifically, it retrieves information such as user ID, product ID, and product name from the database and converts it into a data frame. The input is purchase history data from the database, and the output is purchase history data in data frame format.

[1068] Step 2:

[1069] The server combines the purchase history data as text. Specifically, it aggregates the purchase history data of the same user into a single text data. The input is purchase history data in data frame format, and the output is the purchase history text for each user.

[1070] Step 3:

[1071] The server vectorizes the purchase history text. Specifically, it converts the purchase history text into a TF-IDF (Term Frequency-Inverse Document Frequency) vector. The input is the purchase history text for each user, and the output is a TF-IDF vector.

[1072] Step 4:

[1073] The server calculates the cosine similarity. Specifically, it calculates the similarity of purchase histories between users using the cosine similarity. The input is a TF-IDF vector, and the output is the similarity score between users.

[1074] Step 5:

[1075] The server proposes products based on similarity. Specifically, for a specific user, it proposes products purchased by other users who are highly similar to the user. The input is the similarity score between users and their purchase history data, and the output is a list of suggested products.

[1076] Step 6:

[1077] The server uses the API of generative AI to interact with the user. Specifically, it uses a generative AI model (e.g., GPT-3) to generate an appropriate response to the user's input. The input is the user's input text and prompt sentence, and the output is the generated response text.

[1078] Step 7:

[1079] The terminal displays the suggested product list received from the server and the generated response text to the user. Specifically, the suggested product list and the dialogue content are displayed on the screen of a smartphone or PC. The input is the suggested product list from the server and the generated response text, and the output is the information displayed to the user.

[1080] Example 3

[1081] Next, a description will be given of Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1082] In conventional programming classes, it is difficult for users to receive appropriate support when working on assignments, which often slows down learning progress, especially for beginners. Furthermore, there are limited ways for users to get instant hints when they are stuck on an assignment, which reduces learning efficiency. Furthermore, there is a lack of systems that can accurately check users' answers and provide appropriate feedback. To solve these issues, a support system that allows users to learn programming efficiently is needed.

[1083] The specific processing by the specific processing unit 290 of the data processing device 12 in the third embodiment is realized by the following means.

[1084] In this invention, the server includes a means for a user to log in, a means for the server to issue a question, a means for the user to work on the question, a means for the user to submit an answer, a means for the server to check the answer, a means for the server to provide feedback, a means for the user to request a hint, and a means for the server to provide the hint, thereby enabling a support system that allows users to efficiently learn programming.

[1085] An "API for generative AI" is an application programming interface that allows external use of the functions of generative artificial intelligence.

[1086] "Conversational UI" is a user interface that allows users and systems to communicate two-way using natural language.

[1087] "Means of customer service / suggestion" is a function for suggesting products and services based on the user's needs.

[1088] "Means for providing assistance for programming classes" refers to a function for supporting programming learning.

[1089] "Means for users to log in" refers to the function by which users access the system and perform authentication.

[1090] "Means for the server to provide assignments" refers to the function that enables the server to select programming assignments from the database and present them to the user.

[1091] "Means for users to tackle the assignment" refers to a function that allows users to create a program for the assignment presented to them.

[1092] "Means for users to submit answers" is a function that allows users to send the programs they have created to the server.

[1093] "Means for the server to check answers" refers to the function that the server uses to analyze the accuracy of the programs submitted by users.

[1094] The "means for the server to provide feedback" is a function that allows the server to provide appropriate feedback to the user based on the analysis results of the answer.

[1095] "Means for users to request hints" is a function that allows users to request hints from the server when they are having trouble with a task.

[1096] "Means for the server to provide hints" refers to a function that allows the server to use generative AI to provide appropriate hints to users.

[1097] This invention is an auxiliary system for programming classes that helps users to learn programming efficiently. The system operates in cooperation with three parties: a server, a terminal, and a user.

[1098] The server provides a web interface for users to access. Through this interface, users can complete programming challenges. The server issues the challenges, checks the answers, and provides hints. Specifically, the server does the following:

[1099] 1. Assignment: The server presents the user with programming assignments stored in the database. For example, it assigns an assignment on loop processing in Python.

[1100] 2. Checking the answers: The server receives the code submitted by the user and checks its accuracy using a Python code analysis library.

[1101] 3. Providing Hints: When a user has difficulty with a task, the server generates a hint using a generative AI model (e.g., OpenAI's GPT-3). For example, the server inputs a prompt such as "Please tell me the basic usage of for loops" into the generative AI model and presents the obtained answer to the user.

[1102] A user accesses the system using a terminal and enters their username and password on the login screen. The server receives this and authenticates them by checking the database. If authentication is successful, the server retrieves the user's progress from the database and selects the appropriate task.

[1103] As users work on the assignment, the server automatically saves their code in real time, ensuring that progress isn't lost if the user accidentally closes their browser. Once the user submits their solution, the server instantly analyzes the code and generates feedback, such as "The code is correct. All numbers were printed correctly."

[1104] When a user requests a hint, the server sends a prompt to the generative AI model saying, "Please tell me the basic usage of for loops," and provides the hint within a few seconds. Specifically, the server provides the following hints:

[1105] A for loop is used to cycle through numbers in a specified range. In this example, we'll print the numbers 1 through 10.

[1106] In this way, the server, the terminal, and the user can cooperate to efficiently advance programming learning. The flow of the specific processing in the third embodiment will be described with reference to FIG.

[1107] Step 1:

[1108] A user logs in

[1109] A user accesses the system using a terminal and enters a username and password on the login screen. The server receives this and authenticates it by checking it against the database. The input is the username and password, and the output is the authentication result. If authentication is successful, the user's progress is retrieved from the database.

[1110] Step 2:

[1111] The server issues the assignment

[1112] The server selects a programming task from the database and presents it to the user. The input is the user's progress, and the output is the selected task. For example, the task might be "Write a program that uses a Python for loop to output the numbers from 1 to 10."

[1113] Step 3:

[1114] User tackles the problem

[1115] Users work on assignments on their devices and write programs using a code editor. The input is the assignment and the output is the code written by the user. For example, you might write code like this:

[1116] python

[1117] for i in range(1, 11):

[1118] print(i)

[1119] Step 4:

[1120] User submits answer

[1121] The user submits the answer to the server from their device. When they click the submit button, the code is sent to the server. The input is the code created by the user, and the output is the code sent to the server.

[1122] Step 5:

[1123] The server checks the answer

[1124] The server analyzes the received code and checks its accuracy. It uses a Python code analysis library to do this. The input is the code submitted by the user, and the output is the analysis result. For example, it checks whether the code is error-free and whether it produces the expected output.

[1125] Step 6:

[1126] Server provides feedback

[1127] The server generates feedback based on the check results and presents it to the user. The input is the analysis result, and the output is a feedback message. For example, it provides feedback such as "The code is correct. All numbers were output correctly."

[1128] Step 7:

[1129] The user requests a hint (if necessary)

[1130] When a user has difficulty with a task, they can request a hint from the server. For example, they can input "I don't understand the loop processing." The input is the user's hint request, and the output is a hint request message.

[1131] Step 8:

[1132] The server provides a hint

[1133] The server generates hints using a generative AI model (e.g., OpenAI's GPT-3). For example, the server inputs a prompt such as "Please tell me the basic usage of a for loop" into the generative AI model, and presents the resulting answer to the user. The input is the prompt, and the output is the generated hint. Specifically, the server provides the following hints:

[1134] A for loop is used to cycle through numbers in a specified range. In this example, we'll print the numbers 1 through 10.

[1135] (Application example 3)

[1136] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1137] Previous support systems for programming classes lacked the functionality to provide appropriate hints when users were struggling with a task, resulting in reduced learning efficiency. Furthermore, when learning to program factory robots, there was a lack of support systems that allowed users to learn while actually operating the robots. This made it difficult for users to efficiently acquire programming skills.

[1138] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 3 is realized by the following means.

[1139] In this invention, the server includes a means for using the API of the generative AI to enable a change from touch panel-based operation to an interactive UI based on the generative AI, a means for providing customer service / suggestions, a means for providing assistance in programming classes, a means for supporting factory robot programming learning, a means for checking the user's answers, and a means for providing hints when the user is having trouble with a task, thereby enabling the user to efficiently acquire programming skills.

[1140] An "API for generative AI" is an application programming interface that allows external use of the functions of generative artificial intelligence.

[1141] "Touch panel-based operations" refers to all operations performed by users using a touch panel.

[1142] "Interactive UI based on generative AI" is a user interface that interacts with the user based on generative artificial intelligence.

[1143] "Means of customer service / suggestion" refers to means that have the function of suggesting products and services based on the user's needs.

[1144] "Means for providing assistance for programming classes" are means that have the function of supporting learners in programming classes.

[1145] The "means for supporting learning programming for factory robots" is a means having a support function for learning programming for robots used in factories.

[1146] "Means for checking the user's answers" refers to means that has the function of checking whether the answers entered by the user are correct.

[1147] "Means for providing hints when a user is having trouble with a task" refers to means that has the function of providing hints for solving a task when a user is having difficulty with the task.

[1148] The system for implementing this invention utilizes a generative AI API and includes means for enabling a change from touch panel-based operation to an interactive UI based on generative AI, means for providing customer service / suggestions, means for providing assistance in programming classes, means for supporting learning programming for factory robots, means for checking the user's answers, and means for providing hints when the user is having trouble with a task.

[1149] System configuration

[1150] Hardware

[1151] Server: Hosts the generative AI API and generates appropriate responses to user input.

[1152] Device: A device that a user operates, such as a smartphone, tablet, or factory robot.

[1153] Factory robot: A robot that can be used to learn programming.

[1154] software

[1155] Generative AI API: An application programming interface for externally using the functions of generative artificial intelligence.

[1156] Conversational UI: A user interface that interacts with users based on generative AI.

[1157] Programming learning support application: An application to support learning factory robot programming.

[1158] Data processing and calculation

[1159] Assignment questions

[1160] The server uses the generative AI API to present users with programming challenges, such as "making a robot move forward."

[1161] Answer check

[1162] The server receives the answers entered by the user through the terminal and executes them on the factory robot, checks whether the robot operates correctly, and provides feedback on the results to the user.

[1163] Hints provided

[1164] If the user has difficulty with a task, the server uses the generative AI API to generate an appropriate hint and provide it to the user, for example, "Use the forward command."

[1165] Specific examples

[1166] For example, if a user answers "move forward" to the task "move the robot forward," the server sends the answer to the factory robot and checks whether the robot actually moves forward. If the answer is correct, the robot moves on to the next task; if the answer is incorrect, the server provides a hint: "Use the move forward command."

[1167] Prompt Sentence Examples

[1168] Challenge: Moving the robot forward

[1169] Enter your answer: Advance

[1170] In this way, users can efficiently acquire programming skills.

[1171] The flow of the specific processing in Application Example 3 will be described with reference to FIG.

[1172] Step 1:

[1173] The server uses a generative AI API to present programming challenges to users.

[1174] Input: User's learning progress data

[1175] Data processing: Selecting appropriate tasks based on learning progress data

[1176] Output: The text of the task (e.g., "Move the robot forward")

[1177] Specific operation: The server analyzes the user's past answer history and progress, and generates a challenge of appropriate difficulty. The generated challenge is sent to the device and displayed to the user.

[1178] Step 2:

[1179] Users input their answers to the questions through the terminal.

[1180] Input: Assignment text, user answer

[1181] Data processing: Receive user answers in text format

[1182] Output: User's answer data

[1183] Specific operation: The user enters the answer in the input field on the device and presses the send button. The answer data is sent to the server.

[1184] Step 3:

[1185] The server receives the user's answer and has the factory robot execute it.

[1186] Input: User's answer data

[1187] Data processing: Converting answer data into robot control commands

[1188] Output: Robot control commands

[1189] How it works: The server analyzes the user's answers and converts them into control commands that the robot can understand. The converted commands are then sent to the factory robot.

[1190] Step 4:

[1191] The factory robot operates based on the control commands it receives.

[1192] Input: Robot control command

[1193] Data processing: Execute control commands

[1194] Output: Robot movement results

[1195] Specific operation: The factory robot executes the control command received from the server and performs the specified operation (e.g., moving forward).

[1196] Step 5:

[1197] The server checks the results of the robot's actions and provides feedback to the user.

[1198] Input: Robot action result

[1199] Data processing: Evaluation of operational results

[1200] Output: A feedback message (e.g., "Correct!" or "Incorrect. Hint: Use the forward command")

[1201] Specific behavior: The server receives the robot's behavior results and evaluates whether they meet the requirements of the task. Based on the evaluation results, it generates a feedback message for the user and sends it to the device.

[1202] Step 6:

[1203] When a user has difficulty with a task, the server uses a generative AI API to generate appropriate hints and provide them to the user.

[1204] Input: User answer data, assignment text

[1205] Data processing: Generate hints based on answer data and assignment text

[1206] Output: Hint message (e.g. "Use forward command")

[1207] Specific operation: The server analyzes the user's answer data and the task text, and generates appropriate hints using the generative AI API. The generated hints are sent to the device and displayed to the user.

[1208] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1209] "Example 1"

[1210] One embodiment of the present invention is a system that includes an emotion engine in which a generative AI API recognizes a user's emotions. This system reads emotions from the user's tone of voice, facial expressions, and choice of words, and then engages in dialogue accordingly. For example, if the AI ​​senses that the user is angry, it chooses words to apologize, and if it senses that the user is happy, it chooses words to share that joy.

[1211] "Example 2"

[1212] Another embodiment of the present invention is a customer service / suggestion method using an emotion engine. This system reads the user's emotions and suggests products and services that match those emotions. For example, if the user feels depressed, the AI ​​will suggest products and services that will cheer them up. Conversely, if the user feels happy, the AI ​​will suggest products and services that will amplify that joy.

[1213] "Example 3"

[1214] Another embodiment of the present invention is a programming classroom assistant using an emotion engine. This system reads the user's emotions and provides appropriate learning materials based on those emotions. For example, if the user feels confused, the AI ​​will provide more basic learning materials. Conversely, if the user feels confident, the AI ​​will provide more advanced learning materials.

[1215] The processing flow of each embodiment will be described below.

[1216] "Example 1"

[1217] Step 1: Through dialogue with the user, read their emotions from their tone of voice, facial expressions, choice of words, etc.

[1218] Step 2: The AI ​​adjusts the content of the dialogue based on the emotions it reads.

[1219] Step 3: For example, if the user feels angry, the AI ​​will choose words to apologize, and if the user feels happy, the AI ​​will choose words to share that joy.

[1220] "Example 2"

[1221] Step 1: Interact with the user to understand their emotions.

[1222] Step 2: The AI ​​tailors product and service recommendations based on the emotions it detects.

[1223] Step 3: For example, if the user feels depressed, the AI ​​will suggest products and services that will cheer them up; if the user feels happy, the AI ​​will suggest products and services that will amplify that joy.

[1224] "Example 3"

[1225] Step 1: Interact with the user to understand their emotions.

[1226] Step 2: The AI ​​adjusts the educational material provided based on the emotions it reads.

[1227] Step 3: For example, if the user feels confused, the AI ​​will provide more basic learning materials, and if the user feels confident, the AI ​​will provide more advanced learning materials.

[1228] Example 1

[1229] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1230] Conventional touch panel-based operation systems have the problem that it is difficult for users to operate intuitively, and operation is cumbersome, especially for elderly people and users who are unfamiliar with technology. In addition, the user experience is limited because it is not possible to have a dialogue that takes into account the user's emotions. Furthermore, there is a lack of systems that utilize voice input and emotion recognition, and more natural interaction is required.

[1231] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1232] In this invention, the server includes: a means for utilizing a generative AI API to enable a change from touch panel-based operation to an interactive user interface based on generative AI; a means for providing customer service / suggestions; a means for providing assistance for programming education; a means for collecting user voice input and sending it to the server; a means for the server to convert the voice data into text and analyze it; a means for capturing the user's facial expressions and analyzing their emotions; a means for determining and executing appropriate operations based on the analysis results; a means for generating dialogue using generative AI; and a means for presenting the results to the user. This allows the user to issue operation instructions in natural language, and the system can engage in dialogue that corresponds to the user's emotions.

[1233] "API for generative AI" refers to the application programming interface for generative artificial intelligence, an interface that allows external software and systems to use the functions of generative AI.

[1234] An "interactive user interface" is an interface that allows a user to interact with a system using natural language and to perform operations and give instructions.

[1235] "Means of customer service / suggestion" refers to the function of suggesting products and services based on the user's needs and requests and providing appropriate responses.

[1236] "Means for providing assistance for programming education" refers to means for providing functions and tools to support programming education.

[1237] "Means for collecting voice input and transmitting to a server" refers to a function for collecting the user's voice as digital data and transmitting that data to a server.

[1238] "Means for converting voice data into text and analyzing it" refers to a function for converting collected voice data into text data and analyzing the text to understand the user's intent.

[1239] "Means for capturing facial expressions and analyzing emotions" refers to a function that captures the user's facial expressions with a camera and analyzes the facial expression data to identify the user's emotions.

[1240] "Means for determining and executing appropriate operations" refers to the function of determining the operation that the system should perform based on the analysis results and then executing that operation.

[1241] "Means for generating dialogue" refers to the function of using generative AI to generate dialogue based on user input and emotions.

[1242] "Means for presenting results" refers to the system's ability to visually or audibly present the analysis results and generated dialogue to the user.

[1243] This invention is a system that utilizes a generative AI API to enable a change from touch panel-based operation to an interactive user interface based on generative AI. This system allows users to issue operation instructions in natural language, and the system can engage in dialogue according to the user's emotions.

[1244] Hardware and software used

[1245] Hardware: microphones, cameras, servers, devices (PCs, smartphones, tablets, etc.)

[1246] Software: Generative AI API (e.g., OpenAI GPT-3), emotion engine

[1247] Data processing and calculation

[1248] 1. Acquiring voice input:

[1249] The user issues a command by voice through the microphone of the terminal.

[1250] The device collects the voice data and sends it to the server.

[1251] 2. Analysis of audio data:

[1252] The server receives the voice data and converts it into text using a generative AI API.

[1253] Generative AI analyzes text data and understands the user's intent.

[1254] 3. Emotion recognition:

[1255] The device's camera captures the user's facial expressions and sends them to the emotion engine.

[1256] The emotion engine reads emotions from the user's tone of voice, facial expressions, and choice of words.

[1257] 4. Take appropriate action:

[1258] The server determines the appropriate operation based on the analysis results of the generative AI and the emotional data from the emotion engine.

[1259] For example, in response to an instruction to "search for products," the server searches the product database and presents the results to the user.

[1260] 5. Dialogue generation:

[1261] If the emotion engine determines that the user is angry, the generative AI generates an apology.

[1262] If the emotion engine determines that the user is happy, the generative AI generates words to share that joy.

[1263] 6. Presentation of results:

[1264] The terminal displays the search results and the dialogue message received from the server to the user.

[1265] Specific examples

[1266] User instruction: "Find me a new phone"

[1267] System Action:

[1268] 1. The user says, "Find a new phone."

[1269] 2. The device's microphone collects the audio and sends it to the server.

[1270] 3. The server converts the speech to text using a generative AI API.

[1271] 4. The server analyzes the text data and understands the user's intent.

[1272] 5. The device camera captures the user's facial expressions and sends them to the emotion engine.

[1273] 6. The emotion engine analyzes the user's emotions.

[1274] 7. The server searches the product database and retrieves the results.

[1275] 8. The server uses generative AI to generate a dialogue such as, "I'm glad you found the product you were looking for."

[1276] 9. The terminal displays the search results and interactive messages to the user.

[1277] Prompt Sentence Examples

[1278] Prompt: "If the user indicates they are looking for a new smartphone, search your product database and present the results. Also, if the user is happy, generate a quote to share their joy."

[1279] In this way, the system of the present invention can analyze instructions given by the user in natural language, perform appropriate operations, and generate dialogue that corresponds to the user's emotions.

[1280] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1281] Step 1:

[1282] The user issues a command by voice.

[1283] Input: User's voice command (e.g., "Find a new phone")

[1284] Output: Audio data

[1285] Specific operation: The user speaks into the device's microphone, saying, "Find a new smartphone."

[1286] Step 2:

[1287] The device collects the voice data and sends it to the server.

[1288] Input: Audio data

[1289] Output: Audio data sent to the server

[1290] Specific operation: The device uses the built-in microphone to collect the user's voice as digital voice data and transmits the data to the server.

[1291] Step 3:

[1292] The server converts the audio data into text.

[1293] Input: Audio data sent to the server

[1294] Output: Text data (e.g. "Looking for a new smartphone")

[1295] Specific operation: The server sends the received voice data to a generative AI API (e.g., OpenAI GPT-3) and converts the voice to text.

[1296] Step 4:

[1297] The server analyzes the text data and understands the user's intent.

[1298] Input: Text data

[1299] Output: User intent (e.g., product search request)

[1300] Specific operation: The server analyzes the generated text data and determines that it needs to search the product database based on the instruction "Find a new smartphone."

[1301] Step 5:

[1302] The device captures the user's facial expressions and sends them to the emotion engine.

[1303] Input: User's facial expression data

[1304] Output: Facial expression data sent to the emotion engine

[1305] What it does: The device uses its built-in camera to capture the user's facial expressions and sends that data to the emotion engine.

[1306] Step 6:

[1307] The emotion engine analyzes the user's emotions.

[1308] Input: Facial expression data sent to the emotion engine

[1309] Output: User emotion data (e.g. happy, angry)

[1310] Specific operation: The emotion engine analyzes the received facial expression data and identifies the user's emotion. For example, if the user is smiling, it is judged to be "happy," and if the user has an angry expression, it is judged to be "angry."

[1311] Step 7:

[1312] The server determines and performs the appropriate action.

[1313] Input: User intent, user emotion data

[1314] Output: Operation result (e.g. product search results)

[1315] Specific operation: The server determines the appropriate operation based on the user's intention and emotional data. For example, it searches a product database and retrieves a list of applicable smartphones.

[1316] Step 8:

[1317] The server generates the dialogue using generative AI.

[1318] Input: Operation results, user emotion data

[1319] Output: Dialogue message (e.g. "I'm glad you found the product you were looking for.")

[1320] How it works: The server uses generative AI to generate dialogue based on the user's emotions. For example, if the user is happy, it generates a response such as, "I'm glad you found the product you were looking for."

[1321] Step 9:

[1322] The terminal presents the results to the user.

[1323] Input: Operation result, dialogue message

[1324] Output: Search results and dialogue messages presented to the user

[1325] Specific operation: The device displays the search results and interactive messages received from the server to the user. For example, the device displays a list of smartphones and a message such as "We're glad you found the product you were looking for."

[1326] (Application example 1)

[1327] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1328] Conventional touch panel-based operation systems are difficult for users to operate intuitively, making them particularly difficult to use for elderly people and users unfamiliar with technology. Furthermore, conventional customer service systems are unable to recognize and respond to customer emotions, making it difficult to improve customer satisfaction. Furthermore, even interactive systems using smart glasses are unable to adequately analyze voice input or generate appropriate responses, limiting their use in brick-and-mortar stores.

[1329] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1330] In this invention, the server includes a means for using the API of generative AI to enable a change from touch panel-based operation to a conversational UI based on generative AI, a means for providing customer service / proposals, a means for providing assistance for programming classes, a means for analyzing customer voice input using smart glasses and generating appropriate responses, and a means for recognizing customer emotions and engaging in dialogue accordingly, thereby enabling users to issue operational instructions in natural language and enabling appropriate responses according to the customer's emotions.

[1331] "Generative AI" refers to artificial intelligence that generates new data and information based on user input.

[1332] "API" stands for Application Programming Interface, an interface for exchanging functions and data between different software programs.

[1333] A "touch panel" is a display that can be operated directly with a finger or a stylus.

[1334] An "interactive UI" is a user interface in which the user interacts with the system in natural language.

[1335] "Customer service" refers to the act of providing products and services to customers.

[1336] "Suggestion" refers to the act of recommending products or services based on a user's needs.

[1337] A "programming classroom" is an educational facility that teaches programming knowledge and skills.

[1338] "Assistance" refers to the act of supporting a specific task or activity.

[1339] "Smart glasses" are wearable devices equipped with functions such as a display, camera, and microphone.

[1340] "Voice input" refers to the act of a user giving voice instructions to a system through a microphone.

[1341] "Analysis" refers to the act of analyzing input data and understanding its meaning.

[1342] A "response" is a response that the system gives in response to a user's input.

[1343] "Emotion recognition" is a technology that reads emotions from a user's tone of voice, facial expressions, choice of words, etc.

[1344] "Dialogue" refers to the act of communication between a user and a system.

[1345] The system for implementing this invention utilizes a generative AI API, enabling a change from touch panel-based operation to a conversational UI based on generative AI. The system includes means for providing customer service and suggestions, means for providing assistance in programming classes, means for analyzing customer voice input using smart glasses and generating appropriate responses, and means for recognizing customer emotions and engaging in dialogue accordingly.

[1346] The server uses the generative AI API to analyze the user's voice input and generate an appropriate response. Specifically, when the user issues a command through the smart glasses, the voice data is sent to the server. The server uses speech recognition software to convert the voice data into text and inputs the text into the generative AI model. The generative AI model generates an appropriate response based on the input text and presents the response to the user.

[1347] The server also uses an emotion recognition engine to read emotions from the user's tone of voice and facial expressions. The emotion recognition engine analyzes video data captured by the camera and recognizes the user's emotions. The recognized emotional information is input into a generative AI model and taken into consideration when generating responses.

[1348] The hardware used includes smart glasses, a microphone, and a camera, while the software used includes the OpenAI API, an emotion recognition engine (EmotionRecognizer), speech recognition software, and video capture software (OpenCV).

[1349] As a concrete example, the following prompt sentence is input to the generative AI model:

[1350] "Customer question: 'How do I use this product?'

[1351] Customer sentiment: "Curious"

[1352] Please generate an appropriate response."

[1353] Based on this prompt, the generative AI model generates the following response:

[1354] "This product is very easy to use. First, turn it on, then use the included remote control to operate it. For detailed instructions, please see the manual here."

[1355] In this way, the server can generate an appropriate response based on the user's voice input and emotional information and present it to the user through the smart glasses, allowing the user to issue operational instructions in natural language and respond appropriately according to the customer's emotions.

[1356] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1357] Step 1:

[1358] The user issues voice commands through the smart glasses.

[1359] Input: User's voice commands

[1360] Output: Audio data

[1361] How it works: The user speaks a question or command into the smart glasses' microphone, which then captures the voice and stores it as audio data.

[1362] Step 2:

[1363] The terminal transmits the voice data to the server.

[1364] Input: Audio data

[1365] Output: Audio data sent to the server

[1366] Specific operation: The smart glasses transmit the captured audio data to the server via wireless communication.

[1367] Step 3:

[1368] The server uses speech recognition software to convert the voice data into text.

[1369] Input: Audio data

[1370] Output: Text data

[1371] Specific operation: The server inputs the received voice data into voice recognition software (e.g., OpenAI API) and converts the voice data into text data.

[1372] Step 4:

[1373] The server analyzes the video data captured by the camera and recognizes the user's emotions.

[1374] Input: Video data

[1375] Output: Emotional information

[1376] How it works: The camera in the smart glasses captures the user's facial expressions and sends the video data to the server, which then uses an emotion recognition engine to analyze the video data and recognize the user's emotions.

[1377] Step 5:

[1378] The server inputs text data and emotional information into a generative AI model to generate an appropriate response.

[1379] Input: Text data, emotion information

[1380] Output: Response text

[1381] How it works: The server inputs text data and emotional information as a prompt into a generative AI model (e.g., OpenAI's GPT-3). The generative AI model generates an appropriate response based on the prompt and returns the response text to the server.

[1382] Step 6:

[1383] The server sends the response text to the smart glasses.

[1384] Input: Response text

[1385] Output: Response text sent to the smart glasses

[1386] Specific operation: The server sends the generated response text to the smart glasses, and the smart glasses receive the response text.

[1387] Step 7:

[1388] The smart glasses present the response text to the user.

[1389] Input: Response text

[1390] Output: The response presented to the user

[1391] Specific operation: The smart glasses display the received response text on the display and present it to the user, who can check the response through the smart glasses display.

[1392] Example 2

[1393] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1394] Conventional customer service / recommendation systems were unable to fully utilize users' purchasing history and preferences, and were unable to make recommendations based on the user's emotions, making it difficult to provide optimal products and services to users. Furthermore, it was not easy to switch from touch panel-based operation to a conversational UI based on generative AI, and they lacked support functions for programming education. This resulted in a poor user experience and limited the system's usefulness.

[1395] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1396] In this invention, the server includes a means for using the generative AI API to enable a change from touch panel-based operation to a generative AI-based interactive UI, a means for analyzing a user's purchasing history and preferences and suggesting products and services based on the results, a means for analyzing the user's emotions and suggesting products and services that match those emotions, and a means for providing assistance for programming education. This enables optimal product and service suggestions based on the user's needs and emotions, improving the user experience. It also facilitates the change to a generative AI-based interactive UI, enhancing assistance functions for programming education.

[1397] An "API for generative AI" is an application programming interface that allows external use of the functions of generative artificial intelligence.

[1398] "Conversational UI" is a user interface that allows users and systems to communicate two-way using natural language.

[1399] "Purchase history" is a record of products and services a user has purchased in the past.

[1400] "Preferences" refer to the preferences and interests a user has for a particular category or product.

[1401] "Sentiment analysis" is the process of identifying emotions from a user's text or voice data and assessing their emotional state.

[1402] "Support for programming education" refers to functions and tools to support the learning and teaching of programming.

[1403] "Product and service proposal" is the act of selecting and presenting appropriate products and services to users based on their needs and feelings.

[1404] This invention is a system that utilizes a generative AI API to enable a change from touch panel-based operation to a generative AI-based interactive UI. It also has the ability to analyze a user's purchasing history and preferences and suggest products and services based on those, as well as the ability to analyze a user's emotions and suggest products and services that match those emotions. It also includes support functions for programming education.

[1405] Hardware and software used

[1406] Server: Database server (e.g. MySQL), AI model server (e.g. TensorFlow)

[1407] Device: User interface (e.g. web browser, mobile app)

[1408] Software: Sentiment analysis engines (e.g., IBM Watson), recommendation systems (e.g., Apache Mahout)

[1409] Data processing and calculation

[1410] User Data Collection

[1411] Users input their purchasing history and preferences through the device. For example, they select products they have purchased in the past or categories they are interested in. The device then transmits the information they have entered to the server in real time. Specifically, data is collected through input fields in web forms and mobile apps.

[1412] Data storage

[1413] The server stores the user data sent from the device in a database, such as a MySQL database containing information like user ID, purchase history, and preferred categories. The server establishes a database connection and executes SQL queries to insert the data.

[1414] Analyzing the data

[1415] The server analyzes the stored user data. Specifically, it uses a recommendation system such as Apache Mahout to analyze the user's purchasing patterns and preferences. A sentiment analysis engine uses IBM Watson or similar to analyze emotions from the user's text and voice data. The server retrieves the user data from the database and inputs it into an AI model. The AI ​​model analyzes the user's purchasing history and emotions and returns the results.

[1416] Product and service suggestions

[1417] The server selects appropriate products and services based on the analysis results. For example, if the user has purchased sports equipment in the past, it will suggest new sports equipment products. The emotion engine suggests products and services that match the user's emotions. For example, if the user is feeling down, it will suggest products that will cheer them up. The server integrates the results of the recommendation system and the emotion analysis engine to generate optimal suggestions.

[1418] View Suggestions

[1419] The device displays the suggestions received from the server to the user, for example, by displaying a list of suggested products or services on a web page or mobile app interface. The device receives the response from the server and renders the data in the user interface.

[1420] Examples of specific examples and prompts

[1421] Specific examples

[1422] User purchase history: If the user has purchased sporting goods in the past, we will suggest new sporting goods and related products.

[1423] Emotion-based suggestions: If the emotion engine determines that a user is feeling down, it will suggest products or services that will cheer them up. Conversely, if a user is happy, it will suggest products or services that will increase that joy.

[1424] Prompt Sentence Examples

[1425] "Based on a user's purchase history, suggest products they're likely to buy next."

[1426] "Analyze user emotions and suggest products and services that match those emotions."

[1427] In this way, the server, terminal, and user work together to realize a system that proposes optimal products and services to users.

[1428] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1429] Step 1: Collect user data

[1430] User: The user inputs information about their purchasing history and preferences through the device, for example, selecting products they have purchased in the past and categories they are interested in.

[1431] Input: Purchase history and preference information entered by the user.

[1432] Device: The device transmits user-entered information to the server in real time, specifically collecting data through input fields in web forms and mobile apps.

[1433] Output: User data sent to the server.

[1434] Step 2: Save your data

[1435] Server: The server stores the user data sent from the device in a database. For example, it stores information such as user ID, purchase history, and favorite categories in a MySQL database.

[1436] Input: User data sent from the device.

[1437] What happens: The server establishes a database connection and executes an SQL query to insert data.

[1438] Output: User data stored in the database.

[1439] Step 3: Analyze the data

[1440] Server: The server analyzes the stored user data, specifically using a recommendation system such as Apache Mahout to analyze user purchasing patterns and preferences.

[1441] Input: User data retrieved from the database.

[1442] Sentiment Analysis Engine: The server uses the sentiment analysis engine to analyze emotions from the user's text and voice data.

[1443] How it works: The server retrieves user data from the database and inputs it into the AI ​​model. The AI ​​model analyzes the user's purchasing history and emotions and returns the results.

[1444] Output: User purchasing patterns and sentiment analysis results.

[1445] Step 4: Propose your product or service

[1446] Server: The server selects appropriate products and services based on the analysis results. For example, if the user has purchased sports equipment in the past, it will suggest new sports equipment.

[1447] Input: User purchasing patterns and sentiment analysis results.

[1448] Emotion engine: The server recommends products and services that match the user's emotions. For example, if the user is feeling down, it will recommend products that will cheer them up.

[1449] Specific operation: The server integrates the results of the recommendation system and the sentiment analysis engine to generate optimal suggestions.

[1450] Output: A list of suggested products and services.

[1451] Step 5: Viewing Proposals

[1452] Device: The device displays the suggestions received from the server to the user, for example, by displaying a list of suggested products or services on a web page or mobile app interface.

[1453] Input: Proposal data received from the server.

[1454] Specific operation: The device receives the response from the server and renders the data in the user interface.

[1455] Output: The suggested products or services that are displayed to the user.

[1456] (Application example 2)

[1457] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1458] While existing customer service and recommendation systems can take into account a user's purchasing history and preferences, they are unable to analyze user emotions in real time and recommend products and services based on those emotions. Furthermore, there is a lack of effective ways to improve the shopping experience in physical stores, making it difficult to increase user satisfaction.

[1459] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1460] In this invention, the server includes a means for using a generative AI API to enable a change from touch panel-based operation to a generative AI-based interactive UI, a means for customer service / suggestion, a means for providing assistance for programming classes, a means for analyzing a user's purchase history, preferences, and emotions and suggesting products and services based on the results, and a means for suggesting products and services to the user using smart glasses. This makes it possible to analyze a user's emotions in real time and suggest products and services based on the results. It also improves the shopping experience in physical stores and increases user satisfaction.

[1461] "Generative AI" is an artificial intelligence technology that generates appropriate responses and suggestions based on user input and data.

[1462] "API" stands for Application Programming Interface, an interface for exchanging functions and data between different software programs.

[1463] An "interactive UI" is a user interface in which the user and the system perform operations and exchange information through natural dialogue.

[1464] "Means of customer service / suggestion" is a function for suggesting products and services based on the user's needs and preferences.

[1465] "Means for providing assistance to programming classes" is a function for supporting programming education.

[1466] "Purchase history" is a record of products and services a user has purchased in the past.

[1467] "Preferences" refer to the tendency of users to particularly like certain products or services.

[1468] "Emotion" refers to the user's psychological state or mood.

[1469] "Smart glasses" are wearable devices equipped with functions such as a display and camera.

[1470] "Means for suggesting products and services" is a function for suggesting appropriate products and services based on user data and emotions.

[1471] The following system configuration will be described as an embodiment of the present invention.

[1472] System Configuration

[1473] This system utilizes a generative AI API and includes a means to change from touch panel-based operation to an interactive UI based on generative AI, a means to provide customer service / suggestions, a means to provide assistance for programming classes, a means to analyze users' purchasing history, preferences, and emotions and suggest products and services based on that, and a means to suggest products and services to users using smart glasses.

[1474] Hardware and Software

[1475] Hardware: smart glasses, camera, server

[1476] Software: OpenCV (image processing library), EmotionRecognizer (emotion recognition library), RecommendationEngine (suggestion engine)

[1477] Processing flow

[1478] 1. Camera setup: The camera built into the smart glasses is used to capture the user's face.

[1479] 2. Emotion Recognition: Recognize user emotions from captured images using EmotionRecognizer.

[1480] 3. Generate Recommendations: Use the Recommendation Engine to suggest products and services based on the recognized sentiment.

[1481] 4. Displaying suggestions: Proposed products and services are displayed on the smart glasses display.

[1482] Specific examples

[1483] For example, if a user is in a sporting goods store, the app will suggest new sports shoes and related accessories based on the data of sports equipment purchased in the past. If the app detects that a user is feeling depressed, it will suggest a relaxing scented candle or a refreshing drink.

[1484] Prompt Sentence Examples

[1485] "Please suggest new related products based on the product data of the user's past purchases."

[1486] "Analyze user sentiment in real time and suggest products and services that match those sentiments."

[1487] In this way, it is possible to analyze user sentiment in real time and suggest products and services based on that sentiment, improving the shopping experience in physical stores and increasing user satisfaction.

[1488] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1489] Step 1:

[1490] The server activates the camera built into the smart glasses and captures the user's face.

[1491] Input: Camera image

[1492] Output: Captured user face image

[1493] What happens: The camera detects the user's face and captures its image data.

[1494] Step 2:

[1495] The server inputs the captured facial image into EmotionRecognizer to recognize the user's emotions.

[1496] Input: A captured image of the user's face

[1497] Output: Recognized user emotion data (e.g., happy, sad, surprised, etc.)

[1498] Specific operation: EmotionRecognizer analyzes facial images and performs data calculations to identify emotions.

[1499] Step 3:

[1500] The server inputs the recognized emotional data into the Recommendation Engine, which then compares it with the user's purchasing history and preferences to suggest products and services.

[1501] Input: Recognized user emotion data, user purchase history, user preferences

[1502] Output: A list of suggested products and services

[1503] Specific operation: The Recommendation Engine performs data calculations based on emotional data, purchasing history, and preferences to select the most suitable products and services.

[1504] Step 4:

[1505] The server displays a list of suggested goods and services on the smart glasses' display.

[1506] Input: List of proposed products and services

[1507] Output: Product and service information displayed on the smart glasses display

[1508] Specific operation: Displays product and service information as text and images on the smart glasses display.

[1509] Step 5:

[1510] The user reviews the offers displayed on the smart glasses and makes selections and purchases as needed.

[1511] Input: Product or service information displayed on smart glasses

[1512] Output: User's choice or purchase action

[1513] Specific Action: The user interacts with the smart glasses interface to select or purchase the suggested product or service.

[1514] Example 3

[1515] Next, a description will be given of Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1516] In conventional programming classes, when users find a task difficult, it takes a lot of time and effort to select the appropriate learning materials. It is also difficult to provide appropriate feedback and learning materials according to the user's emotions, which reduces learning efficiency. Furthermore, there is a lack of systems using interactive UIs, which limits the user's learning experience.

[1517] The specific processing by the specific processing unit 290 of the data processing device 12 in the third embodiment is realized by the following means.

[1518] In this invention, the server includes a means for using the API of the generative AI to enable a change from touch panel-based operation to an interactive UI based on the generative AI, a means for providing customer service / suggestions, a means for providing assistance for programming classes, a means for checking the user's answers, a means for providing hints when the user is struggling with an assignment, and a means for reading the user's emotions using an emotion engine and providing appropriate learning materials according to those emotions. This improves the user's learning efficiency and makes it possible to provide appropriate feedback and learning materials.

[1519] A "generative AI API" is an application programming interface for generating data using artificial intelligence.

[1520] An "interactive UI" is a user interface that allows two-way communication between the user and the system using natural language.

[1521] "Means for customer service / suggestions" is a function for suggesting products and services based on the user's needs.

[1522] "Means for providing assistance for programming classes" refers to a function for supporting programming learning.

[1523] The "means for checking the user's answers" is a function for analyzing the answers to the program entered by the user and evaluating their accuracy.

[1524] The "means of providing hints when the user is having trouble with a task" is a function for providing appropriate advice or examples when the user is having difficulty with a task.

[1525] An "emotion engine" is a technology that analyzes a user's facial expressions and voice to determine their emotional state.

[1526] "Means for providing appropriate learning materials" refers to a function for selecting and providing optimal learning materials according to the user's learning situation and emotional state.

[1527] This invention is a support system for programming classes, which uses a generative AI API to realize an interactive UI and support users' learning. This system operates among three parties: a server, a terminal, and a user.

[1528] Hardware and software used

[1529] Server: Cloud computing service (e.g. AWS)

[1530] Device: User's PC or tablet

[1531] Generative AI model: General-purpose generative AI (e.g., GPT-4)

[1532] Emotion Engine: Emotion Analysis Software

[1533] Specific operation of the system

[1534] Assignment questions

[1535] When a user logs in, the server presents a programming assignment. The assignment is randomly selected from a collection of problems stored in a database on the server. For example, the server might display, "Please solve the following assignment. Write a program that uses a 'for' loop to output the numbers from 1 to 10."

[1536] Enter and submit your answers

[1537] The user inputs the answer to the task on the terminal. The terminal sends the input answer to the server. For example, the user inputs "for (int i = 1; i <= 10; i++) { System.out.println(i);}".

[1538] Check your answers

[1539] The server sends the received answer to the generative AI model to check its accuracy. The generative AI model analyzes the answer and determines whether it is correct or incorrect. The server then sends the check results to the device, which displays them to the user. For example, the server might display "The answer is correct" or "There is an error in the answer."

[1540] Providing hints

[1541] When a user has difficulty with a task, the device sends a "hint request" request to the server. The server uses a generative AI model to generate an appropriate hint and sends it to the device. The device then displays the hint to the user. For example, the server might display "The basic syntax of a 'for' loop is as follows: for (initialization; condition; update) { / / processing}."

[1542] Reading emotions

[1543] The device analyzes the user's facial expressions and voice using an emotion engine and sends the emotion data to the server. The server analyzes the emotion data and determines whether the user is confused or confident. For example, the device sends the emotion data saying "the user is confused."

[1544] Provision of teaching materials

[1545] The server selects appropriate learning materials according to the user's emotions. The server sends the selected learning materials to the terminal, and the terminal displays the learning materials to the user. For example, the server displays, "We sense that the user is confused, so we will provide learning materials on basic loop processing."

[1546] Prompt Sentence Examples

[1547] Assignment questions

[1548] "Give the user the following programming assignment: Write a program that uses a 'for' loop to print the numbers 1 through 10."

[1549] Providing hints

[1550] "If a user is struggling with a looping problem, please provide them with basic concepts and concrete code examples."

[1551] Providing emotionally relevant learning materials

[1552] "If you feel the user is confused, provide basic learning materials. Conversely, if you feel the user is confident, provide more advanced learning materials." The flow of the identification process in Example 3 will be explained using Figure 21.

[1553] Step 1:

[1554] User Login

[1555] A user logs into the system using a terminal. The terminal sends the user's authentication information (username and password) to the server. The server checks the received authentication information against a database, and if authentication is successful, returns a login success message to the user. The input is the user's authentication information, and the output is a login success or failure message.

[1556] Step 2:

[1557] Assignment questions

[1558] The server presents programming assignments to logged-in users. The server randomly selects an assignment from the database and sends it to the terminal. The terminal then displays the received assignment to the user. The input is the user's login information, and the output is the content of the assignment. Specifically, the server displays the message, "Please solve the following assignment. Write a program that uses a 'for' loop to output the numbers from 1 to 10."

[1559] Step 3:

[1560] Enter and submit your answers

[1561] The user inputs the answer to the task on the terminal. The terminal sends the input answer to the server. The input is the user's answer, and the output is the sending of the answer to the server. Specifically, the user inputs "for (int i = 1; i <= 10; i++) { System.out.println(i);}".

[1562] Step 4:

[1563] Check your answers

[1564] The server sends the received answer to the generative AI model to check its accuracy. The generative AI model analyzes the answer and determines whether it is correct or incorrect. The server sends the check results to the terminal, which displays them to the user. The input is the user's answer, and the output is a judgment of whether the answer is correct or incorrect. Specifically, the server displays "The answer is correct" or "There is an error in the answer."

[1565] Step 5:

[1566] Providing hints

[1567] When a user has trouble with a task, the device sends a "hint request" request to the server. The server uses a generative AI model to generate an appropriate hint and sends it to the device. The device then displays the hint to the user. The input is the user's request, and the output is the content of the hint. In concrete terms, the server displays "The basic syntax of a 'for' loop is as follows: for (initialization; condition; update) { / / processing}".

[1568] Step 6:

[1569] Reading emotions

[1570] The device analyzes the user's facial expressions and voice using an emotion engine and sends the emotion data to the server. The server analyzes the emotion data and determines whether the user is confused or confident. The input is the user's facial expressions and voice, and the output is emotion data. Specifically, the device sends emotion data stating that "the user is confused."

[1571] Step 7:

[1572] Provision of teaching materials

[1573] The server selects appropriate learning materials according to the user's emotions. The server sends the selected learning materials to the terminal, which then displays the learning materials to the user. The input is emotional data, and the output is appropriate learning materials. Specifically, the server displays the message, "We sense that the user is confused, so we will provide learning materials on basic loop processing."

[1574] (Application example 3)

[1575] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1576] Conventional programming education systems have difficulty providing appropriate learning materials based on the user's emotions and level of understanding, making it difficult to meet individual needs, especially in specialized training for factory robot operators and engineers. Furthermore, touch panel-based operation makes it impossible to realize an intuitive, interactive user interface, preventing users from improving their learning efficiency. To solve these issues, a system combining a conversational UI utilizing generative AI with emotion analysis capabilities is needed.

[1577] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 3 is realized by the following means.

[1578] In this invention, the server includes a means for using the API of the generative AI to enable a change from touch panel-based operation to an interactive UI based on the generative AI, a means for providing customer service / suggestions, a means for providing assistance for programming classes, a means for analyzing the user's emotions and providing appropriate learning materials according to those emotions, and a means for supporting programming education for factory robot operators and engineers. This makes it possible to provide appropriate learning materials according to the user's emotions and level of understanding, thereby realizing effective learning support that meets the individual needs of factory robot operators and engineers, particularly in specialized education.

[1579] "Generative AI" is a system that uses artificial intelligence technology to generate new data and information.

[1580] "API" stands for Application Programming Interface, an interface for exchanging functions and data between different software programs.

[1581] An "interactive UI" is a user interface in which the user and the system perform operations and exchange information through natural dialogue.

[1582] "Means for customer service / suggestions" refers to a system that has the function of suggesting products and services based on the user's needs.

[1583] "Means for providing assistance for programming classes" refers to a system with functions to support learning programming.

[1584] "Means for analyzing a user's emotions and providing appropriate learning materials in accordance with those emotions" refers to a system that has the function of analyzing a user's emotions and providing appropriate learning materials based on the results.

[1585] "A means for supporting programming education for factory robot operators and engineers" is a system that has the function of supporting factory robot operators and engineers when they learn programming.

[1586] The system for implementing this invention includes a means for utilizing a generative AI API to enable a change from touch panel-based operation to an interactive UI based on generative AI, a means for providing customer service / suggestions, a means for providing assistance in programming classes, a means for analyzing user emotions and providing appropriate teaching materials according to those emotions, and a means for supporting programming education for factory robot operators and engineers.

[1587] The server uses a generative AI API to generate appropriate responses to user input. This allows for a shift from touchscreen-based operation to a conversational UI. Specifically, when a user is working on a programming challenge, the server analyzes the user's input and provides appropriate hints and learning materials.

[1588] The terminals are devices such as smartphones and head-mounted displays, allowing users to learn programming in the actual work environment of a factory. The terminals are equipped with cameras and microphones to analyze the user's emotions and send this data to a server.

[1589] To analyze user emotions, the system uses natural language processing (NLP) technology. Specifically, it uses Python and NLP libraries to analyze the user's input text and voice data. Based on the analysis results, the server classifies the user's emotions into categories such as "confused," "confident," or "neutral," and provides appropriate learning materials accordingly.

[1590] As a concrete example, consider a factory robot operator working on a loop processing problem. If the user types "I don't understand the loop processing," the server interprets the user's emotion as "confused" and provides the following hints:

[1591] Let's review some basic concepts of looping. For example, a for loop repeats a specific number of times. Here's a simple example:

[1592] python

[1593] for i in range(5):

[1594] print(i)

[1595] This code will output the numbers 0 through 4.

[1596] Example prompts to input to a generative AI model:

[1597] "When users are struggling with looping challenges, provide them with basic concepts and concrete code examples."

[1598] In this way, it becomes possible to provide appropriate learning materials according to the user's emotions and level of understanding, enabling effective learning support that meets individual needs, particularly in specialized education for factory robot operators and engineers.

[1599] The flow of the specific processing in Application Example 3 will be described with reference to FIG.

[1600] Step 1:

[1601] A user uses a terminal to work on a programming assignment.

[1602] Input: User input for the problem (e.g., "I don't understand loop processing")

[1603] Output: User input data

[1604] Specific actions: The user uses a smartphone or head-mounted display to input questions or difficulties regarding the programming task.

[1605] Step 2:

[1606] The terminal transmits the user's input data to the server.

[1607] Input: User-entered data

[1608] Output: User input data sent to the server

[1609] Specific operation: The terminal collects the user's input data and sends it to a server via the Internet.

[1610] Step 3:

[1611] The server receives the user's input data and analyzes it using a generative AI model.

[1612] Input: User-entered data

[1613] Output: Analysis result (e.g. "confused")

[1614] How it works: The server uses a generative AI model (e.g., a natural language processing library) to analyze the user's input data and classify the user's emotions into categories such as "confused," "confident," or "neutral."

[1615] Step 4:

[1616] The server generates appropriate teaching materials based on the analysis results.

[1617] Input: Analysis result (e.g. "confused")

[1618] Output: Appropriate teaching materials (e.g., basic concepts of loop processing and concrete code examples)

[1619] Specific operation: The server uses the generative AI model to generate learning materials appropriate for the user based on the analysis results. For example, if the user is analyzed as "confused," it will provide basic concepts and specific code examples.

[1620] Step 5:

[1621] The server sends the generated teaching materials to the terminal.

[1622] Input: Generated teaching materials

[1623] Output: The teaching materials sent to the device

[1624] Specific operation: The teaching materials generated by the server are sent to the terminal via the Internet.

[1625] Step 6:

[1626] The terminal displays the received educational material to the user.

[1627] Input: Teaching materials sent from the server

[1628] Output: The material displayed to the user

[1629] Specific operation: The terminal displays the received learning material to the user, allowing the user to study it.

[1630] Step 7:

[1631] The user takes on a new challenge or enters an additional question.

[1632] Input: New input data from the user

[1633] Output: New input data

[1634] Specific Action: The user attempts a new task or enters an additional question and returns to step 1.

[1635] The above processing steps make it possible to provide appropriate learning materials according to the user's emotions and level of understanding, thereby enabling effective learning support that meets individual needs, particularly in specialized education for factory robot operators and engineers.

[1636] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1637] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1638] Another example of generative AI is Gemini (internet search engine). <url: https: gemini.google.com ?hl="ja">) are mentioned.

[1639] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1640] [Third embodiment]

[1641] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1642] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[1643] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1644] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1645] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1646] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1647] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1648] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1649] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1650] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1651] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1652] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[1653] "Example 1"

[1654] As one embodiment of the present invention, a system utilizing a generative AI API is provided. This system enables a shift from touch panel-based operation to a conversational UI based on generative AI. Specifically, when a user issues an operation command in natural language without using a touch panel, the generative AI analyzes it and performs the appropriate operation. For example, in response to an instruction to "search for a product," the system searches a product database and presents the results to the user.

[1655] "Example 2"

[1656] In addition, as one embodiment of the present invention, a customer service / suggestion system is provided. This system proposes products and services based on the user's needs. Specifically, the system analyzes the user's purchasing history and preferences and proposes products and services based on those. For example, if the user has previously purchased sporting goods, the system proposes new sporting goods and related products.

[1657] "Example 3"

[1658] Furthermore, as one embodiment of the present invention, a system for providing assistance in programming classes is provided. This system has a function for supporting programming learning. Specifically, the system presents programming problems and checks the user's answers. The system also provides hints when the user is having trouble with a problem. For example, if the user is having trouble with a loop processing problem, the system presents the basic concept of loop processing and specific code examples.

[1659] The processing flow of each embodiment will be described below.

[1660] "Example 1"

[1661] Step 1: The user issues a command in natural language, such as "Search for a product."

[1662] Step 2: The generative AI analyzes the user's instructions. This analysis is performed using natural language processing technology.

[1663] Step 3: Based on the analysis results, the system takes appropriate action, specifically searching the product database and presenting the results to the user.

[1664] "Example 2"

[1665] Step 1: The system analyzes the user's purchasing history and preferences using machine learning techniques.

[1666] Step 2: Based on the analysis, the system will suggest products and services. For example, if the user has purchased sports equipment in the past, it will suggest new sports equipment and related products.

[1667] "Example 3"

[1668] Step 1: The system presents programming problems, which are chosen appropriately based on the user's learning progress and level of understanding.

[1669] Step 2: The user completes the assignment and submits their answer.

[1670] Step 3: The system checks the user's answer. If the answer is correct, the system moves on to the next task. If the answer is incorrect, the system provides feedback to the user.

[1671] Step 4: If the user is stuck on a task, the system will provide hints, such as explaining the basic concept of looping or providing concrete code examples.

[1672] Example 1

[1673] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1674] Conventional touch panel-based operation systems have the problem that it is difficult for users to operate intuitively, and operation is cumbersome, especially for elderly people and users who are unfamiliar with technology. In addition, when serving customers or making suggestions, it is necessary to respond quickly and accurately to user needs, but conventional systems have not been able to fully achieve this. Furthermore, even for assistance systems for programming classes, there was a need for an interface that allows users to issue operating instructions in natural language.

[1675] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1676] In this invention, the server includes: a means for the user to issue operation instructions in natural language; a means for the terminal to send the user's instructions to the generative AI's API; a means for the generative AI to analyze the user's instructions; a means for the server to receive instructions from the generative AI and search a database; a means for the server to return search results to the generative AI; a means for the generative AI to format the search results in natural language; a means for the terminal to present the search results to the user; a means for enabling a change from touch panel-based operation to a conversational UI based on the generative AI; a means for providing customer service / suggestions; and a means for providing assistance for programming classes. This allows the user to intuitively issue operation instructions in natural language, enabling quick and accurate responses. Furthermore, the accuracy of customer service and suggestions is improved, making it possible to effectively function as an assistance system for programming classes.

[1677] "User" refers to a person who uses the system to issue operational instructions.

[1678] "Terminal" refers to the device through which the user inputs operational instructions and sends them to the generative AI's API.

[1679] "Generative AI" refers to artificial intelligence that analyzes a user's instructions in natural language and generates appropriate responses and operations.

[1680] "API" stands for Application Programming Interface, and refers to an interface for exchanging functions and data between different software programs.

[1681] A "server" refers to a computer system whose role is to receive instructions from the generative AI, search a database, and return the results.

[1682] A "database" refers to a collection of data that is systematically organized and stored to facilitate easy searching and updating.

[1683] "Conversational UI" refers to a user interface in which the user issues instructions in natural language and the system responds accordingly.

[1684] "Customer service / suggestion" refers to the act of suggesting products or services based on the user's needs.

[1685] "Support for programming classes" refers to the provision of functions and services to support programming education.

[1686] This invention is a system that utilizes a generative AI API to enable a change from touch panel-based operation to a conversational UI based on generative AI. This system allows users to issue operation instructions in natural language, and the generative AI analyzes the instructions and performs appropriate responses and operations.

[1687] Hardware and software used

[1688] Hardware: Smartphones, tablets, PCs, servers

[1689] Software: Generative AI APIs (e.g., OpenAI's GPT-3), speech recognition software (e.g., Google Speech-to-Text API), database management systems (e.g., MySQL)

[1690] System configuration

[1691] 1. The user issues instructions in natural language.

[1692] For example: "Search for a product"

[1693] 2. The device receives the user's instructions and sends them to the generative AI API.

[1694] The device converts voice input into text and sends that text to the generative AI API.

[1695] 3. The generative AI analyzes the user's instructions and determines the appropriate operation.

[1696] Generative AI analyzes the text it receives and understands the user's intent.

[1697] 4. The server receives instructions from the generative AI and searches the product database.

[1698] The server receives queries from the generative AI and searches the product database.

[1699] 5. The server sends the search results back to the generative AI.

[1700] The server sends the search results back to the generative AI.

[1701] 6. Generative AI formats the search results in natural language and returns them to the device.

[1702] Generative AI formats the results in natural language.

[1703] 7. The device presents the search results to the user.

[1704] The device presents the natural language results received from the generative AI to the user.

[1705] Specific examples

[1706] When a user commands the device to "search for products," the device converts the command into text using voice recognition software and sends it to the generative AI's API. The generative AI analyzes the command, generates a query to search the product database, and sends it to the server. The server searches the product database and returns the results to the generative AI. The generative AI formats the results in natural language and returns them to the device. Finally, the device presents the search results to the user.

[1707] Prompt Sentence Examples

[1708] "If a user instructs the AI ​​to 'search for a product,' please explain how the generative AI will analyze this and what operations it will perform."

[1709] This system allows users to issue operational instructions in natural language without using a touch panel, making operation more intuitive and efficient. It also improves the accuracy of customer service and suggestions, and can function effectively as an auxiliary system for programming classes.

[1710] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1711] Step 1:

[1712] The user issues an operation instruction in natural language.

[1713] Input: User's voice command (e.g. "Search for products")

[1714] Output: Audio data

[1715] Specific action: The user speaks into the smartphone's microphone.

[1716] Step 2:

[1717] The device sends the user's instructions to the generative AI API.

[1718] Input: Audio data

[1719] Output: Text data

[1720] Specific operation: The device uses voice recognition software (e.g., Google Speech-to-Text API) to convert voice data into text data, and then sends that text to the generative AI API as an HTTP request.

[1721] Step 3:

[1722] Generative AI analyzes user instructions.

[1723] Input: Text data (e.g., "Search for a product")

[1724] Output: Analysis results (e.g., search query)

[1725] How it works: Generative AI (e.g., OpenAI's GPT-3) analyzes text data, understands user intent, and generates search queries.

[1726] Step 4:

[1727] The server receives instructions from the generative AI and searches the product database.

[1728] Input: Search query

[1729] Output: Search result data

[1730] Specific operation: The server uses a database management system (e.g., MySQL) to execute an SQL query against the product database and retrieve search results.

[1731] Step 5:

[1732] The server sends the search results back to the generative AI.

[1733] Input: Search result data

[1734] Output: Search result data (sent to generative AI)

[1735] Specific operation: The server sends the search results in JSON format to the generative AI.

[1736] Step 6:

[1737] Generative AI formats search results in natural language.

[1738] Input: Search result data

[1739] Output: Natural language result text

[1740] Specific operation: The generative AI analyzes the search result data and formats the results in a format such as "The following products were found: Product A, Product B, Product C."

[1741] Step 7:

[1742] The terminal presents the search results to the user.

[1743] Input: Natural language result text

[1744] Output: Search results presented to the user

[1745] What happens: Your device displays search results on the screen or reads them out loud.

[1746] (Application example 1)

[1747] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1748] Conventional touch panel-based operation systems have the problem that it is difficult for users to operate them intuitively, and that operation is cumbersome, especially for elderly people and users who are unfamiliar with technology. Furthermore, even systems that use voice recognition technology have difficulty accurately analyzing the user's natural language instructions and generating appropriate responses. Furthermore, there is a demand for improving the user experience by introducing a conversational UI that uses generative AI.

[1749] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1750] In this invention, the server includes means for using a generative AI API to enable a change from touch panel-based operation to a generative AI-based conversational UI, means for providing customer service / suggestions, means for providing assistance for programming education, means for converting a user's natural language instructions into text using speech recognition, means for analyzing the user's instructions and generating appropriate operations using the generative AI, and means for responding to the user with the generated operation results by voice using speech synthesis. This allows the user to issue operation instructions in natural language without using a touch panel, achieving intuitive and easy operation.

[1751] "Generative AI" is an artificial intelligence technology that analyzes user input and generates appropriate responses and operations.

[1752] "API" stands for Application Program Interface, an interface for exchanging functions and data between different software programs.

[1753] A "touch panel" is an input device that a user can directly operate using a finger or a stylus.

[1754] A "conversational UI" is a user interface that allows users and systems to interact using natural language.

[1755] "Customer service / proposals" refers to the act of proposing products and services based on the user's needs.

[1756] "Support for programming education" means providing functions and tools to support programming education.

[1757] "Speech recognition" is a technology that converts a user's voice into text.

[1758] "Natural language instructions" are instructions given to the system by the user in a language that they normally use.

[1759] "Speech synthesis" is a technology that converts text information into speech and outputs it.

[1760] A system for implementing this invention utilizes a generative AI API to enable a change from touch panel-based operation to a conversational UI based on generative AI. The system includes means for providing customer service / suggestions, means for providing assistance for programming education, means for converting a user's natural language instructions into text using speech recognition, means for analyzing the user's instructions and generating appropriate operations using generative AI, and means for responding to the user with the generated operation results by voice using speech synthesis.

[1761] Hardware and software used

[1762] Hardware: Smartphone, microphone

[1763] Software: OpenAI API, SpeechRecognition library, pyttsx3 library

[1764] Data processing and calculation

[1765] Voice Recognition

[1766] The device uses speech recognition technology to convert the user's speech into text. Specifically, it uses the SpeechRecognition library to capture the user's voice input and converts it into text using Google's speech recognition service.

[1767] Use of generative AI

[1768] The server uses the API of generative AI to analyze the user's text instructions and generate appropriate operations. Specifically, it uses the OpenAI API to generate responses and operations based on the user's instructions.

[1769] Speech synthesis

[1770] The device uses speech synthesis technology to respond to the user with the generated operation results by voice. Specifically, it uses the pyttsx3 library to convert text information into speech and respond to the user.

[1771] Specific examples

[1772] When a user says "Find a new smartphone," the system works as follows:

[1773] 1. The device captures the user's speech and converts it to text using the SpeechRecognition library.

[1774] 2. The server uses the generative AI API to analyze the user's instructions and generate a list of appropriate smartphones.

[1775] 3. The device uses the pyttsx3 library to audibly respond to the user with the generated list of smartphones.

[1776] Prompt Sentence Examples

[1777] User Instructions: Find a new smartphone

[1778] Please take appropriate action.

[1779] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1780] Step 1:

[1781] The user issues a command by voice.

[1782] Input: User's voice commands

[1783] Output: Audio data

[1784] Specific action: The user speaks a natural language command such as "Find a new smartphone."

[1785] Step 2:

[1786] The device performs voice recognition and converts the voice data into text.

[1787] Input: Audio data

[1788] Output: Text data

[1789] How it works: The device uses the microphone to capture voice data and converts it into text through Google's speech recognition service using the SpeechRecognition library.

[1790] Step 3:

[1791] The server uses the generative AI API to analyze the text data and generate appropriate operations.

[1792] Input: Text data

[1793] Output: Generated operation results (e.g., a list of smartphones)

[1794] Specific operation: The server uses OpenAI's API to generate responses and operations based on the user's instructions. For example, in response to the instruction "Find a new smartphone," it generates a list of smartphones.

[1795] Step 4:

[1796] The terminal performs voice synthesis and responds to the user by voice with the generated operation results.

[1797] Input: Generated operation result (text data)

[1798] Output: Audio data

[1799] Specific operation: The device uses the pyttsx3 library to convert the generated list of smartphones into speech and respond to the user.

[1800] Step 5:

[1801] The user receives a voice response.

[1802] Input: Audio data

[1803] Output: User understanding and behavior

[1804] Specific actions: The user listens to the voice response from the device and decides on the next action, such as selecting from a list of smartphones presented.

[1805] Example 2

[1806] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1807] With conventional customer service / recommendation systems, it was difficult to effectively utilize a user's purchasing history and preferences to recommend products and services. Furthermore, it was not easy to switch from touch panel-based operation to a conversational UI based on generative AI, and there was a need to improve the user experience. Furthermore, there was a lack of support functions for programming classes.

[1808] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1809] In this invention, the server includes: means for utilizing a generative AI API to enable a change from touch panel-based operation to an interactive UI based on generative AI; means for collecting data on a user's purchase history and preferences and storing it in a database; means for analyzing data acquired from the database using a data analysis tool; means for generating product and service proposals based on the analysis results using a generative AI model; means for transmitting the generated proposals to a user terminal and presenting them to the user; and means for providing assistance for programming classes. This enables effective product and service proposals based on the user's purchase history and preferences, improving the user experience through an interactive UI and providing assistance functions for programming classes.

[1810] An "API for generative AI" is an application programming interface that allows external use of the functions of generative artificial intelligence.

[1811] "Conversational UI" is a user interface in which the user and the system communicate two-way using natural language.

[1812] "User purchase history" refers to a record of products and services purchased by a user in the past.

[1813] A "database" is a system for efficiently storing, managing, and retrieving data.

[1814] A "data analysis tool" is software used to analyze collected data and extract useful information.

[1815] A "generative AI model" is a model that uses artificial intelligence to generate text and suggestions.

[1816] A "user terminal" is a device such as a computer or smartphone that can be directly operated by a user.

[1817] "Support for programming classes" refers to functions and tools to support programming education.

[1818] This invention is a customer service / recommendation system that suggests products and services based on a user's purchasing history and preferences. The system utilizes a generative AI API, enabling the user to switch from a touch panel-based operation to a conversational UI based on generative AI. It also includes support functions for programming classes.

[1819] Hardware and software used

[1820] Hardware: Servers, user devices (PCs, smartphones, etc.)

[1821] Software: Database management systems (e.g., MySQL), generative AI models (e.g., GPT-4), data analysis tools (e.g., Python libraries)

[1822] Data collection

[1823] The server collects data about the user's purchasing history and preferences and stores it in a database. For example, if a user purchases running shoes, the server stores data such as "Running shoes, October 1, 2023, $100."

[1824] Data analysis

[1825] The server retrieves the user's purchasing history and preference data from the database and analyzes it using data analysis tools. Specifically, it uses a Python library to analyze the user's purchasing history and calculate the purchase frequency and trends of specific products. For example, the server analyzes the user's purchasing history over the past six months and determines that sporting goods are frequently purchased.

[1826] Proposal generation

[1827] The server uses a generative AI model (e.g., GPT-4) to generate product and service suggestions based on the analysis results. An example of a prompt to the generative AI model is, "Please suggest new sports equipment based on User A's purchasing history." An example of a generated suggestion is, "Dear User A, you have previously purchased running shoes and sportswear. We have new running shoes and the latest sportswear in stock, so please take a look."

[1828] Presenting the proposal

[1829] The server sends the generated suggestions to the user's device and presents them to the user. Specifically, the server sends the generated suggestions to the user's smartphone or PC via push notification or email. For example, the server displays a message on the user's smartphone saying, "New running shoes have arrived. Try them out!"

[1830] In this way, the system can suggest appropriate products and services based on the user's needs. The conversational UI based on generative AI also enhances the user experience. Furthermore, it also provides support functions for programming classes, making it suitable for use in educational settings.

[1831] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1832] Step 1: Data collection

[1833] The server collects data on users' purchasing history and preferences and stores it in a database.

[1834] Input: Information when the user purchases a product (product name, purchase date and time, price, etc.)

[1835] What happens: If a user buys running shoes, the server records data like "Running shoes, October 1, 2023, $100" in a database.

[1836] Output: Purchase history data stored in a database

[1837] Step 2: Data analysis

[1838] The server retrieves data on users' purchasing history and preferences from the database and analyzes it using data analysis tools.

[1839] Input: User purchase history data stored in a database

[1840] How it works: The server uses Python libraries to analyze a user's purchase history and calculate the frequency and trends of purchases of specific products. For example, it analyzes purchase history for the past six months and finds that sporting goods are frequently purchased.

[1841] Output: Analysis results (e.g., information that the user frequently purchases sports equipment)

[1842] Step 3: Proposal Generation

[1843] The server uses a generative AI model (e.g., GPT-4) to generate product and service suggestions based on the analysis results.

[1844] Input: Analysis results (user purchasing trends)

[1845] Specific operation: The server inputs a prompt sentence into the generative AI model and generates appropriate suggestions. An example of a prompt sentence is, "Please suggest new sporting goods based on user A's purchasing history."

[1846] Output: Generated suggestions (e.g. "User A, you have purchased running shoes and sportswear in the past. Please take a look at our new running shoes and latest sportswear.")

[1847] Step 4: Present your proposal

[1848] The server transmits the generated proposal to the user terminal and presents it to the user.

[1849] Input: Generated proposals

[1850] Specific operation: The server sends the generated suggestions to the user's smartphone or PC via push notification or email. For example, the user's smartphone displays a message saying, "New running shoes have arrived. Try them out!"

[1851] Output: The proposal message displayed on the user's terminal

[1852] In this way, the system can suggest appropriate products and services based on the user's needs.

[1853] (Application example 2)

[1854] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1855] Conventional customer service / recommendation systems were unable to fully utilize users' purchasing history and preferences, making it difficult to propose individually customized products and services. Furthermore, the accuracy of product recommendations based on user needs was low, making it difficult to improve user satisfaction. Furthermore, there was no established method for product recommendations based on similarity, making it difficult to propose appropriate products to users.

[1856] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1857] In this invention, the server includes a means for using the generative AI API to enable a change from touch panel-based operation to a generative AI-based interactive UI, a means for customer service / suggestion, a means for providing assistance for programming classes, a means for analyzing the user's purchase history and preferences and proposing individually customized products and services, and a means for proposing products based on similarity. This enables individually customized product suggestions based on the user's purchase history and preferences, which is expected to improve user satisfaction. Furthermore, product suggestions based on similarity enable more appropriate product suggestions.

[1858] An "API for generative AI" is an application programming interface (API) that allows external use of the functions of generative artificial intelligence (AI).

[1859] A "conversational UI" is a user interface (UI) that allows users and systems to communicate interactively using natural language.

[1860] "Means of customer service / suggestion" refers to the means of introducing products and services to users and encouraging them to purchase.

[1861] "Means for providing assistance to programming classes" are means for supporting programming education, enabling learners to learn programming efficiently.

[1862] "Means for analyzing user purchasing history and preferences" refers to the means for collecting and analyzing data on the history of products and services purchased by users in the past and their preferences.

[1863] "Means for proposing individually customized products and services" refers to means for proposing products and services optimized for each user based on the user's purchasing history and preferences.

[1864] "Means for suggesting products based on similarity" refers to a means of analyzing a user's purchasing history and preference data, calculating the similarity with other users, and suggesting products purchased by users with similar tastes.

[1865] A system for implementing this invention includes a means for utilizing a generative AI API to enable a change from touch panel-based operation to an interactive UI based on generative AI, a means for providing customer service / suggestions, a means for providing assistance in programming classes, a means for analyzing a user's purchasing history and preferences and suggesting individually customized products and services, and a means for suggesting products based on similarity.

[1866] System Program

[1867] The system is implemented using Python. Specifically, it manages user purchase history data, vectorizes the data, and calculates similarity. The part that interacts with users uses a generative AI API, employing a generative AI model such as OpenAI's GPT-3.

[1868] Program processing explanation

[1869] The server collects users' purchase history data and converts it into a data frame. Then, it combines the purchase history data as text and vectorizes it. This allows it to calculate the similarity of the purchase history between users using cosine similarity. Based on this similarity, it suggests products to specific users.

[1870] Specifically, based on the user's past purchase history of products and services, the system suggests products purchased by other users with similar tastes. For example, if a user has previously purchased "sports shoes" and "running wear," related products such as "yoga mats" and "swimming goggles" will be suggested based on the purchasing history of similar users.

[1871] Hardware and software used

[1872] Hardware: General PCs and servers

[1873] Software: Python, generative AI API (e.g., OpenAI's GPT-3)

[1874] Prompt Sentence Examples

[1875] Examples of prompts to input to a generative AI model include:

[1876] Suggest relevant products based on the user's purchasing history. For example, if the user has purchased sporting goods in the past, suggest new sporting goods and related products.

[1877] In this way, it becomes possible to make individually customized product suggestions based on the user's purchasing history and preferences, which is expected to improve user satisfaction.In addition, product suggestions based on similarity will enable more appropriate product suggestions.

[1878] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1879] Step 1:

[1880] The server collects user purchase history data. Specifically, it retrieves information such as user ID, product ID, and product name from the database and converts it into a data frame. The input is purchase history data from the database, and the output is purchase history data in data frame format.

[1881] Step 2:

[1882] The server combines the purchase history data as text. Specifically, it aggregates the purchase history data of the same user into a single text data. The input is purchase history data in data frame format, and the output is the purchase history text for each user.

[1883] Step 3:

[1884] The server vectorizes the purchase history text. Specifically, it converts the purchase history text into a TF-IDF (Term Frequency-Inverse Document Frequency) vector. The input is the purchase history text for each user, and the output is a TF-IDF vector.

[1885] Step 4:

[1886] The server calculates the cosine similarity. Specifically, it calculates the similarity of purchase histories between users using the cosine similarity. The input is a TF-IDF vector, and the output is the similarity score between users.

[1887] Step 5:

[1888] The server proposes products based on similarity. Specifically, for a specific user, it proposes products purchased by other users who are highly similar to the user. The input is the similarity score between users and their purchase history data, and the output is a list of suggested products.

[1889] Step 6:

[1890] The server uses the API of generative AI to interact with the user. Specifically, it uses a generative AI model (e.g., GPT-3) to generate an appropriate response to the user's input. The input is the user's input text and prompt sentence, and the output is the generated response text.

[1891] Step 7:

[1892] The terminal displays the suggested product list received from the server and the generated response text to the user. Specifically, the suggested product list and the dialogue content are displayed on the screen of a smartphone or PC. The input is the suggested product list from the server and the generated response text, and the output is the information displayed to the user.

[1893] Example 3

[1894] Next, a third embodiment of the third embodiment will be described. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1895] In conventional programming classes, it is difficult for users to receive appropriate support when working on assignments, which often slows down learning progress, especially for beginners. Furthermore, there are limited ways for users to get instant hints when they are stuck on an assignment, which reduces learning efficiency. Furthermore, there is a lack of systems that can accurately check users' answers and provide appropriate feedback. To solve these issues, a support system that allows users to learn programming efficiently is needed.

[1896] The specific processing by the specific processing unit 290 of the data processing device 12 in the third embodiment is realized by the following means.

[1897] In this invention, the server includes a means for a user to log in, a means for the server to issue a question, a means for the user to work on the question, a means for the user to submit an answer, a means for the server to check the answer, a means for the server to provide feedback, a means for the user to request a hint, and a means for the server to provide the hint, thereby enabling a support system that allows users to efficiently learn programming.

[1898] An "API for generative AI" is an application programming interface that allows external use of the functions of generative artificial intelligence.

[1899] "Conversational UI" is a user interface that allows users and systems to communicate two-way using natural language.

[1900] "Means of customer service / suggestion" is a function for suggesting products and services based on the user's needs.

[1901] "Means for providing assistance for programming classes" refers to a function for supporting programming learning.

[1902] "Means for users to log in" refers to the function by which users access the system and perform authentication.

[1903] "Means for the server to provide assignments" refers to the function that enables the server to select programming assignments from the database and present them to the user.

[1904] "Means for users to tackle the assignment" refers to a function that allows users to create a program for the assignment presented to them.

[1905] "Means for users to submit answers" is a function that allows users to send the programs they have created to the server.

[1906] "Means for the server to check answers" refers to the function that the server uses to analyze the accuracy of the programs submitted by users.

[1907] The "means for the server to provide feedback" is a function that allows the server to provide appropriate feedback to the user based on the analysis results of the answer.

[1908] "Means for users to request hints" is a function that allows users to request hints from the server when they are having trouble with a task.

[1909] "Means for the server to provide hints" refers to a function that allows the server to use generative AI to provide appropriate hints to users.

[1910] This invention is an auxiliary system for programming classes that helps users to learn programming efficiently. The system operates in cooperation with three parties: a server, a terminal, and a user.

[1911] The server provides a web interface for users to access. Through this interface, users can complete programming challenges. The server issues the challenges, checks the answers, and provides hints. Specifically, the server does the following:

[1912] 1. Assignment: The server presents the user with programming assignments stored in the database. For example, it assigns an assignment on loop processing in Python.

[1913] 2. Checking the answers: The server receives the code submitted by the user and checks its accuracy using a Python code analysis library.

[1914] 3. Providing Hints: When a user has difficulty with a task, the server generates a hint using a generative AI model (e.g., OpenAI's GPT-3). For example, the server inputs a prompt such as "Please tell me the basic usage of for loops" into the generative AI model and presents the obtained answer to the user.

[1915] A user accesses the system using a terminal and enters their username and password on the login screen. The server receives this and authenticates them by checking the database. If authentication is successful, the server retrieves the user's progress from the database and selects the appropriate task.

[1916] As users work on the assignment, the server automatically saves their code in real time, ensuring that progress isn't lost if the user accidentally closes their browser. Once the user submits their solution, the server instantly analyzes the code and generates feedback, such as "The code is correct. All numbers were printed correctly."

[1917] When a user requests a hint, the server sends a prompt to the generative AI model saying, "Please tell me the basic usage of for loops," and provides the hint within a few seconds. Specifically, the server provides the following hints:

[1918] A for loop is used to cycle through numbers in a specified range. In this example, we'll print the numbers 1 through 10.

[1919] In this way, the server, the terminal, and the user can cooperate to efficiently advance programming learning. The flow of the specific processing in the third embodiment will be described with reference to FIG.

[1920] Step 1:

[1921] A user logs in

[1922] A user accesses the system using a terminal and enters a username and password on the login screen. The server receives this and authenticates it by checking it against the database. The input is the username and password, and the output is the authentication result. If authentication is successful, the user's progress is retrieved from the database.

[1923] Step 2:

[1924] The server issues the assignment

[1925] The server selects a programming task from the database and presents it to the user. The input is the user's progress, and the output is the selected task. For example, the task might be "Write a program that uses a Python for loop to output the numbers from 1 to 10."

[1926] Step 3:

[1927] User tackles the problem

[1928] Users work on assignments on their devices and write programs using a code editor. The input is the assignment and the output is the code written by the user. For example, you might write code like this:

[1929] python

[1930] for i in range(1, 11):

[1931] print(i)

[1932] Step 4:

[1933] User submits answer

[1934] The user submits the answer to the server from their device. When they click the submit button, the code is sent to the server. The input is the code created by the user, and the output is the code sent to the server.

[1935] Step 5:

[1936] The server checks the answer

[1937] The server analyzes the received code and checks its accuracy. It uses a Python code analysis library to do this. The input is the code submitted by the user, and the output is the analysis result. For example, it checks whether the code is error-free and whether it produces the expected output.

[1938] Step 6:

[1939] Server provides feedback

[1940] The server generates feedback based on the check results and presents it to the user. The input is the analysis result, and the output is a feedback message. For example, it provides feedback such as "The code is correct. All numbers were output correctly."

[1941] Step 7:

[1942] The user requests a hint (if necessary)

[1943] When a user has difficulty with a task, they can request a hint from the server. For example, they can input "I don't understand the loop processing." The input is the user's hint request, and the output is a hint request message.

[1944] Step 8:

[1945] The server provides a hint

[1946] The server generates hints using a generative AI model (e.g., OpenAI's GPT-3). For example, the server inputs a prompt such as "Please tell me the basic usage of a for loop" into the generative AI model, and presents the resulting answer to the user. The input is the prompt, and the output is the generated hint. Specifically, the server provides the following hints:

[1947] A for loop is used to cycle through numbers in a specified range. In this example, we'll print the numbers 1 through 10.

[1948] (Application example 3)

[1949] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1950] Previous support systems for programming classes lacked the functionality to provide appropriate hints when users were struggling with a task, resulting in reduced learning efficiency. Furthermore, when learning to program factory robots, there was a lack of support systems that allowed users to learn while actually operating the robots. This made it difficult for users to efficiently acquire programming skills.

[1951] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 3 is realized by the following means.

[1952] In this invention, the server includes a means for using the API of the generative AI to enable a change from touch panel-based operation to an interactive UI based on the generative AI, a means for providing customer service / suggestions, a means for providing assistance in programming classes, a means for supporting factory robot programming learning, a means for checking the user's answers, and a means for providing hints when the user is having trouble with a task, thereby enabling the user to efficiently acquire programming skills.

[1953] An "API for generative AI" is an application programming interface that allows external use of the functions of generative artificial intelligence.

[1954] "Touch panel-based operations" refers to all operations performed by users using a touch panel.

[1955] "Interactive UI based on generative AI" is a user interface that interacts with the user based on generative artificial intelligence.

[1956] "Means of customer service / suggestion" refers to means that have the function of suggesting products and services based on the user's needs.

[1957] "Means for providing assistance for programming classes" are means that have the function of supporting learners in programming classes.

[1958] The "means for supporting learning programming for factory robots" is a means having a support function for learning programming for robots used in factories.

[1959] "Means for checking the user's answers" refers to means that has the function of checking whether the answers entered by the user are correct.

[1960] "Means for providing hints when a user is having trouble with a task" refers to means that has the function of providing hints for solving a task when a user is having difficulty with the task.

[1961] The system for implementing this invention utilizes a generative AI API and includes means for enabling a change from touch panel-based operation to an interactive UI based on generative AI, means for providing customer service / suggestions, means for providing assistance in programming classes, means for supporting learning programming for factory robots, means for checking the user's answers, and means for providing hints when the user is having trouble with a task.

[1962] System configuration

[1963] Hardware

[1964] Server: Hosts the generative AI API and generates appropriate responses to user input.

[1965] Device: A device that a user operates, such as a smartphone, tablet, or factory robot.

[1966] Factory robot: A robot that can be used to learn programming.

[1967] software

[1968] Generative AI API: An application programming interface for externally using the functions of generative artificial intelligence.

[1969] Conversational UI: A user interface that interacts with users based on generative AI.

[1970] Programming learning support application: An application to support learning factory robot programming.

[1971] Data processing and calculation

[1972] Assignment questions

[1973] The server uses the generative AI API to present users with programming challenges, such as "making a robot move forward."

[1974] Answer check

[1975] The server receives the answers entered by the user through the terminal and executes them on the factory robot, checks whether the robot operates correctly, and provides feedback on the results to the user.

[1976] Hints provided

[1977] If the user has difficulty with a task, the server uses the generative AI API to generate an appropriate hint and provide it to the user, for example, "Use the forward command."

[1978] Specific examples

[1979] For example, if a user answers "move forward" to the task "move the robot forward," the server sends the answer to the factory robot and checks whether the robot actually moves forward. If the answer is correct, the robot moves on to the next task; if the answer is incorrect, the server provides a hint: "Use the move forward command."

[1980] Prompt Sentence Examples

[1981] Challenge: Moving the robot forward

[1982] Enter your answer: Advance

[1983] In this way, users can efficiently acquire programming skills.

[1984] The flow of the specific processing in Application Example 3 will be described with reference to FIG.

[1985] Step 1:

[1986] The server uses a generative AI API to present programming challenges to users.

[1987] Input: User's learning progress data

[1988] Data processing: Selecting appropriate tasks based on learning progress data

[1989] Output: The text of the task (e.g., "Move the robot forward")

[1990] Specific operation: The server analyzes the user's past answer history and progress, and generates a challenge of appropriate difficulty. The generated challenge is sent to the device and displayed to the user.

[1991] Step 2:

[1992] Users input their answers to the questions through the terminal.

[1993] Input: Assignment text, user answer

[1994] Data processing: Receive user answers in text format

[1995] Output: User's answer data

[1996] Specific operation: The user enters the answer in the input field on the device and presses the send button. The answer data is sent to the server.

[1997] Step 3:

[1998] The server receives the user's answer and has the factory robot execute it.

[1999] Input: User's answer data

[2000] Data processing: Converting answer data into robot control commands

[2001] Output: Robot control commands

[2002] How it works: The server analyzes the user's answers and converts them into control commands that the robot can understand. The converted commands are then sent to the factory robot.

[2003] Step 4:

[2004] The factory robot operates based on the control commands it receives.

[2005] Input: Robot control command

[2006] Data processing: Execute control commands

[2007] Output: Robot movement results

[2008] Specific operation: The factory robot executes the control command received from the server and performs the specified operation (e.g., moving forward).

[2009] Step 5:

[2010] The server checks the results of the robot's actions and provides feedback to the user.

[2011] Input: Robot action result

[2012] Data processing: Evaluation of operational results

[2013] Output: A feedback message (e.g., "Correct!" or "Incorrect. Hint: Use the forward command")

[2014] Specific behavior: The server receives the robot's behavior results and evaluates whether they meet the requirements of the task. Based on the evaluation results, it generates a feedback message for the user and sends it to the device.

[2015] Step 6:

[2016] When a user has difficulty with a task, the server uses a generative AI API to generate appropriate hints and provide them to the user.

[2017] Input: User answer data, assignment text

[2018] Data processing: Generate hints based on answer data and assignment text

[2019] Output: Hint message (e.g. "Use forward command")

[2020] Specific operation: The server analyzes the user's answer data and the task text, and generates appropriate hints using the generative AI API. The generated hints are sent to the device and displayed to the user.

[2021] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[2022] "Example 1"

[2023] One embodiment of the present invention is a system that includes an emotion engine in which a generative AI API recognizes a user's emotions. This system reads emotions from the user's tone of voice, facial expressions, and choice of words, and then engages in dialogue accordingly. For example, if the AI ​​senses that the user is angry, it chooses words to apologize, and if it senses that the user is happy, it chooses words to share that joy.

[2024] "Example 2"

[2025] Another embodiment of the present invention is a customer service / suggestion method using an emotion engine. This system reads the user's emotions and suggests products and services that match those emotions. For example, if the user feels depressed, the AI ​​will suggest products and services that will cheer them up. Conversely, if the user feels happy, the AI ​​will suggest products and services that will amplify that joy.

[2026] "Example 3"

[2027] Another embodiment of the present invention is a programming classroom assistant using an emotion engine. This system reads the user's emotions and provides appropriate learning materials based on those emotions. For example, if the user feels confused, the AI ​​will provide more basic learning materials. Conversely, if the user feels confident, the AI ​​will provide more advanced learning materials.

[2028] The processing flow of each embodiment will be described below.

[2029] "Example 1"

[2030] Step 1: Through dialogue with the user, read their emotions from their tone of voice, facial expressions, choice of words, etc.

[2031] Step 2: The AI ​​adjusts the content of the dialogue based on the emotions it reads.

[2032] Step 3: For example, if the user feels angry, the AI ​​will choose words to apologize, and if the user feels happy, the AI ​​will choose words to share that joy.

[2033] "Example 2"

[2034] Step 1: Interact with the user to understand their emotions.

[2035] Step 2: The AI ​​tailors product and service recommendations based on the emotions it detects.

[2036] Step 3: For example, if the user feels depressed, the AI ​​will suggest products and services that will cheer them up; if the user feels happy, the AI ​​will suggest products and services that will amplify that joy.

[2037] "Example 3"

[2038] Step 1: Interact with the user to understand their emotions.

[2039] Step 2: The AI ​​adjusts the educational material provided based on the emotions it reads.

[2040] Step 3: For example, if the user feels confused, the AI ​​will provide more basic learning materials, and if the user feels confident, the AI ​​will provide more advanced learning materials.

[2041] Example 1

[2042] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[2043] Conventional touch panel-based operation systems have the problem that it is difficult for users to operate intuitively, and operation is cumbersome, especially for elderly people and users who are unfamiliar with technology. In addition, the user experience is limited because it is not possible to have a dialogue that takes into account the user's emotions. Furthermore, there is a lack of systems that utilize voice input and emotion recognition, and more natural interaction is required.

[2044] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[2045] In this invention, the server includes: a means for utilizing a generative AI API to enable a change from touch panel-based operation to an interactive user interface based on generative AI; a means for providing customer service / suggestions; a means for providing assistance for programming education; a means for collecting user voice input and sending it to the server; a means for the server to convert the voice data into text and analyze it; a means for capturing the user's facial expressions and analyzing their emotions; a means for determining and executing appropriate operations based on the analysis results; a means for generating dialogue using generative AI; and a means for presenting the results to the user. This allows the user to issue operation instructions in natural language, and the system can engage in dialogue that corresponds to the user's emotions.

[2046] "API for generative AI" refers to the application programming interface for generative artificial intelligence, an interface that allows external software and systems to use the functions of generative AI.

[2047] An "interactive user interface" is an interface that allows a user to interact with a system using natural language and to perform operations and give instructions.

[2048] "Means of customer service / suggestion" refers to the function of suggesting products and services based on the u...

Claims

1. A first means for collecting and storing in a database data on a user's purchase history and preferences; a second means for generating a first prompt sentence for outputting a product or service suggestion to the user based on the analysis of the user's purchase history and preference data stored in the database; a third means for inputting the generated first prompt sentence into a generative AI model to generate a product or service suggestion for the user; A fourth means for transmitting the generated product or service proposal to the user to a terminal of the user and presenting the product or service proposal to the user; A system including:

2. a fifth means for analyzing an emotion of the user from the voice data of the user using an emotion engine; The second means generates a second prompt sentence for outputting a product or service suggestion that matches the user's emotion based on the analysis result of the user's purchase history and preference data and the user's emotion; The third means inputs the generated second prompt sentence into a generative AI model to generate a product or service proposal that matches the user's emotions. The system of claim 1 .

3. The third means generates a suggestion for a product or service that cheers up the user when the emotion engine determines that the user's emotion is depressed, and generates a suggestion for a product or service that enhances the user's joy when the emotion engine determines that the user's emotion is happy. The system of claim 2.

Images