System

Abstract

A system is provided.SOLUTION: A system, comprising: means for obtaining user interface data; means for collecting an operation log of a user in real time; means for analyzing the operation log to identify a trip of the user associated with a particular operation; means for generating tutorial information corresponding to the trip; and means for providing the tutorial information to the user.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In recent years, games have become increasingly multifunctional, and it is common for tutorials to take at least 30 minutes. As a result, while users want to start playing the game immediately, long tutorials often prevent them from fully memorizing the content, and as a result, they often have to search for the information again as they play. This also increases game development costs and worsens the user experience. To solve these problems, we propose a system that allows users to receive operation instructions at their own pace whenever they need them. [Means for solving the problem]

[0005] In order to solve the above problems, the present invention provides the following system.

[0006] a means for obtaining user interface data;

[0007] A means of collecting user operation logs in real time,

[0008] A means for analyzing the operation log and identifying user stumbling blocks related to specific operations;

[0009] means for generating tutorial information corresponding to the stumbling block;

[0010] The system includes a means for providing the tutorial information to a user.

[0011] This system allows users to avoid lengthy tutorials and receive appropriate instructions when they need them. Furthermore, by providing tutorial information via voice, users can solve problems in real time, improving the gaming experience.

[0012] "User interface data" refers to the configuration information of the screens and menus that users operate when playing a game.

[0013] A "user operation log" is data that records a series of operations performed by a user while playing a game, and includes information such as which buttons were pressed and which menus were opened.

[0014] "Means of collecting data in real time" refers to methods and technologies for instantly recording and continuously updating user operation logs without delay.

[0015] "Analysis" refers to the process of finding specific information or trends based on collected data; in this case, the purpose is to examine user operation logs to identify stumbling blocks.

[0016] "Stumble" refers to a situation in which a user is unable to understand a particular in-game operation, causing their progress to slow down or stop.

[0017] "Tutorial information" refers to instructions that include explanations and procedures for game operation, and is information that allows users to learn the correct operation within the game.

[0018] "Means for generating" refers to methods or techniques for producing predetermined information based on specific data or conditions.

[0019] "Means of providing" refers to the methods and technologies for appropriately delivering the generated information to the user, and in this case includes presentation by voice, text, etc. [Brief explanation of the drawings]

[0020] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11]FIG. 3 is a sequence diagram illustrating a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0021] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0022] First, the terms used in the following description will be explained.

[0023] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0024] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0025] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0026] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0027] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0028] [First embodiment]

[0029] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0030] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0031] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0032] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0033] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0034] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0035] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0036] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0037] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0038] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0039] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0040] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0041] The following describes an embodiment of the present invention. Specifically, the present invention provides a means and method for realizing an AI tutorial system that provides users with real-time instructions on how to play a game.

[0042] server

[0043] The server connects to the game company's API to obtain user interface data and tutorial data. It trains an AI model based on the obtained data and prepares to analyze user operation logs. It also collects user operation logs sent from the device in real time and detects stumbling blocks related to specific operations. It then generates appropriate tutorial information corresponding to the stumbling block and sends it to the device.

[0044] For example, the server sends API requests to retrieve game operation and tutorial data, stores this data in an internal database, and uses machine learning algorithms to train an AI model, which is then used to analyze user operation logs sent in real time and detect stumbling points.

[0045] Terminal

[0046] The device collects user operation logs in real time and sends them to the server. It records each operation performed by the user (e.g., selecting an item, moving a character, opening and closing a menu, etc.) and periodically sends them to the server. It also receives tutorial information sent from the server and provides it to the user by audio guide or other appropriate means.

[0047] For example, if a user repeatedly opens and closes the same menu in an attempt to change equipment in a game, the operation log is collected in real time and sent to the server. The server analyzes this log, determines that the user does not understand how to change equipment, and sends appropriate tutorial information (e.g., "Press the '+' button to change equipment") to the device. The device then provides this information to the user as audio guidance.

[0048] User

[0049] The user starts the game and continues playing at their own pace. If the user does not know how to perform a particular operation or gets stuck, the system will automatically provide assistance. For example, if the user does not know how to change equipment, the system will automatically detect that the user has repeatedly opened and closed the same menu. The device will provide a voice guide, saying, "You can change it by pressing the '+' button." The user can follow this guide to change their equipment and progress smoothly through the game.

[0050] As described above, the AI ​​tutorial system of the present invention significantly improves the user experience by collecting and analyzing user operation logs in real time and providing the necessary information when needed. This system can be applied to various game platforms, including smartphone games and console games, and will be a useful tool for both game developers and users.

[0051] The processing flow will be explained below.

[0052] Step 1:

[0053] The server connects to the game company's API to retrieve user interface data and tutorial data. Specifically, it sends an API request, receives operation data and tutorial data, and stores it in an internal database.

[0054] Step 2:

[0055] The server trains the AI ​​model based on the acquired data. Specifically, it uses a machine learning algorithm to learn the relationship between the operation data and the results (whether or not there is a stumble). This model is then used.

[0056] Step 3:

[0057] The terminal collects user operation logs in real time and sends them to the server. Specifically, it records user operations (e.g., button presses, menu selections) in a log file and sends the log to the server at regular intervals.

[0058] Step 4:

[0059] The server analyzes the operation logs received from the device in real time. Specifically, it inputs the operation data into an AI model to detect user stumbling. If a certain threshold is exceeded, it is determined to be a stumbling.

[0060] Step 5:

[0061] The server generates tutorial information corresponding to the stumbling point. Specifically, it obtains tutorial data corresponding to the operation data (e.g., "Press the '+' button to change equipment") and sends it to the device.

[0062] Step 6:

[0063] The device receives the tutorial information sent from the server. Specifically, it receives messages from the server using a real-time communication protocol and passes them to the voice assistant system.

[0064] Step 7:

[0065] The device provides the received tutorial information to the user by voice. Specifically, it uses a voice synthesis engine to output the received tutorial information by voice.

[0066] Step 8:

[0067] The user follows the voice guidance to perform operations, specifically, to perform operations within the game according to the presented operating procedures.

[0068] Step 9:

[0069] The device then collects the user's actions after the voice guidance as a log and sends it to the server. Specifically, it records the user's actions after the voice guidance and sends them to the server, contributing to improving the accuracy of the AI ​​model.

[0070] Example 1

[0071] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0072] In conventional game systems, if a user gets stuck on a specific operation, proper support is not provided in real time, which can lead to a poor user experience. Furthermore, delayed feedback on operations can prevent users from continuing the game. Furthermore, existing tutorial systems are often ineffective because they do not take into account individual user behavior patterns and can only provide uniform explanations.

[0073] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0074] In this invention, the server includes means for acquiring user interface data, means for collecting user operation logs in real time, means for analyzing the operation logs and identifying user stumbling blocks related to specific operations, means for generating tutorial information corresponding to the stumbling blocks, means for providing the tutorial information to the user, means for training an AI model using a machine learning algorithm, means for the server to collect and analyze operation logs transmitted from the terminal in real time, and means for the terminal to provide the tutorial information to the user by audio guidance or other means. This makes it possible to provide prompt and appropriate support based on individual behavioral patterns in real time when the user stumbling blocks with a specific operation, improving the user experience.

[0075] "User interface data" refers to data that indicates information about the interface that users use to operate within a game or application.

[0076] A "user operation log" is data that records the operations performed by a user within a game or application.

[0077] "User difficulties related to specific operations" refers to users being unable to perform specific operations correctly, resulting in repeated operations or slow progress.

[0078] "Tutorial information" is information that includes explanations and guides to help users progress smoothly through a game or application.

[0079] A "machine learning algorithm" is a computational method for automatically finding optimal solutions based on data, and is used to train AI models.

[0080] An "AI model" is a data structure that has been trained using machine learning algorithms to generate appropriate outputs for specific inputs.

[0081] A "terminal" is a device that a user uses to operate a game or application, and includes a smartphone, tablet, computer, etc.

[0082] The "server" is a computer system that collects and analyzes user operation logs and provides appropriate tutorial information to users.

[0083] "Audio description" refers to explanatory or instructional information provided aloud to help the user understand a particular operation.

[0084] To practice the present invention, the following procedures and means are used.

[0085] server

[0086] The server first obtains the user interface data and tutorial data through the game company's API. Specifically, it issues a request to the following API endpoint to obtain this data.

[0087] User Interface Data: https: / / api.gamecompany.com / ui_data

[0088] Tutorial Data: https: / / api.gamecompany.com / tutorial_data

[0089] The acquired data is stored in the server's internal database. Next, this data is used to train an AI model using a machine learning algorithm (for example, TensorFlow or PyTorch). During this process, user data is preprocessed and then used to train the model.

[0090] The server also has a means for collecting user operation logs in real time. It receives the operation logs sent from the terminal and analyzes them sequentially. The purpose of the analysis is to identify operations that the user is having trouble with. For example, if the user repeatedly opens and closes the same menu, the server will determine that the operation is difficult for the user.

[0091] Based on the analysis results, the server generates appropriate tutorial information, such as "Press the '+' button to change your equipment," and sends it to the device.

[0092] Terminal

[0093] The device records the user's operation log in real time and periodically sends it to the server. For example, the user's operation of opening the "Settings Menu" in the game is recorded as follows:

[0094] json

[0095] {

[0096] "timestamp": "2023-10-01T12:34:56Z",

[0097] "action": "menu_open",

[0098] "details": "settings_menu"

[0099] }

[0100] This log data is sent to the server in JSON format.

[0101] The device also receives tutorial information sent from the server and provides it to the user as audio guidance or a pop-up message. For example, when receiving tutorial information for changing equipment, the device will play audio guidance such as "Press the '+' button to change equipment."

[0102] User

[0103] The user starts the game and plays freely. If the user is unsure of how to operate the game or has trouble with a particular operation, the system provides real-time support. If the user has trouble with an operation related to changing equipment, the device will provide guidance on how to change equipment. For example, a voice guide will be played saying, "You can change equipment by pressing the '+' button." This allows the user to smoothly change equipment and continue playing the game.

[0104] Specific examples

[0105] If a user does not know how to change equipment and repeatedly opens and closes the same menu, the operation log is sent to the server in real time. The server analyzes the log and determines that the user does not understand how to change equipment. It generates tutorial information such as "Press the '+' button to change equipment" and sends it to the device. The device then provides this information to the user as audio guidance.

[0106] Prompt Sentence Examples

[0107] How do I change equipment in this game?

[0108] "Please provide tutorial information on how to move new characters."

[0109] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0110] Step 1:

[0111] The server retrieves the user interface and tutorial data.

[0112] Specifically, the server sends GET requests to the API endpoints https: / / api.gamecompany.com / ui_data and https: / / api.gamecompany.com / tutorial_data. The input is the API request, and the output is the user interface data and tutorial data returned as a response. These data are stored in an internal database.

[0113] Step 2:

[0114] The server trains the AI ​​model

[0115] Specifically, the server preprocesses the acquired user interface data and tutorial data and applies machine learning algorithms using TensorFlow or PyTorch. The input is the aforementioned dataset, and the output is a trained AI model, which is later used to analyze operation logs.

[0116] Step 3:

[0117] The device collects and sends user operation logs

[0118] Specifically, the device records each operation performed by the user (e.g., opening and closing menus, selecting items, moving characters). For example, the operation of opening the "Settings menu" is recorded as follows:

[0119] json

[0120] {

[0121] "timestamp": "2023-10-01T12:34:56Z",

[0122] "action": "menu_open",

[0123] "details": "settings_menu"

[0124] }

[0125] The input is the user's operation, and the output is an operation log that records this operation. This log data is sent to the server.

[0126] Step 4:

[0127] The server analyzes the operation log.

[0128] Specifically, the server receives operation logs sent from the device in real time and analyzes them using a trained AI model. The input is the operation log, and the output is the identification of operations where the user is having trouble. For example, it can detect when the same menu is repeatedly opened and closed.

[0129] Step 5:

[0130] The server generates the tutorial information

[0131] Specifically, the server generates appropriate tutorial information based on the analysis results. For example, if it detects a problem with changing equipment, it generates a message such as "Press the '+' button to change equipment." The input is the analysis results, and the output is the generated tutorial information.

[0132] Step 6:

[0133] The device receives and provides tutorial information

[0134] Specifically, the terminal receives tutorial information sent from the server, and then provides it to the user as voice guidance or pop-up messages. The input is the tutorial information from the server, and the output is guidance provided to the user through visual or auditory means.

[0135] Step 7:

[0136] Users access tutorial information

[0137] Specifically, the user follows the tutorial information provided by the device and attempts to perform an in-game operation again. For example, the user follows the instruction "Press the '+' button to change equipment" and correctly changes equipment. The input is the tutorial information, and the output is the result of the user's successful operation.

[0138] (Application example 1)

[0139] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0140] With conventional tutorial systems, it was difficult to provide real-time assistance for problems users might encounter in games. It was also difficult to provide appropriate assistance in physical stores when customers couldn't find a specific product or were unsure of the purchasing procedure. This could result in a poor user experience and a decrease in sales.

[0141] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0142] In this invention, the server includes means for acquiring user interface data, means for collecting user operation logs in real time, means for analyzing the operation logs to identify user stumbling blocks related to specific operations, means for generating tutorial information corresponding to the stumbling blocks, means for providing the tutorial information to the user, means for collecting user location information and behavioral data, means for analyzing the collected location information and behavioral data to generate location information of an item desired by the user, and means for providing the location information to the user in the form of audio guidance or map information. This makes it possible to resolve in-game stumbling blocks in real time and also enables users to quickly find the items they desire in physical stores, improving the overall user experience.

[0143] "User interface data" refers to information about the screen layout and display elements that users can use to operate the device.

[0144] "Operation log" refers to the operation history and behavioral record of a user.

[0145] A "stumbling block" refers to a point where a user is unable to smoothly perform a particular operation or action and feels difficulty or confusion.

[0146] "Tutorial information" refers to information that provides explanations and guidance necessary for users to perform operations and actions correctly.

[0147] "Location Information" refers to information about a user's current location.

[0148] "Behavioral data" refers to data such as user behavior and movement history.

[0149] "Audio guidance" refers to a feature that provides information and instructions to the user via voice.

[0150] "Map Information" means data containing geographic information about a user's location or destination.

[0151] The present invention relates to a real-time support system for users to search for and appropriately purchase items in a real-world store. The system includes a server, a terminal, and a user as components.

[0152] server

[0153] The server is used to collect and analyze user interface data, operation logs, location information, and behavioral data. Specifically, the server uses the following software and hardware:

[0154] Software: Python and machine learning libraries (TensorFlow, scikit-learn, etc.) are used for data analysis, along with web frameworks such as Django and Flask and databases (PostgreSQL and MySQL) for real-time data processing.

[0155] Hardware: Use high-performance servers and cloud platforms (AWS, Google Cloud, etc.).

[0156] The server analyzes the user's operation log and location information to detect stumbling blocks related to specific operations or actions, and generates appropriate tutorial information and location information corresponding to the detected stumbling block and sends it to the user's device to help the user find the item quickly.

[0157] Terminal

[0158] The device collects user operation logs, location information, and behavioral data in real time and sends them to the server. It also displays tutorial information and location information sent from the server as audio guides and map information.

[0159] Software: The terminal application is implemented as a smartphone app for iOS or Android. It uses the Google Maps API to display map information and a text-to-speech engine for audio guidance.

[0160] Hardware: Smartphones and tablet devices are used.

[0161] Specifically, when a user scans a QR code installed in the store with their smartphone to log in, operation logs and location information are automatically collected. Appropriate guidance and information are sent from the server and presented on the smartphone as audio guidance and a map display.

[0162] User

[0163] Users use this system when searching for an item in a physical store. The system provides appropriate guidance as the user moves around the store and performs operations using a smartphone app. For example, if a user is looking for a specific item but cannot find it, the system analyzes the user's operation log and location information, and provides appropriate instructions as voice guidance and map information.

[0164] Specific examples

[0165] For example, suppose a user is searching for a specific product in a store. If the user repeatedly opens and closes the product category menu on their smartphone, the system will determine that the user cannot find the product. In this case, a voice guide such as "Here is the location of the product" and map information will be automatically displayed. At this time, the server analyzes the user's behavioral data and location information, and generates and sends appropriate guide information.

[0166] Prompt Sentence Examples

[0167] "When a user is searching for a product in a physical store and repeatedly opens and closes the product category menu, please create a system that provides voice guidance and map information to guide the user to the exact location of the product."

[0168] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0169] Step 1:

[0170] The server acquires user interface data. Specifically, it acquires information about the screen layout and display elements that users use through the game company's or store's API, and stores this information in an internal database. This information is used as the basis for analysis in subsequent processing.

[0171] Step 2:

[0172] The device collects user operation logs and location information in real time. Every time a user performs an operation through the smartphone app in the store, the operation history, behavior record, and location information are recorded and sent to the server at regular intervals. Input data includes the operation details, timestamp, and location information, and is sent to the server.

[0173] Step 3:

[0174] The server analyzes the received operation logs and location information, and uses machine learning algorithms (e.g., TensorFlow, scikit-learn) to analyze whether the user is experiencing difficulties with specific operations or locations. This analysis generates output data that identifies the stumbling points.

[0175] Step 4:

[0176] The server generates tutorial information and location information corresponding to the stumbling points. A pre-trained generative AI model is used to identify stumbling points for specific operations and actions. This training data includes various operation and behavior patterns. The server generates appropriate guide information corresponding to the stumbling points and prepares it for subsequent transmission.

[0177] Step 5:

[0178] The server then sends the generated tutorial and location information, including audio guide text and map information, to the device. The data is sent in real time and is immediately available on the user's device.

[0179] Step 6:

[0180] The device then provides the received tutorial information and location information to the user. Specifically, it generates audio guidance using a text-to-speech engine based on the received guide information, or displays map information using the Google Maps API. This information is presented as a pop-up message or audio guidance on the user's screen.

[0181] Step 7:

[0182] The user operates the device based on the provided tutorial information and location information. By following the guide, the user can find items and learn specific operations, thereby streamlining their activities in the store. This feedback is recorded again in the system and used for subsequent analysis and guide generation.

[0183] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0184] The following describes an embodiment of the present invention. Specifically, the present invention provides a means and method for realizing an AI tutorial system that provides users with real-time instructions on how to play a game, recognizes the user's emotions using an emotion engine, and provides appropriate tutorial information based on the emotions.

[0185] server

[0186] The server connects with the game company's API to acquire user interface data and tutorial data. It trains an AI model based on the acquired data and prepares to analyze user operation logs. It also collects user operation logs sent from the device in real time and detects stumbling blocks related to specific operations. It then generates appropriate tutorial information corresponding to the stumbling block and sends it to the device.

[0187] For example, the server sends API requests to retrieve game operation data and tutorial data. This data is stored in an internal database and used to train an AI model using machine learning algorithms. This model is then used to analyze user operation logs sent in real time and detect stumbling points. It also uses an emotion engine to collect and analyze user emotion data.

[0188] Terminal

[0189] The device collects user operation logs in real time and sends them to the server. It records each operation performed by the user (e.g., selecting an item, moving a character, opening and closing a menu, etc.) and periodically sends them to the server. It receives tutorial information sent from the server and provides it to the user via audio guidance or other appropriate means. It also uses an emotion recognition device to detect the user's emotions and sends the data to the server.

[0190] For example, if a user repeatedly opens and closes the same menu in an attempt to change equipment in a game, the operation log is collected in real time and sent to the server. The server analyzes this log, determines that the user does not understand how to change equipment, and sends appropriate tutorial information (e.g., "Press the '+' button to change equipment") to the device. The device then provides this information to the user as audio guidance. Furthermore, if the user is frustrated, adjustments are made based on their emotions, such as adding more helpful explanations.

[0191] User

[0192] The user starts the game and continues playing at their own pace. If the user is unsure of a particular operation or gets stuck, the system will automatically provide assistance. For example, if the user does not know how to change equipment, the system will automatically detect if they repeatedly open and close the same menu. The device will provide voice guidance, saying, "You can change it by pressing the '+' button." The user can follow this guidance to change their equipment and progress smoothly through the game. In addition, if the user wears an emotion-recognition device and the emotion data is transmitted to the system, the system can adjust the tutorial information provided according to the user's emotions. For example, if the user is frustrated, the system can provide more friendly and polite guidance, and if the user is relaxed, it can provide only brief instructions.

[0193] As described above, the AI ​​tutorial system of the present invention not only collects and analyzes user operation logs in real time and provides necessary information when needed, but also further improves the user experience by using an emotion engine to provide individualized instruction tailored to the user's emotional state. This system can be applied to various game platforms, including smartphone games and console games, and will be a useful tool for both game developers and users.

[0194] The processing flow will be explained below.

[0195] Step 1:

[0196] The server connects to the game company's API to retrieve user interface data and tutorial data. Specifically, it sends an API request, receives operation data and tutorial data, and stores it in an internal database.

[0197] Step 2:

[0198] The server trains the AI ​​model based on the acquired data. Specifically, it uses a machine learning algorithm to learn the relationship between the operation data and the results (whether or not there is a stumble). This model is then used.

[0199] Step 3:

[0200] The terminal collects user operation logs in real time and sends them to the server. Specifically, it records user operations (e.g., button presses, menu selections) in a log file and sends the log to the server at regular intervals.

[0201] Step 4:

[0202] The server analyzes the operation logs received from the device in real time. Specifically, it inputs the operation data into an AI model to detect user stumbling. If a certain threshold is exceeded, it is determined to be a stumbling.

[0203] Step 5:

[0204] The device collects user emotional data using an emotion engine that recognizes the user's emotions in real time. Specifically, data such as heart rate, tone of voice, and facial expressions obtained from the emotion recognition device are input into the emotion engine to identify the user's emotional state.

[0205] Step 6:

[0206] The terminal transmits the collected emotion data to the server. Specifically, the emotion data is transmitted to the server using a real-time communication protocol.

[0207] Step 7:

[0208] The server generates appropriate tutorial information based on the operation log and emotional data. Specifically, it combines data that explains the user's difficulties with the user's emotional state to create optimal tutorial information (e.g., "Press the '+' button to change your equipment" or "You're playing calmly today. Press the '+' button here to change your equipment").

[0209] Step 8:

[0210] The server transmits the generated tutorial information to the terminal. Specifically, the server transmits the generated tutorial information to the terminal using a real-time communication protocol.

[0211] Step 9:

[0212] The device receives the tutorial information sent from the server and provides it to the user as audio guidance. Specifically, it uses a speech synthesis engine to output the received tutorial information as audio.

[0213] Step 10:

[0214] The user follows the voice guidance to operate the game. Specifically, the user performs in-game operations according to the presented operation procedures. Furthermore, the user's experience is improved by receiving feedback according to their emotional state.

[0215] Step 11:

[0216] The device then collects the user's actions after the voice guidance as a log and sends it to the server. Specifically, it records the user's actions after the voice guidance and sends them to the server, contributing to improving the accuracy of the AI ​​model.

[0217] Example 2

[0218] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0219] Conventional game tutorial systems have difficulty providing appropriate support in real time when users get stuck on a specific operation. Furthermore, conventional systems provide uniform tutorial information without considering the user's emotional state, which is why improvements to the user experience cannot be expected. This often leads to users abandoning the game midway, and fails to encourage continued play.

[0220] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0221] In this invention, the server includes means for acquiring user interface data, means for collecting user operation logs in real time, means for analyzing the operation logs and identifying user stumbling blocks related to specific operations, means for generating tutorial information corresponding to the stumbling blocks, means for providing the tutorial information to the user, means for collecting user emotion data, and means for adjusting the tutorial information based on the emotion data, thereby enabling appropriate support in real time according to each user's operation and emotional state.

[0222] "User interface data" refers to information about the game's user interface, specifically including data such as screen layout, menu buttons, and operation methods.

[0223] "User operation log" refers to data that records a series of operations performed by a user in a game in chronological order.

[0224] "User stumbling blocks related to specific operations" refers to situations where a user is unable to understand a specific operation and tries repeatedly or becomes stuck for a long period of time.

[0225] "Tutorial Information" refers to guides and explanations provided to help users understand in-game operations.

[0226] "Emotional data" refers to data that indicates the user's emotional state, derived from facial expressions, tone of voice, and other information.

[0227] "Real-time collection means" refers to the technical means for collecting user actions and emotional states on the spot without delay.

[0228] "Means of analysis" refers to algorithms and systems for analyzing collected operation logs and emotional data and extracting meaningful information.

[0229] "Means for generating" refers to technical means for automatically creating tutorial information appropriate for the user based on the analysis results.

[0230] "Adjusting means" refers to a technical means for dynamically changing the content and presentation method of the tutorial information based on the collected emotional data.

[0231] "Means for providing" refers to the technical means for visually or audibly presenting the generated tutorial information to the user.

[0232] An "emotion recognition device" refers to hardware or software used to detect a user's emotional state by analyzing their facial expressions, tone of voice, etc.

[0233] This invention relates to an AI tutorial system that provides necessary operational instructions in real time when a user plays a game, and provides appropriate tutorial information according to the user's emotional state. This system is composed of three main entities: a server, a terminal, and a user.

[0234] Server Processing

[0235] The server first obtains user interface data using the game company's API. This data acquisition is done, for example, via a REST API, and the obtained data is stored in an internal database. This data is then used to train an AI model. Machine learning algorithms such as TensorFlow and PyTorch are used to train the AI ​​model. This model is then used to analyze user operation logs sent in real time and detect stumbling points.

[0236] As a concrete example, the server operates as follows.

[0237] The server sends a GET request to the game API endpoint to retrieve operation data and tutorial data.

[0238] The acquired data is stored in a database.

[0239] Using TensorFlow, training data is input to build, train, and save an AI model.

[0240] The server also collects user operation logs in real time and analyzes the data. The analysis uses WebSocket and HTTP protocols, and the server receives operation logs sent from the device. If a stumbling point is detected as a result of the analysis, the server generates corresponding tutorial information and sends it to the user's device.

[0241] Terminal handling

[0242] The device collects user operation logs in real time and sends them to the server. It records each operation (e.g., item selection, character movement) and periodically sends them to the server. It receives tutorial information sent from the server and displays it as audio guides or pop-up messages. It also uses emotion recognition devices (e.g., camera, microphone) to collect user emotion data and sends it to the server.

[0243] As a specific example, the operation is as follows.

[0244] The terminal records user operations in a log file and periodically sends it to the server.

[0245] The terminal provides tutorial information from the server as an audio guide.

[0246] The emotion recognition device collects the user's emotion data and sends it to the server.

[0247] User operations

[0248] The user starts the game and continues playing at their own pace. If the user is unsure of a specific operation, the system will automatically assist them. For example, if the user does not know how to change equipment, the system will analyze the user's behavior and provide appropriate guidance. In addition, if the user wears an emotion recognition device and the data is sent to the system, the system will adjust the tutorial information according to the user's emotions.

[0249] As a specific example, the operation is as follows.

[0250] The user controls the character in the game and selects items.

[0251] Check the voice guidance and pop-up messages received from the device and try operating it.

[0252] An emotion recognition device detects the user's emotional state and transmits the emotion data.

[0253] Prompt Sentence Examples

[0254] Below are some examples of specific prompt sentences.

[0255] "Please explain how to analyze user operation logs in real time in a game AI tutorial system, identify stumbling points, and provide appropriate tutorial information. Also, how to customize the tutorial content based on user emotions."

[0256] As a result, the AI ​​tutorial system of the present invention can provide appropriate support in real time according to the user's individual operations and emotional state, improving the user experience. It can also be flexibly applied to at-home gameplay and mobile games, achieving high user satisfaction.

[0257] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0258] Step 1:

[0259] The server retrieves the user interface data and the tutorial data.

[0260] Specifically, the server sends a GET request to the game company's API endpoint.

[0261] Input: The endpoint information of the API you want to access.

[0262] Output: Retrieved user interface and tutorial data in JSON format.

[0263] Data processing / calculation: The acquired data is stored in an internal database.

[0264] Step 2:

[0265] The server uses the acquired data to train the AI ​​model using machine learning algorithms.

[0266] Specifically, TensorFlow is used to input learning data, build, train, and save an AI model.

[0267] Input: Retrieved user interface and tutorial data.

[0268] Output: A trained AI model.

[0269] Data processing / calculation: Train the model using the data and optimize it while evaluating its accuracy.

[0270] Step 3:

[0271] The server collects user operation logs sent from the terminal in real time.

[0272] Specifically, it receives operation logs sent from the terminal via WebSocket or HTTP protocol.

[0273] Input: User operation log.

[0274] Output: Stumbling points as analysis results.

[0275] Data processing / calculation: Collected operation logs are input into an AI model in real time for analysis to detect stumbling points.

[0276] Step 4:

[0277] The server detects stumbling points and generates and transmits appropriate tutorial information.

[0278] Specifically, the system retrieves the corresponding tutorial information from the database based on the analysis results and transmits it to the terminal.

[0279] Input: Stumbling points as analysis results.

[0280] Output: Generated tutorial information.

[0281] Data processing / calculation: Search for appropriate tutorial information from the database, generate it, and send it to the device.

[0282] Step 5:

[0283] The terminal collects user operation logs in real time and sends them to the server.

[0284] Specifically, the operations performed by the user are recorded in a log file and periodically sent to the server.

[0285] Input: User operation data.

[0286] Output: The operation log sent to the server.

[0287] Data processing / calculation: Converts the operation data into a log format and sends it to the server.

[0288] Step 6:

[0289] The terminal receives the tutorial information sent from the server and provides it to the user audibly or visually.

[0290] Specifically, it receives tutorial information from the server via WebSocket or HTTP request and displays it as audio guidance or a pop-up message.

[0291] Input: Tutorial information sent from the server.

[0292] Output: Present as an audio prompt or a pop-up message.

[0293] Data processing / calculation: Display the received tutorial information to the user in an appropriate format.

[0294] Step 7:

[0295] The device uses an emotion recognition device to collect the user's emotion data and transmits it to the server.

[0296] Specifically, it analyzes the user's facial expressions and tone of voice through a camera and microphone to obtain emotional data.

[0297] Input: The user's facial expression and tone of voice.

[0298] Output: Emotion data.

[0299] Data processing / calculation: Analyze the acquired emotion data and send it to the server.

[0300] Step 8:

[0301] The user starts the game and continues playing at their own pace.

[0302] Specific actions include controlling a character in the game, selecting items, and fighting enemies.

[0303] Input: The game's control interface.

[0304] Output: Game progress.

[0305] Data processing / calculation: Performing various in-game actions through the user interface.

[0306] Step 9:

[0307] When a user encounters a problem, they can check the voice guidance or pop-ups on their device and take the appropriate action.

[0308] Specifically, the robot follows instructions received from the terminal and attempts to perform operations such as changing equipment and using items.

[0309] Input: Tutorial information from terminal.

[0310] Output: A new log of user actions.

[0311] Data processing / calculation: Performing in-game actions according to received instructions.

[0312] (Application example 2)

[0313] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0314] Current content distribution services for interactive content and games lack systems that provide appropriate guidance when users encounter difficulties with specific operations. Furthermore, the lack of feedback based on the user's emotional state results in a poor user experience. Therefore, it is necessary to provide a system that allows users to enjoy content comfortably without experiencing difficulties or becoming frustrated.

[0315] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for acquiring user interface data, means for collecting user operation logs in real time, means for analyzing the operation logs and identifying user stumbling blocks related to specific operations, means for generating tutorial information corresponding to the stumbling blocks, means for providing the tutorial information to the user, means for recognizing the user's emotions, and means for adjusting the tutorial information based on the recognized emotions. This prevents the user from stumbling over specific operations and provides appropriate guidance according to the user's emotions, enabling a comfortable user experience.

[0316] "User interface data" refers to information about screens and menus that users directly operate, and is data that includes operational elements that users can visually confirm.

[0317] An "operation log" is data that records a series of operations and actions performed by a user, and includes details such as the frequency and order of operations, as well as failures and successes.

[0318] "Stumbling" refers to a situation where a user repeatedly fails to perform a particular operation or tries and fails repeatedly without understanding it.

[0319] "Tutorial information" refers to supplementary information, guides, explanations, etc. provided to help users understand specific operations or procedures, and may be provided in the form of text, audio, video, etc.

[0320] "Emotion recognition" refers to the technology of analyzing a user's emotional state from their facial expressions and voice, and identifying emotions such as irritation, joy, and confusion.

[0321] "Server" refers to the central computer system that receives, processes, analyzes, stores, and distributes data sent from user devices.

[0322] "Real-time" means that processing and communication are carried out immediately with almost no delay, and refers to providing information in response to user operations and situations.

[0323] "Means" refers to methods, techniques, devices, systems, etc. used to achieve a specific purpose.

[0324] To implement this invention, a system using a server, a user terminal, and an emotion recognition device is constructed, the details of which are described below.

[0325] Server Roles

[0326] The server includes the following means:

[0327] 1. Getting user interface data:

[0328] The server obtains user interface data through the content distribution service's API. This refers to information about the screens and menus that users operate, and stores it in a database on the server.

[0329] 2. Collecting operation logs:

[0330] The server collects operation logs sent in real time from user devices, including user actions such as clicking, selecting, and dragging.

[0331] 3. Analyzing operation logs and identifying stumbling points:

[0332] The server analyzes the collected operation logs to detect whether the user is having trouble with a particular operation, and uses machine learning algorithms to respond if a particular operation fails multiple times.

[0333] 4. Generate tutorial information:

[0334] The server generates appropriate tutorial information based on the user's difficulty and transmits it to the user's terminal. The generated tutorial information is provided in text, audio, or video format.

[0335] 5. Emotion Recognition:

[0336] The server analyzes the user's facial expressions and voice based on the data sent from the user's emotion recognition device, using OpenCV and TensorFlow as the specific software used for emotion recognition.

[0337] 6. Tutorial information adjustment based on emotions:

[0338] The server tailors the tutorial information it provides based on the user's perceived emotions, for example providing more detailed and helpful guidance if the user is frustrated, and only brief instructions if the user is relaxed.

[0339] User device roles

[0340] The user terminal includes the following means:

[0341] 1. Collecting operation logs:

[0342] It collects and sends to the server in real time a log of user actions, including clicks, selections, drags, opening and closing menus, etc.

[0343] 2. Receiving and Providing Tutorial Information:

[0344] Receives tutorial information sent from the server and provides it to the user. When providing audio guidance, text-to-speech (TTS) technology may be used.

[0345] 3. Collaboration with emotion recognition devices:

[0346] Using emotion recognition devices such as smartphones, smart glasses, and head-mounted displays, emotions are recognized from the user's facial expressions and voice. The recognized emotion data is sent to a server.

[0347] User Roles

[0348] The user engages with interactive content and receives feedback from the system.

[0349] Prompt Sentence Examples

[0350] 1. When a user keeps failing to perform a specific operation in a video, analyze where they are getting stuck and use the emotion engine to provide appropriate guidance in real time when the user is frustrated.

[0351] In this way, by specifically defining the mode for carrying out the invention, it is possible to realize a system that improves the user experience and reduces stumbling blocks when using interactive content.

[0352] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0353] Step 1:

[0354] Retrieving User Interface Data

[0355] The server obtains user interface data through the API of the content distribution service. This data includes information about the screens and menus that the user operates. The obtained data is stored in a database on the server. The input is the user interface data from the API, and the output is the user interface information stored in the database.

[0356] Step 2:

[0357] Collecting operation logs

[0358] The terminal collects a log of the user's interaction with interactive content in real time and sends it to the server. This log includes clicks, selections, drags, opening and closing menus, etc. The input is the user's actions, and the output is the collected operation log data. The operation log is periodically sent to the server.

[0359] Step 3:

[0360] Analysis of operation logs

[0361] The server receives the operation logs sent from the device in real time and analyzes their contents. It uses a machine learning algorithm to detect stumbling points, such as repeated failures of a specific operation. The input is the operation logs, and the output is the identified stumbling point data.

[0362] Step 4:

[0363] Generate tutorial information

[0364] The server generates appropriate tutorial information based on the analysis results. The generated tutorial information is sent from the server to the terminal in text, audio, or video format. The input is the stumbling point data and user interface data, and the output is the generated tutorial information.

[0365] Step 5:

[0366] Providing tutorial information

[0367] The terminal provides the user with the tutorial information received from the server. In the case of audio guidance, the audio guidance is provided using text-to-speech (TTS) technology. The input is the tutorial information, and the output is guidance to the user. The terminal plays audio guidance such as "Click here" to the user.

[0368] Step 6:

[0369] Emotion recognition

[0370] The terminal recognizes the user's emotions using an emotion recognition device (smartphone, smart glasses, head-mounted display, etc.). The device analyzes the user's facial and voice data acquired in real time to identify the user's emotional state. This emotion data is sent to a server. The input is the user's facial and voice data, and the output is the identified emotion data.

[0371] Step 7:

[0372] Tutorial information adjustment based on emotions

[0373] The server receives the emotion data and adjusts the tutorial information according to the user's emotion. For example, if the user is frustrated, it generates more detailed and helpful guidance, and if the user is relaxed, it generates only brief instructions. The input is emotion data and tutorial information, and the output is the adjusted tutorial information. This information is then sent back to the terminal and provided to the user.

[0374] In this way, a system can be created that provides a comfortable user experience by preventing users from getting stuck on specific operations and by providing appropriate guidance based on their emotions.

[0375] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0376] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0377] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0378] [Second embodiment]

[0379] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0380] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0381] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0382] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0383] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0384] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0385] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0386] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0387] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0388] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0389] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0390] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0391] The following describes an embodiment of the present invention. Specifically, the present invention provides a means and method for realizing an AI tutorial system that provides users with real-time instructions on how to play a game.

[0392] server

[0393] The server connects to the game company's API to obtain user interface data and tutorial data. It trains an AI model based on the obtained data and prepares to analyze user operation logs. It also collects user operation logs sent from the device in real time and detects stumbling blocks related to specific operations. It then generates appropriate tutorial information corresponding to the stumbling block and sends it to the device.

[0394] For example, the server sends API requests to retrieve game operation and tutorial data, stores this data in an internal database, and uses machine learning algorithms to train an AI model, which is then used to analyze user operation logs sent in real time and detect stumbling points.

[0395] Terminal

[0396] The device collects user operation logs in real time and sends them to the server. It records each operation performed by the user (e.g., selecting an item, moving a character, opening and closing a menu, etc.) and periodically sends them to the server. It also receives tutorial information sent from the server and provides it to the user by audio guide or other appropriate means.

[0397] For example, if a user repeatedly opens and closes the same menu in an attempt to change equipment in a game, the operation log is collected in real time and sent to the server. The server analyzes this log, determines that the user does not understand how to change equipment, and sends appropriate tutorial information (e.g., "Press the '+' button to change equipment") to the device. The device then provides this information to the user as audio guidance.

[0398] User

[0399] The user starts the game and continues playing at their own pace. If the user does not know how to perform a particular operation or gets stuck, the system will automatically provide assistance. For example, if the user does not know how to change equipment, the system will automatically detect that the user has repeatedly opened and closed the same menu. The device will provide a voice guide, saying, "You can change it by pressing the '+' button." The user can follow this guide to change their equipment and progress smoothly through the game.

[0400] As described above, the AI ​​tutorial system of the present invention significantly improves the user experience by collecting and analyzing user operation logs in real time and providing the necessary information when needed. This system can be applied to various game platforms, including smartphone games and console games, and will be a useful tool for both game developers and users.

[0401] The processing flow will be explained below.

[0402] Step 1:

[0403] The server connects to the game company's API to retrieve user interface data and tutorial data. Specifically, it sends an API request, receives operation data and tutorial data, and stores it in an internal database.

[0404] Step 2:

[0405] The server trains the AI ​​model based on the acquired data. Specifically, it uses a machine learning algorithm to learn the relationship between the operation data and the results (whether or not there is a stumble). This model is then used.

[0406] Step 3:

[0407] The terminal collects user operation logs in real time and sends them to the server. Specifically, it records user operations (e.g., button presses, menu selections) in a log file and sends the log to the server at regular intervals.

[0408] Step 4:

[0409] The server analyzes the operation logs received from the device in real time. Specifically, it inputs the operation data into an AI model to detect user stumbling. If a certain threshold is exceeded, it is determined to be a stumbling.

[0410] Step 5:

[0411] The server generates tutorial information corresponding to the stumbling point. Specifically, it obtains tutorial data corresponding to the operation data (e.g., "Press the '+' button to change equipment") and sends it to the device.

[0412] Step 6:

[0413] The device receives the tutorial information sent from the server. Specifically, it receives messages from the server using a real-time communication protocol and passes them to the voice assistant system.

[0414] Step 7:

[0415] The device provides the received tutorial information to the user by voice. Specifically, it uses a voice synthesis engine to output the received tutorial information by voice.

[0416] Step 8:

[0417] The user follows the voice guidance to perform operations, specifically, to perform operations within the game according to the presented operating procedures.

[0418] Step 9:

[0419] The device then collects the user's actions after the voice guidance as a log and sends it to the server. Specifically, it records the user's actions after the voice guidance and sends them to the server, contributing to improving the accuracy of the AI ​​model.

[0420] Example 1

[0421] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0422] In conventional game systems, if a user gets stuck on a specific operation, proper support is not provided in real time, which can lead to a poor user experience. Furthermore, delayed feedback on operations can prevent users from continuing the game. Furthermore, existing tutorial systems are often ineffective because they do not take into account individual user behavior patterns and can only provide uniform explanations.

[0423] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0424] In this invention, the server includes means for acquiring user interface data, means for collecting user operation logs in real time, means for analyzing the operation logs and identifying user stumbling blocks related to specific operations, means for generating tutorial information corresponding to the stumbling blocks, means for providing the tutorial information to the user, means for training an AI model using a machine learning algorithm, means for the server to collect and analyze operation logs transmitted from the terminal in real time, and means for the terminal to provide the tutorial information to the user by audio guidance or other means. This makes it possible to provide prompt and appropriate support based on individual behavioral patterns in real time when the user stumbling blocks with a specific operation, improving the user experience.

[0425] "User interface data" refers to data that indicates information about the interface that users use to operate within a game or application.

[0426] A "user operation log" is data that records the operations performed by a user within a game or application.

[0427] "User difficulties related to specific operations" refers to users being unable to perform specific operations correctly, resulting in repeated operations or slow progress.

[0428] "Tutorial information" is information that includes explanations and guides to help users progress smoothly through a game or application.

[0429] A "machine learning algorithm" is a computational method for automatically finding optimal solutions based on data, and is used to train AI models.

[0430] An "AI model" is a data structure that has been trained using machine learning algorithms to generate appropriate outputs for specific inputs.

[0431] A "terminal" is a device that a user uses to operate a game or application, and includes a smartphone, tablet, computer, etc.

[0432] The "server" is a computer system that collects and analyzes user operation logs and provides appropriate tutorial information to users.

[0433] "Audio description" refers to explanatory or instructional information provided aloud to help the user understand a particular operation.

[0434] To practice the present invention, the following procedures and means are used.

[0435] server

[0436] The server first obtains the user interface data and tutorial data through the game company's API. Specifically, it issues a request to the following API endpoint to obtain this data.

[0437] User Interface Data: https: / / api.gamecompany.com / ui_data

[0438] Tutorial Data: https: / / api.gamecompany.com / tutorial_data

[0439] The acquired data is stored in the server's internal database. Next, this data is used to train an AI model using a machine learning algorithm (for example, TensorFlow or PyTorch). During this process, user data is preprocessed and then used to train the model.

[0440] The server also has a means for collecting user operation logs in real time. It receives the operation logs sent from the terminal and analyzes them sequentially. The purpose of the analysis is to identify operations that the user is having trouble with. For example, if the user repeatedly opens and closes the same menu, the server will determine that the operation is difficult for the user.

[0441] Based on the analysis results, the server generates appropriate tutorial information, such as "Press the '+' button to change your equipment," and sends it to the device.

[0442] Terminal

[0443] The device records the user's operation log in real time and periodically sends it to the server. For example, the user's operation of opening the "Settings Menu" in the game is recorded as follows:

[0444] json

[0445] {

[0446] "timestamp": "2023-10-01T12:34:56Z",

[0447] "action": "menu_open",

[0448] "details": "settings_menu"

[0449] }

[0450] This log data is sent to the server in JSON format.

[0451] The device also receives tutorial information sent from the server and provides it to the user as audio guidance or a pop-up message. For example, when receiving tutorial information for changing equipment, the device will play audio guidance such as "Press the '+' button to change equipment."

[0452] User

[0453] The user starts the game and plays freely. If the user is unsure of how to operate the game or has trouble with a particular operation, the system provides real-time support. If the user has trouble with an operation related to changing equipment, the device will provide guidance on how to change equipment. For example, a voice guide will be played saying, "You can change equipment by pressing the '+' button." This allows the user to smoothly change equipment and continue playing the game.

[0454] Specific examples

[0455] If a user does not know how to change equipment and repeatedly opens and closes the same menu, the operation log is sent to the server in real time. The server analyzes the log and determines that the user does not understand how to change equipment. It generates tutorial information such as "Press the '+' button to change equipment" and sends it to the device. The device then provides this information to the user as audio guidance.

[0456] Prompt Sentence Examples

[0457] How do I change equipment in this game?

[0458] "Please provide tutorial information on how to move new characters."

[0459] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0460] Step 1:

[0461] The server retrieves the user interface and tutorial data.

[0462] Specifically, the server sends GET requests to the API endpoints https: / / api.gamecompany.com / ui_data and https: / / api.gamecompany.com / tutorial_data. The input is the API request, and the output is the user interface data and tutorial data returned as a response. These data are stored in an internal database.

[0463] Step 2:

[0464] The server trains the AI ​​model

[0465] Specifically, the server preprocesses the acquired user interface data and tutorial data and applies machine learning algorithms using TensorFlow or PyTorch. The input is the aforementioned dataset, and the output is a trained AI model, which is later used to analyze operation logs.

[0466] Step 3:

[0467] The device collects and sends user operation logs

[0468] Specifically, the device records each operation performed by the user (e.g., opening and closing menus, selecting items, moving characters). For example, the operation of opening the "Settings menu" is recorded as follows:

[0469] json

[0470] {

[0471] "timestamp": "2023-10-01T12:34:56Z",

[0472] "action": "menu_open",

[0473] "details": "settings_menu"

[0474] }

[0475] The input is the user's operation, and the output is an operation log that records this operation. This log data is sent to the server.

[0476] Step 4:

[0477] The server analyzes the operation log.

[0478] Specifically, the server receives operation logs sent from the device in real time and analyzes them using a trained AI model. The input is the operation log, and the output is the identification of operations where the user is having trouble. For example, it can detect when the same menu is repeatedly opened and closed.

[0479] Step 5:

[0480] The server generates the tutorial information

[0481] Specifically, the server generates appropriate tutorial information based on the analysis results. For example, if it detects a problem with changing equipment, it generates a message such as "Press the '+' button to change equipment." The input is the analysis results, and the output is the generated tutorial information.

[0482] Step 6:

[0483] The device receives and provides tutorial information

[0484] Specifically, the terminal receives tutorial information sent from the server, and then provides it to the user as voice guidance or pop-up messages. The input is the tutorial information from the server, and the output is guidance provided to the user through visual or auditory means.

[0485] Step 7:

[0486] Users access tutorial information

[0487] Specifically, the user follows the tutorial information provided by the device and attempts to perform an in-game operation again. For example, the user follows the instruction "Press the '+' button to change equipment" and correctly changes equipment. The input is the tutorial information, and the output is the result of the user's successful operation.

[0488] (Application example 1)

[0489] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0490] With conventional tutorial systems, it was difficult to provide real-time assistance for problems users might encounter in games. It was also difficult to provide appropriate assistance in physical stores when customers couldn't find a specific product or were unsure of the purchasing procedure. This could result in a poor user experience and a decrease in sales.

[0491] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0492] In this invention, the server includes means for acquiring user interface data, means for collecting user operation logs in real time, means for analyzing the operation logs to identify user stumbling blocks related to specific operations, means for generating tutorial information corresponding to the stumbling blocks, means for providing the tutorial information to the user, means for collecting user location information and behavioral data, means for analyzing the collected location information and behavioral data to generate location information of an item desired by the user, and means for providing the location information to the user in the form of audio guidance or map information. This makes it possible to resolve in-game stumbling blocks in real time and also enables users to quickly find the items they desire in physical stores, improving the overall user experience.

[0493] "User interface data" refers to information about the screen layout and display elements that users can use to operate the device.

[0494] "Operation log" refers to the operation history and behavioral record of a user.

[0495] A "stumbling block" refers to a point where a user is unable to smoothly perform a particular operation or action and feels difficulty or confusion.

[0496] "Tutorial information" refers to information that provides explanations and guidance necessary for users to perform operations and actions correctly.

[0497] "Location Information" refers to information about a user's current location.

[0498] "Behavioral data" refers to data such as user behavior and movement history.

[0499] "Audio guidance" refers to a feature that provides information and instructions to the user via voice.

[0500] "Map Information" means data containing geographic information about a user's location or destination.

[0501] The present invention relates to a real-time support system for users to search for and appropriately purchase items in a real-world store. The system includes a server, a terminal, and a user as components.

[0502] server

[0503] The server is used to collect and analyze user interface data, operation logs, location information, and behavioral data. Specifically, the server uses the following software and hardware:

[0504] Software: Python and machine learning libraries (TensorFlow, scikit-learn, etc.) are used for data analysis, along with web frameworks such as Django and Flask and databases (PostgreSQL and MySQL) for real-time data processing.

[0505] Hardware: Use high-performance servers and cloud platforms (AWS, Google Cloud, etc.).

[0506] The server analyzes the user's operation log and location information to detect stumbling blocks related to specific operations or actions, and generates appropriate tutorial information and location information corresponding to the detected stumbling block and sends it to the user's device to help the user find the item quickly.

[0507] Terminal

[0508] The device collects user operation logs, location information, and behavioral data in real time and sends them to the server. It also displays tutorial information and location information sent from the server as audio guides and map information.

[0509] Software: The terminal application is implemented as a smartphone app for iOS or Android. It uses the Google Maps API to display map information and a text-to-speech engine for audio guidance.

[0510] Hardware: Smartphones and tablet devices are used.

[0511] Specifically, when a user scans a QR code installed in the store with their smartphone to log in, operation logs and location information are automatically collected. Appropriate guidance and information are sent from the server and presented on the smartphone as audio guidance and a map display.

[0512] User

[0513] Users use this system when searching for an item in a physical store. The system provides appropriate guidance as the user moves around the store and performs operations using a smartphone app. For example, if a user is looking for a specific item but cannot find it, the system analyzes the user's operation log and location information, and provides appropriate instructions as voice guidance and map information.

[0514] Specific examples

[0515] For example, suppose a user is searching for a specific product in a store. If the user repeatedly opens and closes the product category menu on their smartphone, the system will determine that the user cannot find the product. In this case, a voice guide such as "Here is the location of the product" and map information will be automatically displayed. At this time, the server analyzes the user's behavioral data and location information, and generates and sends appropriate guide information.

[0516] Prompt Sentence Examples

[0517] "When a user is searching for a product in a physical store and repeatedly opens and closes the product category menu, please create a system that provides voice guidance and map information to guide the user to the exact location of the product."

[0518] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0519] Step 1:

[0520] The server acquires user interface data. Specifically, it acquires information about the screen layout and display elements that users use through the game company's or store's API, and stores this information in an internal database. This information is used as the basis for analysis in subsequent processing.

[0521] Step 2:

[0522] The device collects user operation logs and location information in real time. Every time a user performs an operation through the smartphone app in the store, the operation history, behavior record, and location information are recorded and sent to the server at regular intervals. Input data includes the operation details, timestamp, and location information, and is sent to the server.

[0523] Step 3:

[0524] The server analyzes the received operation logs and location information, and uses machine learning algorithms (e.g., TensorFlow, scikit-learn) to analyze whether the user is experiencing difficulties with specific operations or locations. This analysis generates output data that identifies the stumbling points.

[0525] Step 4:

[0526] The server generates tutorial information and location information corresponding to the stumbling points. A pre-trained generative AI model is used to identify stumbling points for specific operations and actions. This training data includes various operation and behavior patterns. The server generates appropriate guide information corresponding to the stumbling points and prepares it for subsequent transmission.

[0527] Step 5:

[0528] The server then sends the generated tutorial and location information, including audio guide text and map information, to the device. The data is sent in real time and is immediately available on the user's device.

[0529] Step 6:

[0530] The device then provides the received tutorial information and location information to the user. Specifically, it generates audio guidance using a text-to-speech engine based on the received guide information, or displays map information using the Google Maps API. This information is presented as a pop-up message or audio guidance on the user's screen.

[0531] Step 7:

[0532] The user operates the device based on the provided tutorial information and location information. By following the guide, the user can find items and learn specific operations, thereby streamlining their activities in the store. This feedback is recorded again in the system and used for subsequent analysis and guide generation.

[0533] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0534] The following describes an embodiment of the present invention. Specifically, the present invention provides a means and method for realizing an AI tutorial system that provides users with real-time instructions on how to play a game, recognizes the user's emotions using an emotion engine, and provides appropriate tutorial information based on the emotions.

[0535] server

[0536] The server connects with the game company's API to acquire user interface data and tutorial data. It trains an AI model based on the acquired data and prepares to analyze user operation logs. It also collects user operation logs sent from the device in real time and detects stumbling blocks related to specific operations. It then generates appropriate tutorial information corresponding to the stumbling block and sends it to the device.

[0537] For example, the server sends API requests to retrieve game operation data and tutorial data. This data is stored in an internal database and used to train an AI model using machine learning algorithms. This model is then used to analyze user operation logs sent in real time and detect stumbling points. It also uses an emotion engine to collect and analyze user emotion data.

[0538] Terminal

[0539] The device collects user operation logs in real time and sends them to the server. It records each operation performed by the user (e.g., selecting an item, moving a character, opening and closing a menu, etc.) and periodically sends them to the server. It receives tutorial information sent from the server and provides it to the user via audio guidance or other appropriate means. It also uses an emotion recognition device to detect the user's emotions and sends the data to the server.

[0540] For example, if a user repeatedly opens and closes the same menu in an attempt to change equipment in a game, the operation log is collected in real time and sent to the server. The server analyzes this log, determines that the user does not understand how to change equipment, and sends appropriate tutorial information (e.g., "Press the '+' button to change equipment") to the device. The device then provides this information to the user as audio guidance. Furthermore, if the user is frustrated, adjustments are made based on their emotions, such as adding more helpful explanations.

[0541] User

[0542] The user starts the game and continues playing at their own pace. If the user is unsure of a particular operation or gets stuck, the system will automatically provide assistance. For example, if the user does not know how to change equipment, the system will automatically detect if they repeatedly open and close the same menu. The device will provide voice guidance, saying, "You can change it by pressing the '+' button." The user can follow this guidance to change their equipment and progress smoothly through the game. In addition, if the user wears an emotion-recognition device and the emotion data is transmitted to the system, the system can adjust the tutorial information provided according to the user's emotions. For example, if the user is frustrated, the system can provide more friendly and polite guidance, and if the user is relaxed, it can provide only brief instructions.

[0543] As described above, the AI ​​tutorial system of the present invention not only collects and analyzes user operation logs in real time and provides necessary information when needed, but also further improves the user experience by using an emotion engine to provide individualized instruction tailored to the user's emotional state. This system can be applied to various game platforms, including smartphone games and console games, and will be a useful tool for both game developers and users.

[0544] The processing flow will be explained below.

[0545] Step 1:

[0546] The server connects to the game company's API to retrieve user interface data and tutorial data. Specifically, it sends an API request, receives operation data and tutorial data, and stores it in an internal database.

[0547] Step 2:

[0548] The server trains the AI ​​model based on the acquired data. Specifically, it uses a machine learning algorithm to learn the relationship between the operation data and the results (whether or not there is a stumble). This model is then used.

[0549] Step 3:

[0550] The terminal collects user operation logs in real time and sends them to the server. Specifically, it records user operations (e.g., button presses, menu selections) in a log file and sends the log to the server at regular intervals.

[0551] Step 4:

[0552] The server analyzes the operation logs received from the device in real time. Specifically, it inputs the operation data into an AI model to detect user stumbling. If a certain threshold is exceeded, it is determined to be a stumbling.

[0553] Step 5:

[0554] The device collects user emotional data using an emotion engine that recognizes the user's emotions in real time. Specifically, data such as heart rate, tone of voice, and facial expressions obtained from the emotion recognition device are input into the emotion engine to identify the user's emotional state.

[0555] Step 6:

[0556] The terminal transmits the collected emotion data to the server. Specifically, the emotion data is transmitted to the server using a real-time communication protocol.

[0557] Step 7:

[0558] The server generates appropriate tutorial information based on the operation log and emotional data. Specifically, it combines data that explains the user's difficulties with the user's emotional state to create optimal tutorial information (e.g., "Press the '+' button to change your equipment" or "You're playing calmly today. Press the '+' button here to change your equipment").

[0559] Step 8:

[0560] The server transmits the generated tutorial information to the terminal. Specifically, the server transmits the generated tutorial information to the terminal using a real-time communication protocol.

[0561] Step 9:

[0562] The device receives the tutorial information sent from the server and provides it to the user as audio guidance. Specifically, it uses a speech synthesis engine to output the received tutorial information as audio.

[0563] Step 10:

[0564] The user follows the voice guidance to operate the game. Specifically, the user performs in-game operations according to the presented operation procedures. Furthermore, the user's experience is improved by receiving feedback according to their emotional state.

[0565] Step 11:

[0566] The device then collects the user's actions after the voice guidance as a log and sends it to the server. Specifically, it records the user's actions after the voice guidance and sends them to the server, contributing to improving the accuracy of the AI ​​model.

[0567] Example 2

[0568] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0569] Conventional game tutorial systems have difficulty providing appropriate support in real time when users get stuck on a specific operation. Furthermore, conventional systems provide uniform tutorial information without considering the user's emotional state, which is why improvements to the user experience cannot be expected. This often leads to users abandoning the game midway, and fails to encourage continued play.

[0570] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0571] In this invention, the server includes means for acquiring user interface data, means for collecting user operation logs in real time, means for analyzing the operation logs and identifying user stumbling blocks related to specific operations, means for generating tutorial information corresponding to the stumbling blocks, means for providing the tutorial information to the user, means for collecting user emotion data, and means for adjusting the tutorial information based on the emotion data, thereby enabling appropriate support in real time according to each user's operation and emotional state.

[0572] "User interface data" refers to information about the game's user interface, specifically including data such as screen layout, menu buttons, and operation methods.

[0573] "User operation log" refers to data that records a series of operations performed by a user in a game in chronological order.

[0574] "User stumbling blocks related to specific operations" refers to situations where a user is unable to understand a specific operation and tries repeatedly or becomes stuck for a long period of time.

[0575] "Tutorial Information" refers to guides and explanations provided to help users understand in-game operations.

[0576] "Emotional data" refers to data that indicates the user's emotional state, derived from facial expressions, tone of voice, and other information.

[0577] "Real-time collection means" refers to the technical means for collecting user actions and emotional states on the spot without delay.

[0578] "Means of analysis" refers to algorithms and systems for analyzing collected operation logs and emotional data and extracting meaningful information.

[0579] "Means for generating" refers to technical means for automatically creating tutorial information appropriate for the user based on the analysis results.

[0580] "Adjusting means" refers to a technical means for dynamically changing the content and presentation method of the tutorial information based on the collected emotional data.

[0581] "Means for providing" refers to the technical means for visually or audibly presenting the generated tutorial information to the user.

[0582] An "emotion recognition device" refers to hardware or software used to detect a user's emotional state by analyzing their facial expressions, tone of voice, etc.

[0583] This invention relates to an AI tutorial system that provides necessary operational instructions in real time when a user plays a game, and provides appropriate tutorial information according to the user's emotional state. This system is composed of three main entities: a server, a terminal, and a user.

[0584] Server Processing

[0585] The server first obtains user interface data using the game company's API. This data acquisition is done, for example, via a REST API, and the obtained data is stored in an internal database. This data is then used to train an AI model. Machine learning algorithms such as TensorFlow and PyTorch are used to train the AI ​​model. This model is then used to analyze user operation logs sent in real time and detect stumbling points.

[0586] As a concrete example, the server operates as follows.

[0587] The server sends a GET request to the game API endpoint to retrieve operation data and tutorial data.

[0588] The acquired data is stored in a database.

[0589] Using TensorFlow, training data is input to build, train, and save an AI model.

[0590] The server also collects user operation logs in real time and analyzes the data. The analysis uses WebSocket and HTTP protocols, and the server receives operation logs sent from the device. If a stumbling point is detected as a result of the analysis, the server generates corresponding tutorial information and sends it to the user's device.

[0591] Terminal handling

[0592] The device collects user operation logs in real time and sends them to the server. It records each operation (e.g., item selection, character movement) and periodically sends them to the server. It receives tutorial information sent from the server and displays it as audio guides or pop-up messages. It also uses emotion recognition devices (e.g., camera, microphone) to collect user emotion data and sends it to the server.

[0593] As a specific example, the operation is as follows.

[0594] The terminal records user operations in a log file and periodically sends it to the server.

[0595] The terminal provides tutorial information from the server as an audio guide.

[0596] The emotion recognition device collects the user's emotion data and sends it to the server.

[0597] User operations

[0598] The user starts the game and continues playing at their own pace. If the user is unsure of a specific operation, the system will automatically assist them. For example, if the user does not know how to change equipment, the system will analyze the user's behavior and provide appropriate guidance. In addition, if the user wears an emotion recognition device and the data is sent to the system, the system will adjust the tutorial information according to the user's emotions.

[0599] As a specific example, the operation is as follows.

[0600] The user controls the character in the game and selects items.

[0601] Check the voice guidance and pop-up messages received from the device and try operating it.

[0602] An emotion recognition device detects the user's emotional state and transmits the emotion data.

[0603] Prompt Sentence Examples

[0604] Below are some examples of specific prompt sentences.

[0605] "Please explain how to analyze user operation logs in real time in a game AI tutorial system, identify stumbling points, and provide appropriate tutorial information. Also, how to customize the tutorial content based on user emotions."

[0606] As a result, the AI ​​tutorial system of the present invention can provide appropriate support in real time according to the user's individual operations and emotional state, improving the user experience. It can also be flexibly applied to at-home gameplay and mobile games, achieving high user satisfaction.

[0607] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0608] Step 1:

[0609] The server retrieves the user interface data and the tutorial data.

[0610] Specifically, the server sends a GET request to the game company's API endpoint.

[0611] Input: The endpoint information of the API you want to access.

[0612] Output: Retrieved user interface and tutorial data in JSON format.

[0613] Data processing / calculation: The acquired data is stored in an internal database.

[0614] Step 2:

[0615] The server uses the acquired data to train the AI ​​model using machine learning algorithms.

[0616] Specifically, TensorFlow is used to input learning data, build, train, and save an AI model.

[0617] Input: Retrieved user interface and tutorial data.

[0618] Output: A trained AI model.

[0619] Data processing / calculation: Train the model using the data and optimize it while evaluating its accuracy.

[0620] Step 3:

[0621] The server collects user operation logs sent from the terminal in real time.

[0622] Specifically, it receives operation logs sent from the terminal via WebSocket or HTTP protocol.

[0623] Input: User operation log.

[0624] Output: Stumbling points as analysis results.

[0625] Data processing / calculation: Collected operation logs are input into an AI model in real time for analysis to detect stumbling points.

[0626] Step 4:

[0627] The server detects stumbling points and generates and transmits appropriate tutorial information.

[0628] Specifically, the system retrieves the corresponding tutorial information from the database based on the analysis results and transmits it to the terminal.

[0629] Input: Stumbling points as analysis results.

[0630] Output: Generated tutorial information.

[0631] Data processing / calculation: Search for appropriate tutorial information from the database, generate it, and send it to the device.

[0632] Step 5:

[0633] The terminal collects user operation logs in real time and sends them to the server.

[0634] Specifically, the operations performed by the user are recorded in a log file and periodically sent to the server.

[0635] Input: User operation data.

[0636] Output: The operation log sent to the server.

[0637] Data processing / calculation: Converts the operation data into a log format and sends it to the server.

[0638] Step 6:

[0639] The terminal receives the tutorial information sent from the server and provides it to the user audibly or visually.

[0640] Specifically, it receives tutorial information from the server via WebSocket or HTTP request and displays it as audio guidance or a pop-up message.

[0641] Input: Tutorial information sent from the server.

[0642] Output: Present as an audio prompt or a pop-up message.

[0643] Data processing / calculation: Display the received tutorial information to the user in an appropriate format.

[0644] Step 7:

[0645] The device uses an emotion recognition device to collect the user's emotion data and transmits it to the server.

[0646] Specifically, it analyzes the user's facial expressions and tone of voice through a camera and microphone to obtain emotional data.

[0647] Input: The user's facial expression and tone of voice.

[0648] Output: Emotion data.

[0649] Data processing / calculation: Analyze the acquired emotion data and send it to the server.

[0650] Step 8:

[0651] The user starts the game and continues playing at their own pace.

[0652] Specific actions include controlling a character in the game, selecting items, and fighting enemies.

[0653] Input: The game's control interface.

[0654] Output: Game progress.

[0655] Data processing / calculation: Performing various in-game actions through the user interface.

[0656] Step 9:

[0657] When a user encounters a problem, they can check the voice guidance or pop-ups on their device and take the appropriate action.

[0658] Specifically, the robot follows instructions received from the terminal and attempts to perform operations such as changing equipment and using items.

[0659] Input: Tutorial information from terminal.

[0660] Output: A new log of user actions.

[0661] Data processing / calculation: Performing in-game actions according to received instructions.

[0662] (Application example 2)

[0663] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0664] Current content distribution services for interactive content and games lack systems that provide appropriate guidance when users encounter difficulties with specific operations. Furthermore, the lack of feedback based on the user's emotional state results in a poor user experience. Therefore, it is necessary to provide a system that allows users to enjoy content comfortably without experiencing difficulties or becoming frustrated.

[0665] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for acquiring user interface data, means for collecting user operation logs in real time, means for analyzing the operation logs and identifying user stumbling blocks related to specific operations, means for generating tutorial information corresponding to the stumbling blocks, means for providing the tutorial information to the user, means for recognizing the user's emotions, and means for adjusting the tutorial information based on the recognized emotions. This prevents the user from stumbling over specific operations and provides appropriate guidance according to the user's emotions, enabling a comfortable user experience.

[0666] "User interface data" refers to information about screens and menus that users directly operate, and is data that includes operational elements that users can visually confirm.

[0667] An "operation log" is data that records a series of operations and actions performed by a user, and includes details such as the frequency and order of operations, as well as failures and successes.

[0668] "Stumbling" refers to a situation where a user repeatedly fails to perform a particular operation or tries and fails repeatedly without understanding it.

[0669] "Tutorial information" refers to supplementary information, guides, explanations, etc. provided to help users understand specific operations or procedures, and may be provided in the form of text, audio, video, etc.

[0670] "Emotion recognition" refers to the technology of analyzing a user's emotional state from their facial expressions and voice, and identifying emotions such as irritation, joy, and confusion.

[0671] "Server" refers to the central computer system that receives, processes, analyzes, stores, and distributes data sent from user devices.

[0672] "Real-time" means that processing and communication are carried out immediately with almost no delay, and refers to providing information in response to user operations and situations.

[0673] "Means" refers to methods, techniques, devices, systems, etc. used to achieve a specific purpose.

[0674] To implement this invention, a system using a server, a user terminal, and an emotion recognition device is constructed, the details of which are described below.

[0675] Server Roles

[0676] The server includes the following means:

[0677] 1. Getting user interface data:

[0678] The server obtains user interface data through the content distribution service's API. This refers to information about the screens and menus that users operate, and stores it in a database on the server.

[0679] 2. Collecting operation logs:

[0680] The server collects operation logs sent in real time from user devices, including user actions such as clicking, selecting, and dragging.

[0681] 3. Analyzing operation logs and identifying stumbling points:

[0682] The server analyzes the collected operation logs to detect whether the user is having trouble with a particular operation, and uses machine learning algorithms to respond if a particular operation fails multiple times.

[0683] 4. Generate tutorial information:

[0684] The server generates appropriate tutorial information based on the user's difficulty and transmits it to the user's terminal. The generated tutorial information is provided in text, audio, or video format.

[0685] 5. Emotion Recognition:

[0686] The server analyzes the user's facial expressions and voice based on the data sent from the user's emotion recognition device, using OpenCV and TensorFlow as the specific software used for emotion recognition.

[0687] 6. Tutorial information adjustment based on emotions:

[0688] The server tailors the tutorial information it provides based on the user's perceived emotions, for example providing more detailed and helpful guidance if the user is frustrated, and only brief instructions if the user is relaxed.

[0689] User device roles

[0690] The user terminal includes the following means:

[0691] 1. Collecting operation logs:

[0692] It collects and sends to the server in real time a log of user actions, including clicks, selections, drags, opening and closing menus, etc.

[0693] 2. Receiving and Providing Tutorial Information:

[0694] Receives tutorial information sent from the server and provides it to the user. When providing audio guidance, text-to-speech (TTS) technology may be used.

[0695] 3. Collaboration with emotion recognition devices:

[0696] Using emotion recognition devices such as smartphones, smart glasses, and head-mounted displays, emotions are recognized from the user's facial expressions and voice. The recognized emotion data is sent to a server.

[0697] User Roles

[0698] The user engages with interactive content and receives feedback from the system.

[0699] Prompt Sentence Examples

[0700] 1. When a user keeps failing to perform a specific operation in a video, analyze where they are getting stuck and use the emotion engine to provide appropriate guidance in real time when the user is frustrated.

[0701] In this way, by specifically defining the mode for carrying out the invention, it is possible to realize a system that improves the user experience and reduces stumbling blocks when using interactive content.

[0702] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0703] Step 1:

[0704] Retrieving User Interface Data

[0705] The server obtains user interface data through the API of the content distribution service. This data includes information about the screens and menus that the user operates. The obtained data is stored in a database on the server. The input is the user interface data from the API, and the output is the user interface information stored in the database.

[0706] Step 2:

[0707] Collecting operation logs

[0708] The terminal collects a log of the user's interaction with interactive content in real time and sends it to the server. This log includes clicks, selections, drags, opening and closing menus, etc. The input is the user's actions, and the output is the collected operation log data. The operation log is periodically sent to the server.

[0709] Step 3:

[0710] Analysis of operation logs

[0711] The server receives the operation logs sent from the device in real time and analyzes their contents. It uses a machine learning algorithm to detect stumbling points, such as repeated failures of a specific operation. The input is the operation logs, and the output is the identified stumbling point data.

[0712] Step 4:

[0713] Generate tutorial information

[0714] The server generates appropriate tutorial information based on the analysis results. The generated tutorial information is sent from the server to the terminal in text, audio, or video format. The input is the stumbling point data and user interface data, and the output is the generated tutorial information.

[0715] Step 5:

[0716] Providing tutorial information

[0717] The terminal provides the user with the tutorial information received from the server. In the case of audio guidance, the audio guidance is provided using text-to-speech (TTS) technology. The input is the tutorial information, and the output is guidance to the user. The terminal plays audio guidance such as "Click here" to the user.

[0718] Step 6:

[0719] Emotion recognition

[0720] The terminal recognizes the user's emotions using an emotion recognition device (smartphone, smart glasses, head-mounted display, etc.). The device analyzes the user's facial and voice data acquired in real time to identify the user's emotional state. This emotion data is sent to a server. The input is the user's facial and voice data, and the output is the identified emotion data.

[0721] Step 7:

[0722] Tutorial information adjustment based on emotions

[0723] The server receives the emotion data and adjusts the tutorial information according to the user's emotion. For example, if the user is frustrated, it generates more detailed and helpful guidance, and if the user is relaxed, it generates only brief instructions. The input is emotion data and tutorial information, and the output is the adjusted tutorial information. This information is then sent back to the terminal and provided to the user.

[0724] In this way, a system can be created that provides a comfortable user experience by preventing users from getting stuck on specific operations and by providing appropriate guidance based on their emotions.

[0725] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0726] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0727] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0728] [Third embodiment]

[0729] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0730] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0731] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0732] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0733] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0734] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0735] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0736] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0737] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0738] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0739] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0740] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0741] The following describes an embodiment of the present invention. Specifically, the present invention provides a means and method for realizing an AI tutorial system that provides users with real-time instructions on how to play a game.

[0742] server

[0743] The server connects to the game company's API to obtain user interface data and tutorial data. It trains an AI model based on the obtained data and prepares to analyze user operation logs. It also collects user operation logs sent from the device in real time and detects stumbling blocks related to specific operations. It then generates appropriate tutorial information corresponding to the stumbling block and sends it to the device.

[0744] For example, the server sends API requests to retrieve game operation and tutorial data, stores this data in an internal database, and uses machine learning algorithms to train an AI model, which is then used to analyze user operation logs sent in real time and detect stumbling points.

[0745] Terminal

[0746] The device collects user operation logs in real time and sends them to the server. It records each operation performed by the user (e.g., selecting an item, moving a character, opening and closing a menu, etc.) and periodically sends them to the server. It also receives tutorial information sent from the server and provides it to the user by audio guide or other appropriate means.

[0747] For example, if a user repeatedly opens and closes the same menu in an attempt to change equipment in a game, the operation log is collected in real time and sent to the server. The server analyzes this log, determines that the user does not understand how to change equipment, and sends appropriate tutorial information (e.g., "Press the '+' button to change equipment") to the device. The device then provides this information to the user as audio guidance.

[0748] User

[0749] The user starts the game and continues playing at their own pace. If the user does not know how to perform a particular operation or gets stuck, the system will automatically provide assistance. For example, if the user does not know how to change equipment, the system will automatically detect that the user has repeatedly opened and closed the same menu. The device will provide a voice guide, saying, "You can change it by pressing the '+' button." The user can follow this guide to change their equipment and progress smoothly through the game.

[0750] As described above, the AI ​​tutorial system of the present invention significantly improves the user experience by collecting and analyzing user operation logs in real time and providing the necessary information when needed. This system can be applied to various game platforms, including smartphone games and console games, and will be a useful tool for both game developers and users.

[0751] The processing flow will be explained below.

[0752] Step 1:

[0753] The server connects to the game company's API to retrieve user interface data and tutorial data. Specifically, it sends an API request, receives operation data and tutorial data, and stores it in an internal database.

[0754] Step 2:

[0755] The server trains the AI ​​model based on the acquired data. Specifically, it uses a machine learning algorithm to learn the relationship between the operation data and the results (whether or not there is a stumble). This model is then used.

[0756] Step 3:

[0757] The terminal collects user operation logs in real time and sends them to the server. Specifically, it records user operations (e.g., button presses, menu selections) in a log file and sends the log to the server at regular intervals.

[0758] Step 4:

[0759] The server analyzes the operation logs received from the device in real time. Specifically, it inputs the operation data into an AI model to detect user stumbling. If a certain threshold is exceeded, it is determined to be a stumbling.

[0760] Step 5:

[0761] The server generates tutorial information corresponding to the stumbling point. Specifically, it obtains tutorial data corresponding to the operation data (e.g., "Press the '+' button to change equipment") and sends it to the device.

[0762] Step 6:

[0763] The device receives the tutorial information sent from the server. Specifically, it receives messages from the server using a real-time communication protocol and passes them to the voice assistant system.

[0764] Step 7:

[0765] The device provides the received tutorial information to the user by voice. Specifically, it uses a voice synthesis engine to output the received tutorial information by voice.

[0766] Step 8:

[0767] The user follows the voice guidance to perform operations, specifically, to perform operations within the game according to the presented operating procedures.

[0768] Step 9:

[0769] The device then collects the user's actions after the voice guidance as a log and sends it to the server. Specifically, it records the user's actions after the voice guidance and sends them to the server, contributing to improving the accuracy of the AI ​​model.

[0770] Example 1

[0771] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0772] In conventional game systems, if a user gets stuck on a specific operation, proper support is not provided in real time, which can lead to a poor user experience. Furthermore, delayed feedback on operations can prevent users from continuing the game. Furthermore, existing tutorial systems are often ineffective because they do not take into account individual user behavior patterns and can only provide uniform explanations.

[0773] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0774] In this invention, the server includes means for acquiring user interface data, means for collecting user operation logs in real time, means for analyzing the operation logs and identifying user stumbling blocks related to specific operations, means for generating tutorial information corresponding to the stumbling blocks, means for providing the tutorial information to the user, means for training an AI model using a machine learning algorithm, means for the server to collect and analyze operation logs transmitted from the terminal in real time, and means for the terminal to provide the tutorial information to the user by audio guidance or other means. This makes it possible to provide prompt and appropriate support based on individual behavioral patterns in real time when the user stumbling blocks with a specific operation, improving the user experience.

[0775] "User interface data" refers to data that indicates information about the interface that users use to operate within a game or application.

[0776] A "user operation log" is data that records the operations performed by a user within a game or application.

[0777] "User difficulties related to specific operations" refers to users being unable to perform specific operations correctly, resulting in repeated operations or slow progress.

[0778] "Tutorial information" is information that includes explanations and guides to help users progress smoothly through a game or application.

[0779] A "machine learning algorithm" is a computational method for automatically finding optimal solutions based on data, and is used to train AI models.

[0780] An "AI model" is a data structure that has been trained using machine learning algorithms to generate appropriate outputs for specific inputs.

[0781] A "terminal" is a device that a user uses to operate a game or application, and includes a smartphone, tablet, computer, etc.

[0782] The "server" is a computer system that collects and analyzes user operation logs and provides appropriate tutorial information to users.

[0783] "Audio description" refers to explanatory or instructional information provided aloud to help the user understand a particular operation.

[0784] To practice the present invention, the following procedures and means are used.

[0785] server

[0786] The server first obtains the user interface data and tutorial data through the game company's API. Specifically, it issues a request to the following API endpoint to obtain this data.

[0787] User Interface Data: https: / / api.gamecompany.com / ui_data

[0788] Tutorial Data: https: / / api.gamecompany.com / tutorial_data

[0789] The acquired data is stored in the server's internal database. Next, this data is used to train an AI model using a machine learning algorithm (for example, TensorFlow or PyTorch). During this process, user data is preprocessed and then used to train the model.

[0790] The server also has a means for collecting user operation logs in real time. It receives the operation logs sent from the terminal and analyzes them sequentially. The purpose of the analysis is to identify operations that the user is having trouble with. For example, if the user repeatedly opens and closes the same menu, the server will determine that the operation is difficult for the user.

[0791] Based on the analysis results, the server generates appropriate tutorial information, such as "Press the '+' button to change your equipment," and sends it to the device.

[0792] Terminal

[0793] The device records the user's operation log in real time and periodically sends it to the server. For example, the user's operation of opening the "Settings Menu" in the game is recorded as follows:

[0794] json

[0795] {

[0796] "timestamp": "2023-10-01T12:34:56Z",

[0797] "action": "menu_open",

[0798] "details": "settings_menu"

[0799] }

[0800] This log data is sent to the server in JSON format.

[0801] The device also receives tutorial information sent from the server and provides it to the user as audio guidance or a pop-up message. For example, when receiving tutorial information for changing equipment, the device will play audio guidance such as "Press the '+' button to change equipment."

[0802] User

[0803] The user starts the game and plays freely. If the user is unsure of how to operate the game or has trouble with a particular operation, the system provides real-time support. If the user has trouble with an operation related to changing equipment, the device will provide guidance on how to change equipment. For example, a voice guide will be played saying, "You can change equipment by pressing the '+' button." This allows the user to smoothly change equipment and continue playing the game.

[0804] Specific examples

[0805] If a user does not know how to change equipment and repeatedly opens and closes the same menu, the operation log is sent to the server in real time. The server analyzes the log and determines that the user does not understand how to change equipment. It generates tutorial information such as "Press the '+' button to change equipment" and sends it to the device. The device then provides this information to the user as audio guidance.

[0806] Prompt Sentence Examples

[0807] How do I change equipment in this game?

[0808] "Please provide tutorial information on how to move new characters."

[0809] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0810] Step 1:

[0811] The server retrieves the user interface and tutorial data.

[0812] Specifically, the server sends GET requests to the API endpoints https: / / api.gamecompany.com / ui_data and https: / / api.gamecompany.com / tutorial_data. The input is the API request, and the output is the user interface data and tutorial data returned as a response. These data are stored in an internal database.

[0813] Step 2:

[0814] The server trains the AI ​​model

[0815] Specifically, the server preprocesses the acquired user interface data and tutorial data and applies machine learning algorithms using TensorFlow or PyTorch. The input is the aforementioned dataset, and the output is a trained AI model, which is later used to analyze operation logs.

[0816] Step 3:

[0817] The device collects and sends user operation logs

[0818] Specifically, the device records each operation performed by the user (e.g., opening and closing menus, selecting items, moving characters). For example, the operation of opening the "Settings menu" is recorded as follows:

[0819] json

[0820] {

[0821] "timestamp": "2023-10-01T12:34:56Z",

[0822] "action": "menu_open",

[0823] "details": "settings_menu"

[0824] }

[0825] The input is the user's operation, and the output is an operation log that records this operation. This log data is sent to the server.

[0826] Step 4:

[0827] The server analyzes the operation log.

[0828] Specifically, the server receives operation logs sent from the device in real time and analyzes them using a trained AI model. The input is the operation log, and the output is the identification of operations where the user is having trouble. For example, it can detect when the same menu is repeatedly opened and closed.

[0829] Step 5:

[0830] The server generates the tutorial information

[0831] Specifically, the server generates appropriate tutorial information based on the analysis results. For example, if it detects a problem with changing equipment, it generates a message such as "Press the '+' button to change equipment." The input is the analysis results, and the output is the generated tutorial information.

[0832] Step 6:

[0833] The device receives and provides tutorial information

[0834] Specifically, the terminal receives tutorial information sent from the server, and then provides it to the user as voice guidance or pop-up messages. The input is the tutorial information from the server, and the output is guidance provided to the user through visual or auditory means.

[0835] Step 7:

[0836] Users access tutorial information

[0837] Specifically, the user follows the tutorial information provided by the device and attempts to perform an in-game operation again. For example, the user follows the instruction "Press the '+' button to change equipment" and correctly changes equipment. The input is the tutorial information, and the output is the result of the user's successful operation.

[0838] (Application example 1)

[0839] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0840] With conventional tutorial systems, it was difficult to provide real-time assistance for problems users might encounter in games. It was also difficult to provide appropriate assistance in physical stores when customers couldn't find a specific product or were unsure of the purchasing procedure. This could result in a poor user experience and a decrease in sales.

[0841] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0842] In this invention, the server includes means for acquiring user interface data, means for collecting user operation logs in real time, means for analyzing the operation logs to identify user stumbling blocks related to specific operations, means for generating tutorial information corresponding to the stumbling blocks, means for providing the tutorial information to the user, means for collecting user location information and behavioral data, means for analyzing the collected location information and behavioral data to generate location information of an item desired by the user, and means for providing the location information to the user in the form of audio guidance or map information. This makes it possible to resolve in-game stumbling blocks in real time and also enables users to quickly find the items they desire in physical stores, improving the overall user experience.

[0843] "User interface data" refers to information about the screen layout and display elements that users can use to operate the device.

[0844] "Operation log" refers to the operation history and behavioral record of a user.

[0845] A "stumbling block" refers to a point where a user is unable to smoothly perform a particular operation or action and feels difficulty or confusion.

[0846] "Tutorial information" refers to information that provides explanations and guidance necessary for users to perform operations and actions correctly.

[0847] "Location Information" refers to information about a user's current location.

[0848] "Behavioral data" refers to data such as user behavior and movement history.

[0849] "Audio guidance" refers to a feature that provides information and instructions to the user via voice.

[0850] "Map Information" means data containing geographic information about a user's location or destination.

[0851] The present invention relates to a real-time support system for users to search for and appropriately purchase items in a real-world store. The system includes a server, a terminal, and a user as components.

[0852] server

[0853] The server is used to collect and analyze user interface data, operation logs, location information, and behavioral data. Specifically, the server uses the following software and hardware:

[0854] Software: Python and machine learning libraries (TensorFlow, scikit-learn, etc.) are used for data analysis, along with web frameworks such as Django and Flask and databases (PostgreSQL and MySQL) for real-time data processing.

[0855] Hardware: Use high-performance servers and cloud platforms (AWS, Google Cloud, etc.).

[0856] The server analyzes the user's operation log and location information to detect stumbling blocks related to specific operations or actions, and generates appropriate tutorial information and location information corresponding to the detected stumbling block and sends it to the user's device to help the user find the item quickly.

[0857] Terminal

[0858] The device collects user operation logs, location information, and behavioral data in real time and sends them to the server. It also displays tutorial information and location information sent from the server as audio guides and map information.

[0859] Software: The terminal application is implemented as a smartphone app for iOS or Android. It uses the Google Maps API to display map information and a text-to-speech engine for audio guidance.

[0860] Hardware: Smartphones and tablet devices are used.

[0861] Specifically, when a user scans a QR code installed in the store with their smartphone to log in, operation logs and location information are automatically collected. Appropriate guidance and information are sent from the server and presented on the smartphone as audio guidance and a map display.

[0862] User

[0863] Users use this system when searching for an item in a physical store. The system provides appropriate guidance as the user moves around the store and performs operations using a smartphone app. For example, if a user is looking for a specific item but cannot find it, the system analyzes the user's operation log and location information, and provides appropriate instructions as voice guidance and map information.

[0864] Specific examples

[0865] For example, suppose a user is searching for a specific product in a store. If the user repeatedly opens and closes the product category menu on their smartphone, the system will determine that the user cannot find the product. In this case, a voice guide such as "Here is the location of the product" and map information will be automatically displayed. At this time, the server analyzes the user's behavioral data and location information, and generates and sends appropriate guide information.

[0866] Prompt Sentence Examples

[0867] "When a user is searching for a product in a physical store and repeatedly opens and closes the product category menu, please create a system that provides voice guidance and map information to guide the user to the exact location of the product."

[0868] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0869] Step 1:

[0870] The server acquires user interface data. Specifically, it acquires information about the screen layout and display elements that users use through the game company's or store's API, and stores this information in an internal database. This information is used as the basis for analysis in subsequent processing.

[0871] Step 2:

[0872] The device collects user operation logs and location information in real time. Every time a user performs an operation through the smartphone app in the store, the operation history, behavior record, and location information are recorded and sent to the server at regular intervals. Input data includes the operation details, timestamp, and location information, and is sent to the server.

[0873] Step 3:

[0874] The server analyzes the received operation logs and location information, and uses machine learning algorithms (e.g., TensorFlow, scikit-learn) to analyze whether the user is experiencing difficulties with specific operations or locations. This analysis generates output data that identifies the stumbling points.

[0875] Step 4:

[0876] The server generates tutorial information and location information corresponding to the stumbling points. A pre-trained generative AI model is used to identify stumbling points for specific operations and actions. This training data includes various operation and behavior patterns. The server generates appropriate guide information corresponding to the stumbling points and prepares it for subsequent transmission.

[0877] Step 5:

[0878] The server then sends the generated tutorial and location information, including audio guide text and map information, to the device. The data is sent in real time and is immediately available on the user's device.

[0879] Step 6:

[0880] The device then provides the received tutorial information and location information to the user. Specifically, it generates audio guidance using a text-to-speech engine based on the received guide information, or displays map information using the Google Maps API. This information is presented as a pop-up message or audio guidance on the user's screen.

[0881] Step 7:

[0882] The user operates the device based on the provided tutorial information and location information. By following the guide, the user can find items and learn specific operations, thereby streamlining their activities in the store. This feedback is recorded again in the system and used for subsequent analysis and guide generation.

[0883] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0884] The following describes an embodiment of the present invention. Specifically, the present invention provides a means and method for realizing an AI tutorial system that provides users with real-time instructions on how to play a game, recognizes the user's emotions using an emotion engine, and provides appropriate tutorial information based on the emotions.

[0885] server

[0886] The server connects with the game company's API to acquire user interface data and tutorial data. It trains an AI model based on the acquired data and prepares to analyze user operation logs. It also collects user operation logs sent from the device in real time and detects stumbling blocks related to specific operations. It then generates appropriate tutorial information corresponding to the stumbling block and sends it to the device.

[0887] For example, the server sends API requests to retrieve game operation data and tutorial data. This data is stored in an internal database and used to train an AI model using machine learning algorithms. This model is then used to analyze user operation logs sent in real time and detect stumbling points. It also uses an emotion engine to collect and analyze user emotion data.

[0888] Terminal

[0889] The device collects user operation logs in real time and sends them to the server. It records each operation performed by the user (e.g., selecting an item, moving a character, opening and closing a menu, etc.) and periodically sends them to the server. It receives tutorial information sent from the server and provides it to the user via audio guidance or other appropriate means. It also uses an emotion recognition device to detect the user's emotions and sends the data to the server.

[0890] For example, if a user repeatedly opens and closes the same menu in an attempt to change equipment in a game, the operation log is collected in real time and sent to the server. The server analyzes this log, determines that the user does not understand how to change equipment, and sends appropriate tutorial information (e.g., "Press the '+' button to change equipment") to the device. The device then provides this information to the user as audio guidance. Furthermore, if the user is frustrated, adjustments are made based on their emotions, such as adding more helpful explanations.

[0891] User

[0892] The user starts the game and continues playing at their own pace. If the user is unsure of a particular operation or gets stuck, the system will automatically provide assistance. For example, if the user does not know how to change equipment, the system will automatically detect if they repeatedly open and close the same menu. The device will provide voice guidance, saying, "You can change it by pressing the '+' button." The user can follow this guidance to change their equipment and progress smoothly through the game. In addition, if the user wears an emotion-recognition device and the emotion data is transmitted to the system, the system can adjust the tutorial information provided according to the user's emotions. For example, if the user is frustrated, the system can provide more friendly and polite guidance, and if the user is relaxed, it can provide only brief instructions.

[0893] As described above, the AI ​​tutorial system of the present invention not only collects and analyzes user operation logs in real time and provides necessary information when needed, but also further improves the user experience by using an emotion engine to provide individualized instruction tailored to the user's emotional state. This system can be applied to various game platforms, including smartphone games and console games, and will be a useful tool for both game developers and users.

[0894] The processing flow will be explained below.

[0895] Step 1:

[0896] The server connects to the game company's API to retrieve user interface data and tutorial data. Specifically, it sends an API request, receives operation data and tutorial data, and stores it in an internal database.

[0897] Step 2:

[0898] The server trains the AI ​​model based on the acquired data. Specifically, it uses a machine learning algorithm to learn the relationship between the operation data and the results (whether or not there is a stumble). This model is then used.

[0899] Step 3:

[0900] The terminal collects user operation logs in real time and sends them to the server. Specifically, it records user operations (e.g., button presses, menu selections) in a log file and sends the log to the server at regular intervals.

[0901] Step 4:

[0902] The server analyzes the operation logs received from the device in real time. Specifically, it inputs the operation data into an AI model to detect user stumbling. If a certain threshold is exceeded, it is determined to be a stumbling.

[0903] Step 5:

[0904] The device collects user emotional data using an emotion engine that recognizes the user's emotions in real time. Specifically, data such as heart rate, tone of voice, and facial expressions obtained from the emotion recognition device are input into the emotion engine to identify the user's emotional state.

[0905] Step 6:

[0906] The terminal transmits the collected emotion data to the server. Specifically, the emotion data is transmitted to the server using a real-time communication protocol.

[0907] Step 7:

[0908] The server generates appropriate tutorial information based on the operation log and emotional data. Specifically, it combines data that explains the user's difficulties with the user's emotional state to create optimal tutorial information (e.g., "Press the '+' button to change your equipment" or "You're playing calmly today. Press the '+' button here to change your equipment").

[0909] Step 8:

[0910] The server transmits the generated tutorial information to the terminal. Specifically, the server transmits the generated tutorial information to the terminal using a real-time communication protocol.

[0911] Step 9:

[0912] The device receives the tutorial information sent from the server and provides it to the user as audio guidance. Specifically, it uses a speech synthesis engine to output the received tutorial information as audio.

[0913] Step 10:

[0914] The user follows the voice guidance to operate the game. Specifically, the user performs in-game operations according to the presented operation procedures. Furthermore, the user's experience is improved by receiving feedback according to their emotional state.

[0915] Step 11:

[0916] The device then collects the user's actions after the voice guidance as a log and sends it to the server. Specifically, it records the user's actions after the voice guidance and sends them to the server, contributing to improving the accuracy of the AI ​​model.

[0917] Example 2

[0918] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0919] Conventional game tutorial systems have difficulty providing appropriate support in real time when users get stuck on a specific operation. Furthermore, conventional systems provide uniform tutorial information without considering the user's emotional state, which is why improvements to the user experience cannot be expected. This often leads to users abandoning the game midway, and fails to encourage continued play.

[0920] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0921] In this invention, the server includes means for acquiring user interface data, means for collecting user operation logs in real time, means for analyzing the operation logs and identifying user stumbling blocks related to specific operations, means for generating tutorial information corresponding to the stumbling blocks, means for providing the tutorial information to the user, means for collecting user emotion data, and means for adjusting the tutorial information based on the emotion data, thereby enabling appropriate support in real time according to each user's operation and emotional state.

[0922] "User interface data" refers to information about the game's user interface, specifically including data such as screen layout, menu buttons, and operation methods.

[0923] "User operation log" refers to data that records a series of operations performed by a user in a game in chronological order.

[0924] "User stumbling blocks related to specific operations" refers to situations where a user is unable to understand a specific operation and tries repeatedly or becomes stuck for a long period of time.

[0925] "Tutorial Information" refers to guides and explanations provided to help users understand in-game operations.

[0926] "Emotional data" refers to data that indicates the user's emotional state, derived from facial expressions, tone of voice, and other information.

[0927] "Real-time collection means" refers to the technical means for collecting user actions and emotional states on the spot without delay.

[0928] "Means of analysis" refers to algorithms and systems for analyzing collected operation logs and emotional data and extracting meaningful information.

[0929] "Means for generating" refers to technical means for automatically creating tutorial information appropriate for the user based on the analysis results.

[0930] "Adjusting means" refers to a technical means for dynamically changing the content and presentation method of the tutorial information based on the collected emotional data.

[0931] "Means for providing" refers to the technical means for visually or audibly presenting the generated tutorial information to the user.

[0932] An "emotion recognition device" refers to hardware or software used to detect a user's emotional state by analyzing their facial expressions, tone of voice, etc.

[0933] This invention relates to an AI tutorial system that provides necessary operational instructions in real time when a user plays a game, and provides appropriate tutorial information according to the user's emotional state. This system is composed of three main entities: a server, a terminal, and a user.

[0934] Server Processing

[0935] The server first obtains user interface data using the game company's API. This data acquisition is done, for example, via a REST API, and the obtained data is stored in an internal database. This data is then used to train an AI model. Machine learning algorithms such as TensorFlow and PyTorch are used to train the AI ​​model. This model is then used to analyze user operation logs sent in real time and detect stumbling points.

[0936] As a concrete example, the server operates as follows.

[0937] The server sends a GET request to the game API endpoint to retrieve operation data and tutorial data.

[0938] The acquired data is stored in a database.

[0939] Using TensorFlow, training data is input to build, train, and save an AI model.

[0940] The server also collects user operation logs in real time and analyzes the data. The analysis uses WebSocket and HTTP protocols, and the server receives operation logs sent from the device. If a stumbling point is detected as a result of the analysis, the server generates corresponding tutorial information and sends it to the user's device.

[0941] Terminal handling

[0942] The device collects user operation logs in real time and sends them to the server. It records each operation (e.g., item selection, character movement) and periodically sends them to the server. It receives tutorial information sent from the server and displays it as audio guides or pop-up messages. It also uses emotion recognition devices (e.g., camera, microphone) to collect user emotion data and sends it to the server.

[0943] As a specific example, the operation is as follows.

[0944] The terminal records user operations in a log file and periodically sends it to the server.

[0945] The terminal provides tutorial information from the server as an audio guide.

[0946] The emotion recognition device collects the user's emotion data and sends it to the server.

[0947] User operations

[0948] The user starts the game and continues playing at their own pace. If the user is unsure of a specific operation, the system will automatically assist them. For example, if the user does not know how to change equipment, the system will analyze the user's behavior and provide appropriate guidance. In addition, if the user wears an emotion recognition device and the data is sent to the system, the system will adjust the tutorial information according to the user's emotions.

[0949] As a specific example, the operation is as follows.

[0950] The user controls the character in the game and selects items.

[0951] Check the voice guidance and pop-up messages received from the device and try operating it.

[0952] An emotion recognition device detects the user's emotional state and transmits the emotion data.

[0953] Prompt Sentence Examples

[0954] Below are some examples of specific prompt sentences.

[0955] "Please explain how to analyze user operation logs in real time in a game AI tutorial system, identify stumbling points, and provide appropriate tutorial information. Also, how to customize the tutorial content based on user emotions."

[0956] As a result, the AI ​​tutorial system of the present invention can provide appropriate support in real time according to the user's individual operations and emotional state, improving the user experience. It can also be flexibly applied to at-home gameplay and mobile games, achieving high user satisfaction.

[0957] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0958] Step 1:

[0959] The server retrieves the user interface data and the tutorial data.

[0960] Specifically, the server sends a GET request to the game company's API endpoint.

[0961] Input: The endpoint information of the API you want to access.

[0962] Output: Retrieved user interface and tutorial data in JSON format.

[0963] Data processing / calculation: The acquired data is stored in an internal database.

[0964] Step 2:

[0965] The server uses the acquired data to train the AI ​​model using machine learning algorithms.

[0966] Specifically, TensorFlow is used to input learning data, build, train, and save an AI model.

[0967] Input: Retrieved user interface and tutorial data.

[0968] Output: A trained AI model.

[0969] Data processing / calculation: Train the model using the data and optimize it while evaluating its accuracy.

[0970] Step 3:

[0971] The server collects user operation logs sent from the terminal in real time.

[0972] Specifically, it receives operation logs sent from the terminal via WebSocket or HTTP protocol.

[0973] Input: User operation log.

[0974] Output: Stumbling points as analysis results.

[0975] Data processing / calculation: Collected operation logs are input into an AI model in real time for analysis to detect stumbling points.

[0976] Step 4:

[0977] The server detects stumbling points and generates and transmits appropriate tutorial information.

[0978] Specifically, the system retrieves the corresponding tutorial information from the database based on the analysis results and transmits it to the terminal.

[0979] Input: Stumbling points as analysis results.

[0980] Output: Generated tutorial information.

[0981] Data processing / calculation: Search for appropriate tutorial information from the database, generate it, and send it to the device.

[0982] Step 5:

[0983] The terminal collects user operation logs in real time and sends them to the server.

[0984] Specifically, the operations performed by the user are recorded in a log file and periodically sent to the server.

[0985] Input: User operation data.

[0986] Output: The operation log sent to the server.

[0987] Data processing / calculation: Converts the operation data into a log format and sends it to the server.

[0988] Step 6:

[0989] The terminal receives the tutorial information sent from the server and provides it to the user audibly or visually.

[0990] Specifically, it receives tutorial information from the server via WebSocket or HTTP request and displays it as audio guidance or a pop-up message.

[0991] Input: Tutorial information sent from the server.

[0992] Output: Present as an audio prompt or a pop-up message.

[0993] Data processing / calculation: Display the received tutorial information to the user in an appropriate format.

[0994] Step 7:

[0995] The device uses an emotion recognition device to collect the user's emotion data and transmits it to the server.

[0996] Specifically, it analyzes the user's facial expressions and tone of voice through a camera and microphone to obtain emotional data.

[0997] Input: The user's facial expression and tone of voice.

[0998] Output: Emotion data.

[0999] Data processing / calculation: Analyze the acquired emotion data and send it to the server.

[1000] Step 8:

[1001] The user starts the game and continues playing at their own pace.

[1002] Specific actions include controlling a character in the game, selecting items, and fighting enemies.

[1003] Input: The game's control interface.

[1004] Output: Game progress.

[1005] Data processing / calculation: Performing various in-game actions through the user interface.

[1006] Step 9:

[1007] When a user encounters a problem, they can check the voice guidance or pop-ups on their device and take the appropriate action.

[1008] Specifically, the robot follows instructions received from the terminal and attempts to perform operations such as changing equipment and using items.

[1009] Input: Tutorial information from terminal.

[1010] Output: A new log of user actions.

[1011] Data processing / calculation: Performing in-game actions according to received instructions.

[1012] (Application example 2)

[1013] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1014] Current content distribution services for interactive content and games lack systems that provide appropriate guidance when users encounter difficulties with specific operations. Furthermore, the lack of feedback based on the user's emotional state results in a poor user experience. Therefore, it is necessary to provide a system that allows users to enjoy content comfortably without experiencing difficulties or becoming frustrated.

[1015] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for acquiring user interface data, means for collecting user operation logs in real time, means for analyzing the operation logs and identifying user stumbling blocks related to specific operations, means for generating tutorial information corresponding to the stumbling blocks, means for providing the tutorial information to the user, means for recognizing the user's emotions, and means for adjusting the tutorial information based on the recognized emotions. This prevents the user from stumbling over specific operations and provides appropriate guidance according to the user's emotions, enabling a comfortable user experience.

[1016] "User interface data" refers to information about screens and menus that users directly operate, and is data that includes operational elements that users can visually confirm.

[1017] An "operation log" is data that records a series of operations and actions performed by a user, and includes details such as the frequency and order of operations, as well as failures and successes.

[1018] "Stumbling" refers to a situation where a user repeatedly fails to perform a particular operation or tries and fails repeatedly without understanding it.

[1019] "Tutorial information" refers to supplementary information, guides, explanations, etc. provided to help users understand specific operations or procedures, and may be provided in the form of text, audio, video, etc.

[1020] "Emotion recognition" refers to the technology of analyzing a user's emotional state from their facial expressions and voice, and identifying emotions such as irritation, joy, and confusion.

[1021] "Server" refers to the central computer system that receives, processes, analyzes, stores, and distributes data sent from user devices.

[1022] "Real-time" means that processing and communication are carried out immediately with almost no delay, and refers to providing information in response to user operations and situations.

[1023] "Means" refers to methods, techniques, devices, systems, etc. used to achieve a specific purpose.

[1024] To implement this invention, a system using a server, a user terminal, and an emotion recognition device is constructed, the details of which are described below.

[1025] Server Roles

[1026] The server includes the following means:

[1027] 1. Getting user interface data:

[1028] The server obtains user interface data through the content distribution service's API. This refers to information about the screens and menus that users operate, and stores it in a database on the server.

[1029] 2. Collecting operation logs:

[1030] The server collects operation logs sent in real time from user devices, including user actions such as clicking, selecting, and dragging.

[1031] 3. Analyzing operation logs and identifying stumbling points:

[1032] The server analyzes the collected operation logs to detect whether the user is having trouble with a particular operation, and uses machine learning algorithms to respond if a particular operation fails multiple times.

[1033] 4. Generate tutorial information:

[1034] The server generates appropriate tutorial information based on the user's difficulty and transmits it to the user's terminal. The generated tutorial information is provided in text, audio, or video format.

[1035] 5. Emotion Recognition:

[1036] The server analyzes the user's facial expressions and voice based on the data sent from the user's emotion recognition device, using OpenCV and TensorFlow as the specific software used for emotion recognition.

[1037] 6. Tutorial information adjustment based on emotions:

[1038] The server tailors the tutorial information it provides based on the user's perceived emotions, for example providing more detailed and helpful guidance if the user is frustrated, and only brief instructions if the user is relaxed.

[1039] User device roles

[1040] The user terminal includes the following means:

[1041] 1. Collecting operation logs:

[1042] It collects and sends to the server in real time a log of user actions, including clicks, selections, drags, opening and closing menus, etc.

[1043] 2. Receiving and Providing Tutorial Information:

[1044] Receives tutorial information sent from the server and provides it to the user. When providing audio guidance, text-to-speech (TTS) technology may be used.

[1045] 3. Collaboration with emotion recognition devices:

[1046] Using emotion recognition devices such as smartphones, smart glasses, and head-mounted displays, emotions are recognized from the user's facial expressions and voice. The recognized emotion data is sent to a server.

[1047] User Roles

[1048] The user engages with interactive content and receives feedback from the system.

[1049] Prompt Sentence Examples

[1050] 1. When a user keeps failing to perform a specific operation in a video, analyze where they are getting stuck and use the emotion engine to provide appropriate guidance in real time when the user is frustrated.

[1051] In this way, by specifically defining the mode for carrying out the invention, it is possible to realize a system that improves the user experience and reduces stumbling blocks when using interactive content.

[1052] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1053] Step 1:

[1054] Retrieving User Interface Data

[1055] The server obtains user interface data through the API of the content distribution service. This data includes information about the screens and menus that the user operates. The obtained data is stored in a database on the server. The input is the user interface data from the API, and the output is the user interface information stored in the database.

[1056] Step 2:

[1057] Collecting operation logs

[1058] The terminal collects a log of the user's interaction with interactive content in real time and sends it to the server. This log includes clicks, selections, drags, opening and closing menus, etc. The input is the user's actions, and the output is the collected operation log data. The operation log is periodically sent to the server.

[1059] Step 3:

[1060] Analysis of operation logs

[1061] The server receives the operation logs sent from the device in real time and analyzes their contents. It uses a machine learning algorithm to detect stumbling points, such as repeated failures of a specific operation. The input is the operation logs, and the output is the identified stumbling point data.

[1062] Step 4:

[1063] Generate tutorial information

[1064] The server generates appropriate tutorial information based on the analysis results. The generated tutorial information is sent from the server to the terminal in text, audio, or video format. The input is the stumbling point data and user interface data, and the output is the generated tutorial information.

[1065] Step 5:

[1066] Providing tutorial information

[1067] The terminal provides the user with the tutorial information received from the server. In the case of audio guidance, the audio guidance is provided using text-to-speech (TTS) technology. The input is the tutorial information, and the output is guidance to the user. The terminal plays audio guidance such as "Click here" to the user.

[1068] Step 6:

[1069] Emotion recognition

[1070] The terminal recognizes the user's emotions using an emotion recognition device (smartphone, smart glasses, head-mounted display, etc.). The device analyzes the user's facial and voice data acquired in real time to identify the user's emotional state. This emotion data is sent to a server. The input is the user's facial and voice data, and the output is the identified emotion data.

[1071] Step 7:

[1072] Tutorial information adjustment based on emotions

[1073] The server receives the emotion data and adjusts the tutorial information according to the user's emotion. For example, if the user is frustrated, it generates more detailed and helpful guidance, and if the user is relaxed, it generates only brief instructions. The input is emotion data and tutorial information, and the output is the adjusted tutorial information. This information is then sent back to the terminal and provided to the user.

[1074] In this way, a system can be created that provides a comfortable user experience by preventing users from getting stuck on specific operations and by providing appropriate guidance based on their emotions.

[1075] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1076] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1077] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1078] [Fourth embodiment]

[1079] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1080] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1081] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1082] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1083] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1084] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1085] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1086] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1087] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1088] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1089] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1090] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1091] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1092] The following describes an embodiment of the present invention. Specifically, the present invention provides a means and method for realizing an AI tutorial system that provides users with real-time instructions on how to play a game.

[1093] server

[1094] The server connects to the game company's API to obtain user interface data and tutorial data. It trains an AI model based on the obtained data and prepares to analyze user operation logs. It also collects user operation logs sent from the device in real time and detects stumbling blocks related to specific operations. It then generates appropriate tutorial information corresponding to the stumbling block and sends it to the device.

[1095] For example, the server sends API requests to retrieve game operation and tutorial data, stores this data in an internal database, and uses machine learning algorithms to train an AI model, which is then used to analyze user operation logs sent in real time and detect stumbling points.

[1096] Terminal

[1097] The device collects user operation logs in real time and sends them to the server. It records each operation performed by the user (e.g., selecting an item, moving a character, opening and closing a menu, etc.) and periodically sends them to the server. It also receives tutorial information sent from the server and provides it to the user by audio guide or other appropriate means.

[1098] For example, if a user repeatedly opens and closes the same menu in an attempt to change equipment in a game, the operation log is collected in real time and sent to the server. The server analyzes this log, determines that the user does not understand how to change equipment, and sends appropriate tutorial information (e.g., "Press the '+' button to change equipment") to the device. The device then provides this information to the user as audio guidance.

[1099] User

[1100] The user starts the game and continues playing at their own pace. If the user does not know how to perform a particular operation or gets stuck, the system will automatically provide assistance. For example, if the user does not know how to change equipment, the system will automatically detect that the user has repeatedly opened and closed the same menu. The device will provide a voice guide, saying, "You can change it by pressing the '+' button." The user can follow this guide to change their equipment and progress smoothly through the game.

[1101] As described above, the AI ​​tutorial system of the present invention significantly improves the user experience by collecting and analyzing user operation logs in real time and providing the necessary information when needed. This system can be applied to various game platforms, including smartphone games and console games, and will be a useful tool for both game developers and users.

[1102] The processing flow will be explained below.

[1103] Step 1:

[1104] The server connects to the game company's API to retrieve user interface data and tutorial data. Specifically, it sends an API request, receives operation data and tutorial data, and stores it in an internal database.

[1105] Step 2:

[1106] The server trains the AI ​​model based on the acquired data. Specifically, it uses a machine learning algorithm to learn the relationship between the operation data and the results (whether or not there is a stumble). This model is then used.

[1107] Step 3:

[1108] The terminal collects user operation logs in real time and sends them to the server. Specifically, it records user operations (e.g., button presses, menu selections) in a log file and sends the log to the server at regular intervals.

[1109] Step 4:

[1110] The server analyzes the operation logs received from the device in real time. Specifically, it inputs the operation data into an AI model to detect user stumbling. If a certain threshold is exceeded, it is determined to be a stumbling.

[1111] Step 5:

[1112] The server generates tutorial information corresponding to the stumbling point. Specifically, it obtains tutorial data corresponding to the operation data (e.g., "Press the '+' button to change equipment") and sends it to the device.

[1113] Step 6:

[1114] The device receives the tutorial information sent from the server. Specifically, it receives messages from the server using a real-time communication protocol and passes them to the voice assistant system.

[1115] Step 7:

[1116] The device provides the received tutorial information to the user by voice. Specifically, it uses a voice synthesis engine to output the received tutorial information by voice.

[1117] Step 8:

[1118] The user follows the voice guidance to perform operations, specifically, to perform operations within the game according to the presented operating procedures.

[1119] Step 9:

[1120] The device then collects the user's actions after the voice guidance as a log and sends it to the server. Specifically, it records the user's actions after the voice guidance and sends them to the server, contributing to improving the accuracy of the AI ​​model.

[1121] Example 1

[1122] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1123] In conventional game systems, if a user gets stuck on a specific operation, proper support is not provided in real time, which can lead to a poor user experience. Furthermore, delayed feedback on operations can prevent users from continuing the game. Furthermore, existing tutorial systems are often ineffective because they do not take into account individual user behavior patterns and can only provide uniform explanations.

[1124] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1125] In this invention, the server includes means for acquiring user interface data, means for collecting user operation logs in real time, means for analyzing the operation logs and identifying user stumbling blocks related to specific operations, means for generating tutorial information corresponding to the stumbling blocks, means for providing the tutorial information to the user, means for training an AI model using a machine learning algorithm, means for the server to collect and analyze operation logs transmitted from the terminal in real time, and means for the terminal to provide the tutorial information to the user by audio guidance or other means. This makes it possible to provide prompt and appropriate support based on individual behavioral patterns in real time when the user stumbling blocks with a specific operation, improving the user experience.

[1126] "User interface data" refers to data that indicates information about the interface that users use to operate within a game or application.

[1127] A "user operation log" is data that records the operations performed by a user within a game or application.

[1128] "User difficulties related to specific operations" refers to users being unable to perform specific operations correctly, resulting in repeated operations or slow progress.

[1129] "Tutorial information" is information that includes explanations and guides to help users progress smoothly through a game or application.

[1130] A "machine learning algorithm" is a computational method for automatically finding optimal solutions based on data, and is used to train AI models.

[1131] An "AI model" is a data structure that has been trained using machine learning algorithms to generate appropriate outputs for specific inputs.

[1132] A "terminal" is a device that a user uses to operate a game or application, and includes a smartphone, tablet, computer, etc.

[1133] The "server" is a computer system that collects and analyzes user operation logs and provides appropriate tutorial information to users.

[1134] "Audio description" refers to explanatory or instructional information provided aloud to help the user understand a particular operation.

[1135] To practice the present invention, the following procedures and means are used.

[1136] server

[1137] The server first obtains the user interface data and tutorial data through the game company's API. Specifically, it issues a request to the following API endpoint to obtain this data.

[1138] User Interface Data: https: / / api.gamecompany.com / ui_data

[1139] Tutorial Data: https: / / api.gamecompany.com / tutorial_data

[1140] The acquired data is stored in the server's internal database. Next, this data is used to train an AI model using a machine learning algorithm (for example, TensorFlow or PyTorch). During this process, user data is preprocessed and then used to train the model.

[1141] The server also has a means for collecting user operation logs in real time. It receives the operation logs sent from the terminal and analyzes them sequentially. The purpose of the analysis is to identify operations that the user is having trouble with. For example, if the user repeatedly opens and closes the same menu, the server will determine that the operation is difficult for the user.

[1142] Based on the analysis results, the server generates appropriate tutorial information, such as "Press the '+' button to change your equipment," and sends it to the device.

[1143] Terminal

[1144] The device records the user's operation log in real time and periodically sends it to the server. For example, the user's operation of opening the "Settings Menu" in the game is recorded as follows:

[1145] json

[1146] {

[1147] "timestamp": "2023-10-01T12:34:56Z",

[1148] "action": "menu_open",

[1149] "details": "settings_menu"

[1150] }

[1151] This log data is sent to the server in JSON format.

[1152] The device also receives tutorial information sent from the server and provides it to the user as audio guidance or a pop-up message. For example, when receiving tutorial information for changing equipment, the device will play audio guidance such as "Press the '+' button to change equipment."

[1153] User

[1154] The user starts the game and plays freely. If the user is unsure of how to operate the game or has trouble with a particular operation, the system provides real-time support. If the user has trouble with an operation related to changing equipment, the device will provide guidance on how to change equipment. For example, a voice guide will be played saying, "You can change equipment by pressing the '+' button." This allows the user to smoothly change equipment and continue playing the game.

[1155] Specific examples

[1156] If a user does not know how to change equipment and repeatedly opens and closes the same menu, the operation log is sent to the server in real time. The server analyzes the log and determines that the user does not understand how to change equipment. It generates tutorial information such as "Press the '+' button to change equipment" and sends it to the device. The device then provides this information to the user as audio guidance.

[1157] Prompt Sentence Examples

[1158] How do I change equipment in this game?

[1159] "Please provide tutorial information on how to move new characters."

[1160] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1161] Step 1:

[1162] The server retrieves the user interface and tutorial data.

[1163] Specifically, the server sends GET requests to the API endpoints https: / / api.gamecompany.com / ui_data and https: / / api.gamecompany.com / tutorial_data. The input is the API request, and the output is the user interface data and tutorial data returned as a response. These data are stored in an internal database.

[1164] Step 2:

[1165] The server trains the AI ​​model

[1166] Specifically, the server preprocesses the acquired user interface data and tutorial data and applies machine learning algorithms using TensorFlow or PyTorch. The input is the aforementioned dataset, and the output is a trained AI model, which is later used to analyze operation logs.

[1167] Step 3:

[1168] The device collects and sends user operation logs

[1169] Specifically, the device records each operation performed by the user (e.g., opening and closing menus, selecting items, moving characters). For example, the operation of opening the "Settings menu" is recorded as follows:

[1170] json

[1171] {

[1172] "timestamp": "2023-10-01T12:34:56Z",

[1173] "action": "menu_open",

[1174] "details": "settings_menu"

[1175] }

[1176] The input is the user's operation, and the output is an operation log that records this operation. This log data is sent to the server.

[1177] Step 4:

[1178] The server analyzes the operation log.

[1179] Specifically, the server receives operation logs sent from the device in real time and analyzes them using a trained AI model. The input is the operation log, and the output is the identification of operations where the user is having trouble. For example, it can detect when the same menu is repeatedly opened and closed.

[1180] Step 5:

[1181] The server generates the tutorial information

[1182] Specifically, the server generates appropriate tutorial information based on the analysis results. For example, if it detects a problem with changing equipment, it generates a message such as "Press the '+' button to change equipment." The input is the analysis results, and the output is the generated tutorial information.

[1183] Step 6:

[1184] The device receives and provides tutorial information

[1185] Specifically, the terminal receives tutorial information sent from the server, and then provides it to the user as voice guidance or pop-up messages. The input is the tutorial information from the server, and the output is guidance provided to the user through visual or auditory means.

[1186] Step 7:

[1187] Users access tutorial information

[1188] Specifically, the user follows the tutorial information provided by the device and attempts to perform an in-game operation again. For example, the user follows the instruction "Press the '+' button to change equipment" and correctly changes equipment. The input is the tutorial information, and the output is the result of the user's successful operation.

[1189] (Application example 1)

[1190] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1191] With conventional tutorial systems, it was difficult to provide real-time assistance for problems users might encounter in games. It was also difficult to provide appropriate assistance in physical stores when customers couldn't find a specific product or were unsure of the purchasing procedure. This could result in a poor user experience and a decrease in sales.

[1192] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1193] In this invention, the server includes means for acquiring user interface data, means for collecting user operation logs in real time, means for analyzing the operation logs to identify user stumbling blocks related to specific operations, means for generating tutorial information corresponding to the stumbling blocks, means for providing the tutorial information to the user, means for collecting user location information and behavioral data, means for analyzing the collected location information and behavioral data to generate location information of an item desired by the user, and means for providing the location information to the user in the form of audio guidance or map information. This makes it possible to resolve in-game stumbling blocks in real time and also enables users to quickly find the items they desire in physical stores, improving the overall user experience.

[1194] "User interface data" refers to information about the screen layout and display elements that users can use to operate the device.

[1195] "Operation log" refers to the operation history and behavioral record of a user.

[1196] A "stumbling block" refers to a point where a user is unable to smoothly perform a particular operation or action and feels difficulty or confusion.

[1197] "Tutorial information" refers to information that provides explanations and guidance necessary for users to perform operations and actions correctly.

[1198] "Location Information" refers to information about a user's current location.

[1199] "Behavioral data" refers to data such as user behavior and movement history.

[1200] "Audio guidance" refers to a feature that provides information and instructions to the user via voice.

[1201] "Map Information" means data containing geographic information about a user's location or destination.

[1202] The present invention relates to a real-time support system for users to search for and appropriately purchase items in a real-world store. The system includes a server, a terminal, and a user as components.

[1203] server

[1204] The server is used to collect and analyze user interface data, operation logs, location information, and behavioral data. Specifically, the server uses the following software and hardware:

[1205] Software: Python and machine learning libraries (TensorFlow, scikit-learn, etc.) are used for data analysis, along with web frameworks such as Django and Flask and databases (PostgreSQL and MySQL) for real-time data processing.

[1206] Hardware: Use high-performance servers and cloud platforms (AWS, Google Cloud, etc.).

[1207] The server analyzes the user's operation log and location information to detect stumbling blocks related to specific operations or actions, and generates appropriate tutorial information and location information corresponding to the detected stumbling block and sends it to the user's device to help the user find the item quickly.

[1208] Terminal

[1209] The device collects user operation logs, location information, and behavioral data in real time and sends them to the server. It also displays tutorial information and location information sent from the server as audio guides and map information.

[1210] Software: The terminal application is implemented as a smartphone app for iOS or Android. It uses the Google Maps API to display map information and a text-to-speech engine for audio guidance.

[1211] Hardware: Smartphones and tablet devices are used.

[1212] Specifically, when a user scans a QR code installed in the store with their smartphone to log in, operation logs and location information are automatically collected. Appropriate guidance and information are sent from the server and presented on the smartphone as audio guidance and a map display.

[1213] User

[1214] Users use this system when searching for an item in a physical store. The system provides appropriate guidance as the user moves around the store and performs operations using a smartphone app. For example, if a user is looking for a specific item but cannot find it, the system analyzes the user's operation log and location information, and provides appropriate instructions as voice guidance and map information.

[1215] Specific examples

[1216] For example, suppose a user is searching for a specific product in a store. If the user repeatedly opens and closes the product category menu on their smartphone, the system will determine that the user cannot find the product. In this case, a voice guide such as "Here is the location of the product" and map information will be automatically displayed. At this time, the server analyzes the user's behavioral data and location information, and generates and sends appropriate guide information.

[1217] Prompt Sentence Examples

[1218] "When a user is searching for a product in a physical store and repeatedly opens and closes the product category menu, please create a system that provides voice guidance and map information to guide the user to the exact location of the product."

[1219] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1220] Step 1:

[1221] The server acquires user interface data. Specifically, it acquires information about the screen layout and display elements that users use through the game company's or store's API, and stores this information in an internal database. This information is used as the basis for analysis in subsequent processing.

[1222] Step 2:

[1223] The device collects user operation logs and location information in real time. Every time a user performs an operation through the smartphone app in the store, the operation history, behavior record, and location information are recorded and sent to the server at regular intervals. Input data includes the operation details, timestamp, and location information, and is sent to the server.

[1224] Step 3:

[1225] The server analyzes the received operation logs and location information, and uses machine learning algorithms (e.g., TensorFlow, scikit-learn) to analyze whether the user is experiencing difficulties with specific operations or locations. This analysis generates output data that identifies the stumbling points.

[1226] Step 4:

[1227] The server generates tutorial information and location information corresponding to the stumbling points. A pre-trained generative AI model is used to identify stumbling points for specific operations and actions. This training data includes various operation and behavior patterns. The server generates appropriate guide information corresponding to the stumbling points and prepares it for subsequent transmission.

[1228] Step 5:

[1229] The server then sends the generated tutorial and location information, including audio guide text and map information, to the device. The data is sent in real time and is immediately available on the user's device.

[1230] Step 6:

[1231] The device then provides the received tutorial information and location information to the user. Specifically, it generates audio guidance using a text-to-speech engine based on the received guide information, or displays map information using the Google Maps API. This information is presented as a pop-up message or audio guidance on the user's screen.

[1232] Step 7:

[1233] The user operates the device based on the provided tutorial information and location information. By following the guide, the user can find items and learn specific operations, thereby streamlining their activities in the store. This feedback is recorded again in the system and used for subsequent analysis and guide generation.

[1234] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1235] The following describes an embodiment of the present invention. Specifically, the present invention provides a means and method for realizing an AI tutorial system that provides users with real-time instructions on how to play a game, recognizes the user's emotions using an emotion engine, and provides appropriate tutorial information based on the emotions.

[1236] server

[1237] The server connects with the game company's API to acquire user interface data and tutorial data. It trains an AI model based on the acquired data and prepares to analyze user operation logs. It also collects user operation logs sent from the device in real time and detects stumbling blocks related to specific operations. It then generates appropriate tutorial information corresponding to the stumbling block and sends it to the device.

[1238] For example, the server sends API requests to retrieve game operation data and tutorial data. This data is stored in an internal database and used to train an AI model using machine learning algorithms. This model is then used to analyze user operation logs sent in real time and detect stumbling points. It also uses an emotion engine to collect and analyze user emotion data.

[1239] Terminal

[1240] The device collects user operation logs in real time and sends them to the server. It records each operation performed by the user (e.g., selecting an item, moving a character, opening and closing a menu, etc.) and periodically sends them to the server. It receives tutorial information sent from the server and provides it to the user via audio guidance or other appropriate means. It also uses an emotion recognition device to detect the user's emotions and sends the data to the server.

[1241] For example, if a user repeatedly opens and closes the same menu in an attempt to change equipment in a game, the operation log is collected in real time and sent to the server. The server analyzes this log, determines that the user does not understand how to change equipment, and sends appropriate tutorial information (e.g., "Press the '+' button to change equipment") to the device. The device then provides this information to the user as audio guidance. Furthermore, if the user is frustrated, adjustments are made based on their emotions, such as adding more helpful explanations.

[1242] User

[1243] The user starts the game and continues playing at their own pace. If the user is unsure of a particular operation or gets stuck, the system will automatically provide assistance. For example, if the user does not know how to change equipment, the system will automatically detect if they repeatedly open and close the same menu. The device will provide voice guidance, saying, "You can change it by pressing the '+' button." The user can follow this guidance to change their equipment and progress smoothly through the game. In addition, if the user wears an emotion-recognition device and the emotion data is transmitted to the system, the system can adjust the tutorial information provided according to the user's emotions. For example, if the user is frustrated, the system can provide more friendly and polite guidance, and if the user is relaxed, it can provide only brief instructions.

[1244] As described above, the AI ​​tutorial system of the present invention not only collects and analyzes user operation logs in real time and provides necessary information when needed, but also further improves the user experience by using an emotion engine to provide individualized instruction tailored to the user's emotional state. This system can be applied to various game platforms, including smartphone games and console games, and will be a useful tool for both game developers and users.

[1245] The processing flow will be explained below.

[1246] Step 1:

[1247] The server connects to the game company's API to retrieve user interface data and tutorial data. Specifically, it sends an API request, receives operation data and tutorial data, and stores it in an internal database.

[1248] Step 2:

[1249] The server trains the AI ​​model based on the acquired data. Specifically, it uses a machine learning algorithm to learn the relationship between the operation data and the results (whether or not there is a stumble). This model is then used.

[1250] Step 3:

[1251] The terminal collects user operation logs in real time and sends them to the server. Specifically, it records user operations (e.g., button presses, menu selections) in a log file and sends the log to the server at regular intervals.

[1252] Step 4:

[1253] The server analyzes the operation logs received from the device in real time. Specifically, it inputs the operation data into an AI model to detect user stumbling. If a certain threshold is exceeded, it is determined to be a stumbling.

[1254] Step 5:

[1255] The device collects user emotional data using an emotion engine that recognizes the user's emotions in real time. Specifically, data such as heart rate, tone of voice, and facial expressions obtained from the emotion recognition device are input into the emotion engine to identify the user's emotional state.

[1256] Step 6:

[1257] The terminal transmits the collected emotion data to the server. Specifically, the emotion data is transmitted to the server using a real-time communication protocol.

[1258] Step 7:

[1259] The server generates appropriate tutorial information based on the operation log and emotional data. Specifically, it combines data that explains the user's difficulties with the user's emotional state to create optimal tutorial information (e.g., "Press the '+' button to change your equipment" or "You're playing calmly today. Press the '+' button here to change your equipment").

[1260] Step 8:

[1261] The server transmits the generated tutorial information to the terminal. Specifically, the server transmits the generated tutorial information to the terminal using a real-time communication protocol.

[1262] Step 9:

[1263] The device receives the tutorial information sent from the server and provides it to the user as audio guidance. Specifically, it uses a speech synthesis engine to output the received tutorial information as audio.

[1264] Step 10:

[1265] The user follows the voice guidance to operate the game. Specifically, the user performs in-game operations according to the presented operation procedures. Furthermore, the user's experience is improved by receiving feedback according to their emotional state.

[1266] Step 11:

[1267] The device then collects the user's actions after the voice guidance as a log and sends it to the server. Specifically, it records the user's actions after the voice guidance and sends them to the server, contributing to improving the accuracy of the AI ​​model.

[1268] Example 2

[1269] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1270] Conventional game tutorial systems have difficulty providing appropriate support in real time when users get stuck on a specific operation. Furthermore, conventional systems provide uniform tutorial information without considering the user's emotional state, which is why improvements to the user experience cannot be expected. This often leads to users abandoning the game midway, and fails to encourage continued play.

[1271] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1272] In this invention, the server includes means for acquiring user interface data, means for collecting user operation logs in real time, means for analyzing the operation logs and identifying user stumbling blocks related to specific operations, means for generating tutorial information corresponding to the stumbling blocks, means for providing the tutorial information to the user, means for collecting user emotion data, and means for adjusting the tutorial information based on the emotion data, thereby enabling appropriate support in real time according to each user's operation and emotional state.

[1273] "User interface data" refers to information about the game's user interface, specifically including data such as screen layout, menu buttons, and operation methods.

[1274] "User operation log" refers to data that records a series of operations performed by a user in a game in chronological order.

[1275] "User stumbling blocks related to specific operations" refers to situations where a user is unable to understand a specific operation and tries repeatedly or becomes stuck for a long period of time.

[1276] "Tutorial Information" refers to guides and explanations provided to help users understand in-game operations.

[1277] "Emotional data" refers to data that indicates the user's emotional state, derived from facial expressions, tone of voice, and other information.

[1278] "Real-time collection means" refers to the technical means for collecting user actions and emotional states on the spot without delay.

[1279] "Means of analysis" refers to algorithms and systems for analyzing collected operation logs and emotional data and extracting meaningful information.

[1280] "Means for generating" refers to technical means for automatically creating tutorial information appropriate for the user based on the analysis results.

[1281] "Adjusting means" refers to a technical means for dynamically changing the content and presentation method of the tutorial information based on the collected emotional data.

[1282] "Means for providing" refers to the technical means for visually or audibly presenting the generated tutorial information to the user.

[1283] An "emotion recognition device" refers to hardware or software used to detect a user's emotional state by analyzing their facial expressions, tone of voice, etc.

[1284] This invention relates to an AI tutorial system that provides necessary operational instructions in real time when a user plays a game, and provides appropriate tutorial information according to the user's emotional state. This system is composed of three main entities: a server, a terminal, and a user.

[1285] Server Processing

[1286] The server first obtains user interface data using the game company's API. This data acquisition is done, for example, via a REST API, and the obtained data is stored in an internal database. This data is then used to train an AI model. Machine learning algorithms such as TensorFlow and PyTorch are used to train the AI ​​model. This model is then used to analyze user operation logs sent in real time and detect stumbling points.

[1287] As a concrete example, the server operates as follows.

[1288] The server sends a GET request to the game API endpoint to retrieve operation data and tutorial data.

[1289] The acquired data is stored in a database.

[1290] Using TensorFlow, training data is input to build, train, and save an AI model.

[1291] The server also collects user operation logs in real time and analyzes the data. The analysis uses WebSocket and HTTP protocols, and the server receives operation logs sent from the device. If a stumbling point is detected as a result of the analysis, the server generates corresponding tutorial information and sends it to the user's device.

[1292] Terminal handling

[1293] The device collects user operation logs in real time and sends them to the server. It records each operation (e.g., item selection, character movement) and periodically sends them to the server. It receives tutorial information sent from the server and displays it as audio guides or pop-up messages. It also uses emotion recognition devices (e.g., camera, microphone) to collect user emotion data and sends it to the server.

[1294] As a specific example, the operation is as follows.

[1295] The terminal records user operations in a log file and periodically sends it to the server.

[1296] The terminal provides tutorial information from the server as an audio guide.

[1297] The emotion recognition device collects the user's emotion data and sends it to the server.

[1298] User operations

[1299] The user starts the game and continues playing at their own pace. If the user is unsure of a specific operation, the system will automatically assist them. For example, if the user does not know how to change equipment, the system will analyze the user's behavior and provide appropriate guidance. In addition, if the user wears an emotion recognition device and the data is sent to the system, the system will adjust the tutorial information according to the user's emotions.

[1300] As a specific example, the operation is as follows.

[1301] The user controls the character in the game and selects items.

[1302] Check the voice guidance and pop-up messages received from the device and try operating it.

[1303] An emotion recognition device detects the user's emotional state and transmits the emotion data.

[1304] Prompt Sentence Examples

[1305] Below are some examples of specific prompt sentences.

[1306] "Please explain how to analyze user operation logs in real time in a game AI tutorial system, identify stumbling points, and provide appropriate tutorial information. Also, how to customize the tutorial content based on user emotions."

[1307] As a result, the AI ​​tutorial system of the present invention can provide appropriate support in real time according to the user's individual operations and emotional state, improving the user experience. It can also be flexibly applied to at-home gameplay and mobile games, achieving high user satisfaction.

[1308] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1309] Step 1:

[1310] The server retrieves the user interface data and the tutorial data.

[1311] Specifically, the server sends a GET request to the game company's API endpoint.

[1312] Input: The endpoint information of the API you want to access.

[1313] Output: Retrieved user interface and tutorial data in JSON format.

[1314] Data processing / calculation: The acquired data is stored in an internal database.

[1315] Step 2:

[1316] The server uses the acquired data to train the AI ​​model using machine learning algorithms.

[1317] Specifically, TensorFlow is used to input learning data, build, train, and save an AI model.

[1318] Input: Retrieved user interface and tutorial data.

[1319] Output: A trained AI model.

[1320] Data processing / calculation: Train the model using the data and optimize it while evaluating its accuracy.

[1321] Step 3:

[1322] The server collects user operation logs sent from the terminal in real time.

[1323] Specifically, it receives operation logs sent from the terminal via WebSocket or HTTP protocol.

[1324] Input: User operation log.

[1325] Output: Stumbling points as analysis results.

[1326] Data processing / calculation: Collected operation logs are input into an AI model in real time for analysis to detect stumbling points.

[1327] Step 4:

[1328] The server detects stumbling points and generates and transmits appropriate tutorial information.

[1329] Specifically, the system retrieves the corresponding tutorial information from the database based on the analysis results and transmits it to the terminal.

[1330] Input: Stumbling points as analysis results.

[1331] Output: Generated tutorial information.

[1332] Data processing / calculation: Search for appropriate tutorial information from the database, generate it, and send it to the device.

[1333] Step 5:

[1334] The terminal collects user operation logs in real time and sends them to the server.

[1335] Specifically, the operations performed by the user are recorded in a log file and periodically sent to the server.

[1336] Input: User operation data.

[1337] Output: The operation log sent to the server.

[1338] Data processing / calculation: Converts the operation data into a log format and sends it to the server.

[1339] Step 6:

[1340] The terminal receives the tutorial information sent from the server and provides it to the user audibly or visually.

[1341] Specifically, it receives tutorial information from the server via WebSocket or HTTP request and displays it as audio guidance or a pop-up message.

[1342] Input: Tutorial information sent from the server.

[1343] Output: Present as an audio prompt or a pop-up message.

[1344] Data processing / calculation: Display the received tutorial information to the user in an appropriate format.

[1345] Step 7:

[1346] The device uses an emotion recognition device to collect the user's emotion data and transmits it to the server.

[1347] Specifically, it analyzes the user's facial expressions and tone of voice through a camera and microphone to obtain emotional data.

[1348] Input: The user's facial expression and tone of voice.

[1349] Output: Emotion data.

[1350] Data processing / calculation: Analyze the acquired emotion data and send it to the server.

[1351] Step 8:

[1352] The user starts the game and continues playing at their own pace.

[1353] Specific actions include controlling a character in the game, selecting items, and fighting enemies.

[1354] Input: The game's control interface.

[1355] Output: Game progress.

[1356] Data processing / calculation: Performing various in-game actions through the user interface.

[1357] Step 9:

[1358] When a user encounters a problem, they can check the voice guidance or pop-ups on their device and take the appropriate action.

[1359] Specifically, the robot follows instructions received from the terminal and attempts to perform operations such as changing equipment and using items.

[1360] Input: Tutorial information from terminal.

[1361] Output: A new log of user actions.

[1362] Data processing / calculation: Performing in-game actions according to received instructions.

[1363] (Application example 2)

[1364] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1365] Current content distribution services for interactive content and games lack systems that provide appropriate guidance when users encounter difficulties with specific operations. Furthermore, the lack of feedback based on the user's emotional state results in a poor user experience. Therefore, it is necessary to provide a system that allows users to enjoy content comfortably without experiencing difficulties or becoming frustrated.

[1366] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for acquiring user interface data, means for collecting user operation logs in real time, means for analyzing the operation logs and identifying user stumbling blocks related to specific operations, means for generating tutorial information corresponding to the stumbling blocks, means for providing the tutorial information to the user, means for recognizing the user's emotions, and means for adjusting the tutorial information based on the recognized emotions. This prevents the user from stumbling over specific operations and provides appropriate guidance according to the user's emotions, enabling a comfortable user experience.

[1367] "User interface data" refers to information about screens and menus that users directly operate, and is data that includes operational elements that users can visually confirm.

[1368] An "operation log" is data that records a series of operations and actions performed by a user, and includes details such as the frequency and order of operations, as well as failures and successes.

[1369] "Stumbling" refers to a situation where a user repeatedly fails to perform a particular operation or tries and fails repeatedly without understanding it.

[1370] "Tutorial information" refers to supplementary information, guides, explanations, etc. provided to help users understand specific operations or procedures, and may be provided in the form of text, audio, video, etc.

[1371] "Emotion recognition" refers to the technology of analyzing a user's emotional state from their facial expressions and voice, and identifying emotions such as irritation, joy, and confusion.

[1372] "Server" refers to the central computer system that receives, processes, analyzes, stores, and distributes data sent from user devices.

[1373] "Real-time" means that processing and communication are carried out immediately with almost no delay, and refers to providing information in response to user operations and situations.

[1374] "Means" refers to methods, techniques, devices, systems, etc. used to achieve a specific purpose.

[1375] To implement this invention, a system using a server, a user terminal, and an emotion recognition device is constructed, the details of which are described below.

[1376] Server Roles

[1377] The server includes the following means:

[1378] 1. Getting user interface data:

[1379] The server obtains user interface data through the content distribution service's API. This refers to information about the screens and menus that users operate, and stores it in a database on the server.

[1380] 2. Collecting operation logs:

[1381] The server collects operation logs sent in real time from user devices, including user actions such as clicking, selecting, and dragging.

[1382] 3. Analyzing operation logs and identifying stumbling points:

[1383] The server analyzes the collected operation logs to detect whether the user is having trouble with a particular operation, and uses machine learning algorithms to respond if a particular operation fails multiple times.

[1384] 4. Generate tutorial information:

[1385] The server generates appropriate tutorial information based on the user's difficulty and transmits it to the user's terminal. The generated tutorial information is provided in text, audio, or video format.

[1386] 5. Emotion Recognition:

[1387] The server analyzes the user's facial expressions and voice based on the data sent from the user's emotion recognition device, using OpenCV and TensorFlow as the specific software used for emotion recognition.

[1388] 6. Tutorial information adjustment based on emotions:

[1389] The server tailors the tutorial information it provides based on the user's perceived emotions, for example providing more detailed and helpful guidance if the user is frustrated, and only brief instructions if the user is relaxed.

[1390] User device roles

[1391] The user terminal includes the following means:

[1392] 1. Collecting operation logs:

[1393] It collects and sends to the server in real time a log of user actions, including clicks, selections, drags, opening and closing menus, etc.

[1394] 2. Receiving and Providing Tutorial Information:

[1395] Receives tutorial information sent from the server and provides it to the user. When providing audio guidance, text-to-speech (TTS) technology may be used.

[1396] 3. Collaboration with emotion recognition devices:

[1397] Using emotion recognition devices such as smartphones, smart glasses, and head-mounted displays, emotions are recognized from the user's facial expressions and voice. The recognized emotion data is sent to a server.

[1398] User Roles

[1399] The user engages with interactive content and receives feedback from the system.

[1400] Prompt Sentence Examples

[1401] 1. When a user keeps failing to perform a specific operation in a video, analyze where they are getting stuck and use the emotion engine to provide appropriate guidance in real time when the user is frustrated.

[1402] In this way, by specifically defining the mode for carrying out the invention, it is possible to realize a system that improves the user experience and reduces stumbling blocks when using interactive content.

[1403] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1404] Step 1:

[1405] Retrieving User Interface Data

[1406] The server obtains user interface data through the API of the content distribution service. This data includes information about the screens and menus that the user operates. The obtained data is stored in a database on the server. The input is the user interface data from the API, and the output is the user interface information stored in the database.

[1407] Step 2:

[1408] Collecting operation logs

[1409] The terminal collects a log of the user's interaction with interactive content in real time and sends it to the server. This log includes clicks, selections, drags, opening and closing menus, etc. The input is the user's actions, and the output is the collected operation log data. The operation log is periodically sent to the server.

[1410] Step 3:

[1411] Analysis of operation logs

[1412] The server receives the operation logs sent from the device in real time and analyzes their contents. It uses a machine learning algorithm to detect stumbling points, such as repeated failures of a specific operation. The input is the operation logs, and the output is the identified stumbling point data.

[1413] Step 4:

[1414] Generate tutorial information

[1415] The server generates appropriate tutorial information based on the analysis results. The generated tutorial information is sent from the server to the terminal in text, audio, or video format. The input is the stumbling point data and user interface data, and the output is the generated tutorial information.

[1416] Step 5:

[1417] Providing tutorial information

[1418] The terminal provides the user with the tutorial information received from the server. In the case of audio guidance, the audio guidance is provided using text-to-speech (TTS) technology. The input is the tutorial information, and the output is guidance to the user. The terminal plays audio guidance such as "Click here" to the user.

[1419] Step 6:

[1420] Emotion recognition

[1421] The terminal recognizes the user's emotions using an emotion recognition device (smartphone, smart glasses, head-mounted display, etc.). The device analyzes the user's facial and voice data acquired in real time to identify the user's emotional state. This emotion data is sent to a server. The input is the user's facial and voice data, and the output is the identified emotion data.

[1422] Step 7:

[1423] Tutorial information adjustment based on emotions

[1424] The server receives the emotion data and adjusts the tutorial information according to the user's emotion. For example, if the user is frustrated, it generates more detailed and helpful guidance, and if the user is relaxed, it generates only brief instructions. The input is emotion data and tutorial information, and the output is the adjusted tutorial information. This information is then sent back to the terminal and provided to the user.

[1425] In this way, a system can be created that provides a comfortable user experience by preventing users from getting stuck on specific operations and by providing appropriate guidance based on their emotions.

[1426] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1427] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1428] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1429] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1430] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1431] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1432] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1433] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1434] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1435] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1436] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1437] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1438] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1439] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1440] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1441] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1442] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1443] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1444] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1445] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1446] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1447] The following is further disclosed regarding the above embodiment.

[1448] (Claim 1)

[1449] a means for obtaining user interface data;

[1450] A means of collecting user operation logs in real time,

[1451] A means for analyzing the operation log and identifying user stumbling blocks related to specific operations;

[1452] means for generating tutorial information corresponding to the stumbling block;

[1453] A system including means for providing said tutorial information to a user.

[1454] (Claim 2)

[1455] 10. The system of claim 1, further comprising means for providing said tutorial information audibly.

[1456] (Claim 3)

[1457] means for transmitting the user's operation log to a server;

[1458] 10. The system of claim 1, further comprising: means for receiving tutorial information from the server.

[1459] "Example 1"

[1460] (Claim 1)

[1461] a means for obtaining user interface data;

[1462] A means of collecting user operation logs in real time,

[1463] A means for analyzing the operation log and identifying user stumbling blocks related to specific operations;

[1464] means for generating tutorial information corresponding to the stumbling block;

[1465] means for providing said tutorial information to a user;

[1466] A means for training an AI model using machine learning algorithms; and

[1467] A means for the server to collect and analyze operation logs sent from the terminal in real time;

[1468] A system that includes a means for the device to provide tutorial information to the user by audio prompts or other means.

[1469] (Claim 2)

[1470] means for providing said tutorial information audibly.

[1471] 10. The system of claim 1.

[1472] (Claim 3)

[1473] means for transmitting the user's operation log to a server;

[1474] means for receiving tutorial information from the server

[1475] 10. The system of claim 1.

[1476] "Application Example 1"

[1477] (Claim 1)

[1478] a means for obtaining user interface data;

[1479] A means of collecting user operation logs in real time,

[1480] A means for analyzing the operation log and identifying user stumbling blocks related to specific operations;

[1481] means for generating tutorial information corresponding to the stumbling block;

[1482] means for providing said tutorial information to a user;

[1483] Means of collecting user location and behavioral data;

[1484] A means for analyzing the collected location information and behavioral data to generate location information for items desired by the user;

[1485] The system includes a means for providing the location information to the user in the form of an audio guide or map information.

[1486] (Claim 2)

[1487] 10. The system of claim 1, further comprising means for providing the tutorial information and location information by voice.

[1488] (Claim 3)

[1489] means for transmitting the user's operation log, location information, and behavior data to a server;

[1490] 10. The system of claim 1, further comprising means for receiving tutorial information and location information from the server.

[1491] "Example 2: Combining Emotion Engines"

[1492] (Claim 1)

[1493] a means for obtaining user interface data;

[1494] A means of collecting user operation logs in real time,

[1495] A means for analyzing the operation log and identifying user stumbling blocks related to specific operations;

[1496] means for generating tutorial information corresponding to the stumbling block;

[1497] means for providing said tutorial information to a user;

[1498] a means for collecting user emotional data;

[1499] The system includes means for adjusting tutorial information based on the emotion data.

[1500] (Claim 2)

[1501] 10. The system of claim 1, further comprising means for providing said tutorial information audibly.

[1502] (Claim 3)

[1503] means for transmitting the user's operation log to a server;

[1504] 10. The system of claim 1, further comprising: means for receiving tutorial information from the server.

[1505] "Application example 2 when combining emotion engines"

[1506] (Claim 1)

[1507] a means for obtaining user interface data;

[1508] A means of collecting user operation logs in real time,

[1509] A means for analyzing operation logs to identify user stumbling blocks related to specific operations;

[1510] A means for generating tutorial information corresponding to the stumbling block;

[1511] a means for providing tutorial information to users;

[1512] a means of recognizing a user's emotions;

[1513] The system includes a means for adjusting tutorial information based on the recognized emotion.

[1514] (Claim 2)

[1515] 10. The system of claim 1, further comprising means for providing the tutorial information audibly.

[1516] (Claim 3)

[1517] A means for transmitting user operation logs to a server;

[1518] 10. The system of claim 1, further comprising means for receiving tutorial information from a server. [Explanation of symbols]

[1519] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. a means for obtaining user interface data; A means of collecting user operation logs in real time, A means for analyzing the operation log and identifying user stumbling blocks related to specific operations; means for generating tutorial information corresponding to the stumbling block; A system including means for providing said tutorial information to a user.

2. 10. The system of claim 1, further comprising means for providing said tutorial information audibly.

3. means for transmitting the user's operation log to a server; 2. The system of claim 1, further comprising means for receiving tutorial information from said server.

Images