system

Abstract

We provide the system. [Solution] A means of inputting user actions, Means for transmitting input operation actions and status information of the information processing device via a communication network, A processing means that analyzes the received operation actions and state information of the information processing device, and generates operation commands for the information processing device based on the digital data generated using a generation AI model, A means for transmitting the generated operation command to an information processing device and controlling the state of the information processing device, A system that includes this.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a persona chatbot control method performed by at least one processor, including steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In recent years, the number of elderly smartphone users has been increasing, but problems such as the complexity of operation, accidental deletion of data due to incorrect operation, and confusion in settings frequently occur. As a result, users require a great deal of effort and time, and in some cases, may need specialized support, but the problem is that there is a lack of support means that can respond quickly and efficiently.

Means for Solving the Problems

[0005] This invention provides a system that collects user operation information and terminal status information via a communication network, and remotely transmits operation commands generated based on this information to the terminal for control. This enables an AI agent to quickly and appropriately resolve terminal problems, creating an environment where operational problems faced by the elderly can be easily solved.

[0006] "User operation information" refers to data related to actions and inputs that a user performs on a device.

[0007] "Device status information" refers to data that includes the terminal's current OS version, settings, and installed application information.

[0008] A "communication network" is a wireless or wired network used to send and receive information.

[0009] "Analysis" is the process of identifying a problem and deriving a solution based on the information received.

[0010] "Generated data" refers to data that includes operational instructions and recommended solutions obtained as a result of the analysis.

[0011] An "operation command" is a command that causes a device to perform a specific action.

[0012] "To control" means to change or adjust the operation or settings of a device to a desired state. [Brief explanation of the drawing]

[0013] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This is a conceptual diagram showing an example of the essential functions of a data processing device and a smart device according to the first embodiment. [Figure 3] This is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] It is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] It is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] It is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] It is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] It is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] It shows an emotion map to which a plurality of emotions are mapped. [Figure 10] It shows an emotion map to which a plurality of emotions are mapped. [Figure 11] It is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] It is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] It is a sequence diagram showing the processing flow of the data processing system in Example 2 when an emotion engine is combined. [Figure 14] It is a sequence diagram showing the processing flow of the data processing system in Application Example 2 when an emotion engine is combined.

Embodiments for Carrying Out the Invention

[0014] Hereinafter, an example of an embodiment of a system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0015] First, the terms used in the following description will be explained.

[0016] In the following embodiments, the numbered processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0017] In the following embodiments, the numbered RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0018] In the following embodiments, the numbered storage is one or more non-volatile storage devices that store various programs and various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes, and the like.

[0019] In the following embodiments, the numbered communication I / F (Interface) is an interface including a communication processor and an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applicable to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark), and the like.

[0020] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0021] [First Embodiment]

[0022] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0023] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0024] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0025] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0026] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0027] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0028] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0029] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0030] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0031] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0032] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0033] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0034] This invention provides a system for quickly resolving various problems that occur on a user's terminal. This system transmits user operation information and device status information to a server via a communication network, and an AI agent analyzes the problem and generates a solution, thereby executing remote operation commands for the terminal.

[0035] Users use a device with a dedicated support app installed. If a problem occurs, the user launches the app and reports the issue. The device automatically collects this information along with current status information and sends it to the server.

[0036] The server analyzes the received information and uses a trained AI model to derive the optimal solution. For example, if an application is accidentally deleted, the server generates an instruction to reinstall the application. This instruction is immediately sent to the user's device.

[0037] The terminal receives operation commands from the server and automatically executes the commanded procedure. If further confirmation from the user is required during this process, a confirmation message will be displayed on the terminal's screen. If the command is successful, the terminal will notify the user of the operation result and report that the problem has been resolved.

[0038] This system can handle a wide range of issues, from basic to complex, and is particularly effective for users who are unfamiliar with technical operations, such as the elderly. For example, even if multiple apps are crashing for unknown reasons, the server can send commands to optimize the settings of each app, quickly restoring them to a normal state. It can also resolve power supply problems and network connection issues by sending appropriate commands, minimizing user intervention.

[0039] The following describes the processing flow.

[0040] Step 1:

[0041] Users launch a dedicated support app on their device and report the problem they are experiencing by selecting the issue or by voice input. This inputs the user's operation information into the device.

[0042] Step 2:

[0043] The terminal collects operational information related to the reported problem and current terminal status information (OS version, installed applications, network status, etc.) and sends this information to the server.

[0044] Step 3:

[0045] The server analyzes the operation and status information received from the terminal. Using the accumulated database and trained AI models, it identifies the cause of the problem and generates the optimal solution.

[0046] Step 4:

[0047] Based on the generated data, the server generates specific operation commands for the terminal and transmits them to the terminal via the communication network.

[0048] Step 5:

[0049] The terminal interprets the operation commands received from the server and automatically performs the actions in accordance with the instructions. If user confirmation is required, a confirmation message will be displayed on the terminal screen, and the terminal will proceed to the next step only after obtaining the user's consent.

[0050] Step 6:

[0051] The terminal reports the results of the operations performed to the server and simultaneously notifies the user of the results. The user can then confirm that the problem has been successfully resolved and request additional support if necessary.

[0052] (Example 1)

[0053] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0054] Modern information terminals possess diverse functions, allowing users to perform a wide range of operations, but this also increases the likelihood of problems occurring. In particular, there is a lack of means to quickly and appropriately resolve problems when users unfamiliar with technology encounter them. Furthermore, providing support remotely is difficult, as it is challenging to accurately understand the user's terminal status and provide efficient instructions. Therefore, there is a need for methods to reduce the burden on users and enable appropriate troubleshooting.

[0055] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0056] In this invention, the server includes means for analyzing received operation information and status information, and using a generated AI model to identify problems and derive solutions; means for transmitting the generated operation commands to a device, which then automatically executes the operation commands, and displays a confirmation message if user confirmation is required; and means for inputting user operation information. This makes it possible to quickly and appropriately resolve technical problems faced by the user and minimize the user's effort.

[0057] "User operation information" refers to information about the operations and inputs that a user performs on their device.

[0058] "Device status information" refers to information regarding the operating status of the terminal's hardware and software.

[0059] A "communication network" refers to the network infrastructure used for sending and receiving data.

[0060] A "generative AI model" refers to an artificial intelligence model that learns from past data and examples to analyze specific problems and provide solutions.

[0061] An "operation command" refers to a command given to a device in order to perform a specific operation.

[0062] "Encryption technology" refers to the technology used to encode data in order to prevent unauthorized access or tampering during data transmission and reception.

[0063] A "prompt message" refers to instructions or messages that provide information to the user or the system.

[0064] This system is an integrated platform for quickly resolving various technical problems that occur on the user's device. Users install a dedicated support app on their device and report problems through this app when they arise. The device collects user-entered operation information and device status information and transmits it to the server via a communication network.

[0065] The server receives and analyzes transmitted information using a high-performance computing infrastructure. This analysis utilizes a generative AI model, which, having learned from a large amount of trouble data and solution examples, can identify problems with high accuracy and derive optimal solutions. For example, if common problems such as accidental application deletion or configuration errors are detected, the server quickly formulates a solution and generates corresponding operational commands.

[0066] The generated operation commands are transmitted to the user's terminal via the communication network. The terminal receives these commands and automatically performs the necessary operations based on them. Specifically, these operations include reinstalling applications, modifying configuration files, and restarting the system. If user confirmation is required during this process, a confirmation message is displayed through the user interface on the terminal to request the user's consent. After the work is completed, the terminal reports to the user that the problem has been resolved.

[0067] As a concrete example of this system, in cases where multiple applications are experiencing unexplained crashes, the server can create commands to optimize the settings of each application, quickly restoring them to a normal state. Furthermore, it can resolve power supply issues and network connection problems through appropriate commands.

[0068] A good example of a prompt would be, "A specific app on my device crashes frequently. Please tell me the possible causes and solutions." This requires concise and clear instructions that describe the user's specific problem. This allows the generative AI model to provide more accurate solutions.

[0069] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0070] Step 1:

[0071] Users report problems using a dedicated support app installed on their device. During this process, users input specific problems as text through the app's interface. The device also automatically collects status information, such as currently running apps and network connectivity. This input information is combined with user operation data and device status information.

[0072] Step 2:

[0073] The terminal transmits user operation information and device status information to the server via the communication network, protecting the data using encryption technology. During this process, data encryption is performed to ensure that the input data reaches the server securely without security risks. The output consists of encrypted user operation information and device status information.

[0074] Step 3:

[0075] The server receives encrypted data, decrypts it to obtain the original user operation information and device status information. The server then analyzes this information using a generative AI model. Specifically, it performs pattern matching and predictive calculations with a database based on the input data to identify the root cause of the problem. As a result of this process, it outputs the optimal solution for resolving the problem.

[0076] Step 4:

[0077] The server generates specific operational commands based on the solutions derived by the generated AI model. These commands include specific processes and changes to be performed on the terminal. These commands are then transmitted to the user's terminal via the communication network. The output is operational commands in an executable format.

[0078] Step 5:

[0079] The terminal analyzes the operation commands received from the server and automatically executes the necessary steps based on them. Following the commands, the terminal might, for example, reinstall a specific application or change system settings. If user confirmation is required during this process, the terminal displays a confirmation message through the user interface. The output is the result of the performed operation.

[0080] Step 6:

[0081] After all operations are completed, the device notifies the user of the results of the processes performed. This notification includes confirmation that the problem has been resolved and details of the specific processes that were carried out. This allows the user to continue using the device with peace of mind. The output is the notification of the processing results to the user.

[0082] (Application Example 1)

[0083] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0084] Technical problems that occur during the use of information processing devices are a major source of frustration, especially for users unfamiliar with technology. Such problems cause significant inconvenience to users and can lead to inaccurate operations and misunderstandings. In particular, in the context of electronic payments, such malfunctions can result in direct economic losses and a diminished user experience, making these issues urgently resolvable.

[0085] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0086] In this invention, the server includes means for inputting user actions, means for transmitting the input actions and status information of the information processing device via a communication network, and processing means for analyzing the received actions and status information of the information processing device and generating operation commands for the information processing device based on digital data generated using a generation AI model. This makes it possible to quickly and automatically derive solutions when technical problems occur and to maintain the normal operation of the information processing device.

[0087] "User actions" refer to a series of operations and inputs performed by a user during the process of using an information processing device.

[0088] "Status information of an information processing device" refers to data that represents the operating status and settings of an information processing device, and is used for system diagnosis and analysis.

[0089] A "communication network" refers to the network infrastructure used to send and receive data, and includes the internet and dedicated lines.

[0090] A "generative AI model" is a model trained using artificial intelligence technology for a specific purpose, and it analyzes problems and derives solutions based on the data it receives.

[0091] "Digital data" refers to information that is represented electronically and can be used in various forms such as numbers and strings of characters.

[0092] An "operation command for an information processing device" refers to an instruction generated to cause an information processing device to perform a specific operation.

[0093] "Adjusting the visual display" refers to the act of changing or optimizing the content displayed on the screen of an information processing device.

[0094] This invention provides a system for automatically and quickly solving technical problems in information processing equipment. The system is connected to a user's terminal via a communication network and operates in cooperation with a secure server.

[0095] The server receives user actions and information about the state of the information processing device, and uses a generated AI model based on this information to analyze the problem. Suitable AI platforms for this model include commonly used platforms such as TENSORFLOW® and PyTorch. The server analyzes the received digital data, generates optimal action commands, and transmits them to the information processing device.

[0096] When a terminal receives an operational command from the server, it automatically responds based on the instruction. This process may include adjusting the visual display. The user receives a notification on the terminal screen about how the problem was addressed. In particular, in the context of electronic payments, the quick and effective handling of problems ensures that the user's operation is completed successfully.

[0097] Server-based analysis requires referencing data from similar past problems. This method allows for more accurate identification of the cause and solution to errors. For example, if the app crashes while paying at a cafe, the server will prompt the user to restart the problematic module and check the network status, attempting to reconnect automatically.

[0098] An example of a prompt message is: "The app crashed while you were paying at the cafe. Please check your current network status and version information, and attempt to automatically reconnect and fix the module."

[0099] This system enables users of information processing equipment to solve problems efficiently, regardless of their technical skills.

[0100] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0101] Step 1:

[0102] The terminal collects user interaction information and status information from the information processing device. Specifically, the terminal acquires and records data from sensors and input devices. Inputs include user touch operations and application status. As output, this information is prepared for transmission to the server.

[0103] Step 2:

[0104] The terminal transmits collected operational behavior and status information to the server via the communication network. Specifically, the terminal encrypts the data before sending it to the server using the network protocol. The input is the data collected in step 1. The output is the data securely delivered to the server.

[0105] Step 3:

[0106] The server receives operational actions and status information transmitted from the terminal. Specifically, the server receives data via an internet gateway and stores it in secure storage. The input includes all data sent from the terminal. The output is organized data ready for analysis.

[0107] Step 4:

[0108] The server analyzes the received data using a generative AI model. Specifically, the server inputs data into the model to identify the problem and infer the optimal solution. The data sorted in step 3 is used as input. The output derives action commands and recommended solutions.

[0109] Step 5:

[0110] The server sends the generated operation command to the terminal. Specifically, the server re-encrypts the operation command and sends it to the terminal via the communication network. The operation command obtained in step 4 is used as input. The specific command that the terminal should receive is sent as output.

[0111] Step 6:

[0112] The terminal automatically performs actions according to the received operational commands. Specifically, the terminal may restart applications or adjust settings. The input is commands sent from the server. The output is that the information processing device returns to normal operation and the user is notified accordingly.

[0113] Step 7:

[0114] The user confirms the results through visual notifications from the device. Specifically, the device displays the operation results and problem resolution reports on its screen. The execution results obtained in step 6 are used as input. The output provides information that allows the user to confirm that the problem has been resolved.

[0115] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0116] This invention provides a system that assists in troubleshooting smart devices by combining an emotion engine that recognizes the user's emotions. This system has the function of generating commands for problem solving based on the user's operation information and terminal status information, and further adjusting those commands according to the user's emotional state.

[0117] Users use a dedicated support app to report problems when they occur. The device's built-in emotion engine analyzes the user's facial expressions and voice tone to obtain real-time emotional information. This information plays a crucial role in the troubleshooting process.

[0118] The emotional information acquired by the device, along with operation and state information, is sent to the server. The server analyzes this information and uses an AI model to generate optimal operation commands. Furthermore, based on the emotional information, it is possible to adjust the operation commands, for example, by suggesting more careful and step-by-step procedures if the user is experiencing stress, even for the same solution.

[0119] For example, if a user accidentally deletes an application and feels anxious or worried, the emotion engine recognizes this emotion, and the server generates instructions that, in addition to the usual reinstallation procedure, display guidance messages to reassure the user. This allows the user to confidently follow the instructions and resolve the problem efficiently.

[0120] This system enables flexible and personalized support that takes into account the user's psychological state, providing a supportive environment that is particularly user-friendly for the elderly and those who have anxieties about technology.

[0121] The following describes the processing flow.

[0122] Step 1:

[0123] The user launches the support app on their smart device and checks the problem they are experiencing. The troubleshooting process begins when the user selects the problem within the app or reports it by voice.

[0124] Step 2:

[0125] The device collects user operation information and device status information (e.g., OS version and app installation status). Simultaneously, an emotion engine analyzes emotional information in real time from the user's facial expressions and tone of voice, and this is also collected.

[0126] Step 3:

[0127] The device transmits collected operation information, status information, and emotion information to the server. This data is encrypted and securely transmitted over the communication network.

[0128] Step 4:

[0129] The server uses an AI model to analyze the problem based on the information it receives and creates optimal instructions that take into account the user's emotional state. For example, if the emotion engine determines that the user is feeling stressed, the server will include a reassuring message in the instructions.

[0130] Step 5:

[0131] The server sends the generated operation command to the user's terminal. This command includes specific steps for resolving the problem, and, if necessary, advice to support the user.

[0132] Step 6:

[0133] The device will follow the received operation commands and perform the necessary steps to resolve the problem. During the process, interactive messages will appear on the device screen if user confirmation or consent is required. This makes it easier for the user to understand the procedure and proceed with confidence.

[0134] Step 7:

[0135] The terminal reports to the server that the operation is complete and also notifies the user that the problem has been resolved. If necessary, it provides the user with the option to request further support.

[0136] (Example 2)

[0137] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0138] While there is a need to provide flexible and personalized support that takes into account the user's emotional state when they encounter technical problems while using smart devices, conventional systems have been unable to troubleshoot problems that reflect the user's psychological state. As a result, it has been difficult to provide appropriate and reassuring support, especially for elderly users and those who are technologically insecure.

[0139] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0140] In this invention, the server includes means for inputting user operation information, means for transmitting the input operation information and device status information via a communication network, emotion recognition means for analyzing the user's emotional state based on the transmitted information and generating emotion information, processing means for analyzing the received operation information, device status information and emotion information, generating device operation commands using a generation AI model and adjusting the commands according to the user's emotional state, and means for transmitting the generated operation commands to the device, controlling the device's state, and providing the commands to the user. This enables flexible and effective problem solving that takes into account the user's psychological state.

[0141] "User operation information" refers to input data generated when a user operates a smart device.

[0142] "Device status information" refers to data that shows the current operating status and settings of a smart device.

[0143] A "communication network" is a network infrastructure used to send and receive data from a terminal to a server, or vice versa.

[0144] "Emotion recognition means" refers to technology that analyzes a user's facial expressions and voice data to determine the user's psychological state.

[0145] A "generative AI model" is a framework of artificial intelligence that includes algorithms that derive optimal solutions and instructions based on data.

[0146] "Processing means" refers to a machine or program that analyzes received data, generates necessary commands, and outputs them.

[0147] "Means for controlling the state of the device" refers to technology that adjusts and manages the operation of a smart device based on generated operation commands.

[0148] This invention relates to a smart device troubleshooting system that takes user emotions into consideration. The system is initiated when a user reports a problem using a dedicated support app. The user inputs the specific problem or situation that occurred on the device into the app. The app utilizes the built-in camera and microphone to capture the user's facial expressions and voice tone, and analyzes the user's emotions using emotion recognition means.

[0149] The terminal transmits acquired user operation information, device status information, and emotional information to the server via the communication network. The server uses a generative AI model to analyze the received information and generate optimal operation commands. In doing so, the server takes the user's emotional information into consideration and adjusts the commands accordingly. For example, if the user is feeling stressed, the command will be modified to provide more detailed guidance.

[0150] As a concrete example, consider a situation where a user has accidentally deleted an application and is feeling anxious. In this case, the server can send a reassuring guidance message in addition to the usual reinstallation procedure. The system is designed to provide reassurance to the user by adding a message such as, "The app will be available again soon. Please relax and wait."

[0151] For example, one prompt to input into a generative AI model might be, "Generate steps to take if a user has accidentally deleted an app and is panicking." By generating emotionally sensitive instructions in this way, users can solve problems more efficiently.

[0152] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0153] Step 1:

[0154] When a user experiences a problem with their smart device, they launch a dedicated support app and enter details about the problem and how it occurred. The information entered includes operational details such as the nature of the problem and a description of the circumstances at the time it occurred. Based on this information, the app generates input data to understand the overview of the problem.

[0155] Step 2:

[0156] The device uses its built-in camera and microphone to acquire user facial expression data and voice data. This collects data that reflects the user's emotional state. The device inputs this data into an emotion recognition system and acquires emotional information in real time. As a result of the analysis, emotional information indicating the user's psychological state (e.g., anxiety, worry) is output.

[0157] Step 3:

[0158] The terminal transmits acquired user operation information, device status information, and emotional information to the server via the communication network. This data integration process creates an information package containing the user's operation history, emotional state, and device status. The server uses the received data as the basic input for analysis.

[0159] Step 4:

[0160] The server generates optimal operation commands using a generative AI model based on received operation information, device status information, and emotion information. This process involves data calculations to determine appropriate troubleshooting steps based on data analysis. The output is a set of basic commands for problem solving.

[0161] Step 5:

[0162] The server takes the user's emotional information into account and adjusts the generated instructions accordingly. Specifically, if the user is experiencing stress, the instructions are modified to include more polite and easy-to-understand steps. This emotion-based adjustment process adds reassuring guidance messages. As output, emotion-sensitive instructions are generated.

[0163] Step 6:

[0164] The device provides the user with pre-configured instructions sent from the server. These instructions, displayed on the screen, include details about what the user should do next. For example, a message like, "Please reinstall the app. Please relax and wait," might be displayed as a specific instruction. This allows the user to follow the instructions with reduced stress.

[0165] (Application Example 2)

[0166] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0167] In recent years, smart devices have become more diverse, and the age range and technical proficiency of users have also broadened. As a result, users are increasingly experiencing stress when operating devices or troubleshooting problems. Furthermore, traditional systems tend to provide uniform responses without considering user emotions, which can amplify anxiety and confusion. This poses a particular challenge for elderly and technologically unfamiliar users, as it increases psychological burden and makes efficient problem-solving more difficult.

[0168] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0169] In this invention, the server includes means for acquiring user operation information, means for collecting device status information, and means for acquiring user emotion information using an emotion recognition module. This enables flexible and personalized troubleshooting tailored to the user's emotional state.

[0170] "User operation information" refers to log data of inputs and operations performed by the user on the device.

[0171] "Device status information" refers to data related to the internal operation of the device itself, such as its operating status and error status.

[0172] "Means of transmission via a communication network" refers to systems and protocols for transmitting information to external servers via the internet or other networks.

[0173] An "emotion recognition module" is software or hardware that analyzes a user's facial expressions, tone of voice, etc., to estimate their emotions at that time.

[0174] "Processing means" refers to the software or hardware component that analyzes the collected data and generates commands or feedback based on that analysis.

[0175] "Generating operation commands" refers to the act of creating specific instructions based on acquired data to determine the operation of the device and how to respond to the user.

[0176] A "speech synthesis device" is a device that converts text data into speech and outputs it as speech through a speaker or similar device.

[0177] "Engaging in dialogue" refers to the process by which a user and a machine respond to the user's instructions and questions.

[0178] The system for implementing this invention utilizes operation information obtained from the device's user interface and state information obtained from the device's internal sensors. Furthermore, it collects user emotion information using peripheral devices such as cameras and microphones equipped with emotion recognition modules.

[0179] The server receives this information via the communication network and uses an AI model for data analysis. The analyzed data is used to generate optimal operating instructions to control the device's state. Based on emotional information, if the server determines that the user is experiencing stress, it provides the user with more careful and step-by-step instructions. This process utilizes AI development frameworks such as TensorFlow and PyTorch.

[0180] The terminal performs actions such as changing the screen display and engaging in emotionally sensitive dialogue using a speech synthesis device, based on operation commands sent from the server. Specifically, it can utilize speech synthesis technology to provide navigation messages to the user in a soft tone.

[0181] For example, if a user appears confused while using the device, the device can emit a voice message saying, "You seem to be having trouble, is there anything I can do to help?" Also, if a user accidentally deletes an application, the server will guide them through the reinstallation process and display a reassuring message on the screen.

[0182] An example of a prompt statement is, "Provide guidance on how to generate reassuring messages when the user appears anxious." This allows the program to improve the user experience and provide flexible support tailored to the specific application.

[0183] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0184] Step 1:

[0185] The user operates the device, generating operational information in the process. This operational information includes log data such as touch input, button presses, and application launches and shutdowns. The device's internal sensors acquire device status information, including battery level and memory usage. The input consists of operational and status information, and the output is a dataset of this information.

[0186] Step 2:

[0187] The device uses an emotion recognition module to detect the user's facial expressions and voice tone in real time through the camera and microphone. The input obtained here is the user's image and audio data, and emotional information is output based on this. By analyzing the data using an emotion recognition algorithm, the user's emotional state is understood.

[0188] Step 3:

[0189] The terminal transmits acquired operation information, status information, and emotion information to the server via the communication network. The input is this information, and the output is the appropriate data transfer to the server. The transmitted information becomes an integrated dataset on the server.

[0190] Step 4:

[0191] The server receives an integrated dataset and analyzes it using a generative AI model. The input is the received dataset, and the output is the analysis result. Through data analysis, it generates optimal operational commands. Furthermore, it adjusts the command content based on emotional information to create commands that are appropriate for the user's psychological state.

[0192] Step 5:

[0193] The server sends the generated operation commands to the terminal. The input is the operation commands constructed by the server, and the output is the transmission of commands to the terminal. Based on the transmitted operation commands, the terminal adjusts the user's environment.

[0194] Step 6:

[0195] The terminal executes received operation commands and provides feedback to the user. The input is the received operation command, and the output is the changed state of the device and feedback to the user. This includes specific actions such as changing the screen display and providing user-friendly voice guidance using a speech synthesizer. This allows the user to use the device with confidence.

[0196] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0197] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0198] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0199] [Second Embodiment]

[0200] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0201] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0202] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0203] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0204] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0205] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0206] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0207] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0208] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0209] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0210] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0211] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0212] This invention provides a system for quickly resolving various problems that occur on a user's terminal. This system transmits user operation information and device status information to a server via a communication network, and an AI agent analyzes the problem and generates a solution, thereby executing remote operation commands for the terminal.

[0213] Users use a device with a dedicated support app installed. If a problem occurs, the user launches the app and reports the issue. The device automatically collects this information along with current status information and sends it to the server.

[0214] The server analyzes the received information and uses a trained AI model to derive the optimal solution. For example, if an application is accidentally deleted, the server generates an instruction to reinstall the application. This instruction is immediately sent to the user's device.

[0215] The terminal receives operation commands from the server and automatically executes the commanded procedure. If further confirmation from the user is required during this process, a confirmation message will be displayed on the terminal's screen. If the command is successful, the terminal will notify the user of the operation result and report that the problem has been resolved.

[0216] This system can handle a wide range of issues, from basic to complex, and is particularly effective for users who are unfamiliar with technical operations, such as the elderly. For example, even if multiple apps are crashing for unknown reasons, the server can send commands to optimize the settings of each app, quickly restoring them to a normal state. It can also resolve power supply problems and network connection issues by sending appropriate commands, minimizing user intervention.

[0217] The following describes the processing flow.

[0218] Step 1:

[0219] Users launch a dedicated support app on their device and report the problem they are experiencing by selecting the issue or by voice input. This inputs the user's operation information into the device.

[0220] Step 2:

[0221] The terminal collects operational information related to the reported problem and current terminal status information (OS version, installed applications, network status, etc.) and sends this information to the server.

[0222] Step 3:

[0223] The server analyzes the operation and status information received from the terminal. Using the accumulated database and trained AI models, it identifies the cause of the problem and generates the optimal solution.

[0224] Step 4:

[0225] Based on the generated data, the server generates specific operation commands for the terminal and transmits them to the terminal via the communication network.

[0226] Step 5:

[0227] The terminal interprets the operation commands received from the server and automatically performs the actions in accordance with the instructions. If user confirmation is required, a confirmation message will be displayed on the terminal screen, and the terminal will proceed to the next step only after obtaining the user's consent.

[0228] Step 6:

[0229] The terminal reports the results of the operations performed to the server and simultaneously notifies the user of the results. The user can then confirm that the problem has been successfully resolved and request additional support if necessary.

[0230] (Example 1)

[0231] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0232] Modern information terminals possess diverse functions, allowing users to perform a wide range of operations, but this also increases the likelihood of problems occurring. In particular, there is a lack of means to quickly and appropriately resolve problems when users unfamiliar with technology encounter them. Furthermore, providing support remotely is difficult, as it is challenging to accurately understand the user's terminal status and provide efficient instructions. Therefore, there is a need for methods to reduce the burden on users and enable appropriate troubleshooting.

[0233] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0234] In this invention, the server includes means for analyzing received operation information and status information, and using a generated AI model to identify problems and derive solutions; means for transmitting the generated operation commands to a device, which then automatically executes the operation commands, and displays a confirmation message if user confirmation is required; and means for inputting user operation information. This makes it possible to quickly and appropriately resolve technical problems faced by the user and minimize the user's effort.

[0235] "User operation information" refers to information about the operations and inputs that a user performs on their device.

[0236] "Device status information" refers to information regarding the operating status of the terminal's hardware and software.

[0237] A "communication network" refers to the network infrastructure used for sending and receiving data.

[0238] A "generative AI model" refers to an artificial intelligence model that learns from past data and examples to analyze specific problems and provide solutions.

[0239] An "operation command" refers to a command given to a device in order to perform a specific operation.

[0240] "Encryption technology" refers to the technology used to encode data in order to prevent unauthorized access or tampering during data transmission and reception.

[0241] A "prompt message" refers to instructions or messages that provide information to the user or the system.

[0242] This system is an integrated platform for quickly resolving various technical problems that occur on the user's device. Users install a dedicated support app on their device and report problems through this app when they arise. The device collects user-entered operation information and device status information and transmits it to the server via a communication network.

[0243] The server receives and analyzes transmitted information using a high-performance computing infrastructure. This analysis utilizes a generative AI model, which, having learned from a large amount of trouble data and solution examples, can identify problems with high accuracy and derive optimal solutions. For example, if common problems such as accidental application deletion or configuration errors are detected, the server quickly formulates a solution and generates corresponding operational commands.

[0244] The generated operation commands are transmitted to the user's terminal via the communication network. The terminal receives these commands and automatically performs the necessary operations based on them. Specifically, these operations include reinstalling applications, modifying configuration files, and restarting the system. If user confirmation is required during this process, a confirmation message is displayed through the user interface on the terminal to request the user's consent. After the work is completed, the terminal reports to the user that the problem has been resolved.

[0245] As a concrete example of this system, in cases where multiple applications are experiencing unexplained crashes, the server can create commands to optimize the settings of each application, quickly restoring them to a normal state. Furthermore, it can resolve power supply issues and network connection problems through appropriate commands.

[0246] A good example of a prompt would be, "A specific app on my device crashes frequently. Please tell me the possible causes and solutions." This requires concise and clear instructions that describe the user's specific problem. This allows the generative AI model to provide more accurate solutions.

[0247] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0248] Step 1:

[0249] Users report problems using a dedicated support app installed on their device. During this process, users input specific problems as text through the app's interface. The device also automatically collects status information, such as currently running apps and network connectivity. This input information is combined with user operation data and device status information.

[0250] Step 2:

[0251] The terminal transmits user operation information and device status information to the server via the communication network, protecting the data using encryption technology. During this process, data encryption is performed to ensure that the input data reaches the server securely without security risks. The output consists of encrypted user operation information and device status information.

[0252] Step 3:

[0253] The server receives encrypted data, decrypts it to obtain the original user operation information and device status information. The server then analyzes this information using a generative AI model. Specifically, it performs pattern matching and predictive calculations with a database based on the input data to identify the root cause of the problem. As a result of this process, it outputs the optimal solution for resolving the problem.

[0254] Step 4:

[0255] The server generates specific operational commands based on the solutions derived by the generated AI model. These commands include specific processes and changes to be performed on the terminal. These commands are then transmitted to the user's terminal via the communication network. The output is operational commands in an executable format.

[0256] Step 5:

[0257] The terminal analyzes the operation commands received from the server and automatically executes the necessary steps based on them. Following the commands, the terminal might, for example, reinstall a specific application or change system settings. If user confirmation is required during this process, the terminal displays a confirmation message through the user interface. The output is the result of the performed operation.

[0258] Step 6:

[0259] After all operations are completed, the device notifies the user of the results of the processes performed. This notification includes confirmation that the problem has been resolved and details of the specific processes that were carried out. This allows the user to continue using the device with peace of mind. The output is the notification of the processing results to the user.

[0260] (Application Example 1)

[0261] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0262] Technical problems that occur during the use of information processing devices are a major source of frustration, especially for users unfamiliar with technology. Such problems cause significant inconvenience to users and can lead to inaccurate operations and misunderstandings. In particular, in the context of electronic payments, such malfunctions can result in direct economic losses and a diminished user experience, making these issues urgently resolvable.

[0263] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0264] In this invention, the server includes means for inputting user actions, means for transmitting the input actions and status information of the information processing device via a communication network, and processing means for analyzing the received actions and status information of the information processing device and generating operation commands for the information processing device based on digital data generated using a generation AI model. This makes it possible to quickly and automatically derive solutions when technical problems occur and to maintain the normal operation of the information processing device.

[0265] "User actions" refer to a series of operations and inputs performed by a user during the process of using an information processing device.

[0266] "Status information of an information processing device" refers to data that represents the operating status and settings of an information processing device, and is used for system diagnosis and analysis.

[0267] A "communication network" refers to the network infrastructure used to send and receive data, and includes the internet and dedicated lines.

[0268] A "generative AI model" is a model trained using artificial intelligence technology for a specific purpose, and it analyzes problems and derives solutions based on the data it receives.

[0269] "Digital data" refers to information that is represented electronically and can be used in various forms such as numbers and strings of characters.

[0270] An "operation command for an information processing device" refers to an instruction generated to cause an information processing device to perform a specific operation.

[0271] "Adjusting the visual display" refers to the act of changing or optimizing the content displayed on the screen of an information processing device.

[0272] This invention provides a system for automatically and quickly solving technical problems in information processing equipment. The system is connected to a user's terminal via a communication network and operates in cooperation with a secure server.

[0273] The server receives user actions and information about the state of the information processing device, and uses a generated AI model based on this information to analyze the problem. Suitable AI platforms for this model include commonly used platforms such as TensorFlow and PyTorch. The server analyzes the received digital data, generates optimal action commands, and sends them to the information processing device.

[0274] When a terminal receives an operational command from the server, it automatically responds based on the instruction. This process may include adjusting the visual display. The user receives a notification on the terminal screen about how the problem was addressed. In particular, in the context of electronic payments, the quick and effective handling of problems ensures that the user's operation is completed successfully.

[0275] Server-based analysis requires referencing data from similar past problems. This method allows for more accurate identification of the cause and solution to errors. For example, if the app crashes while paying at a cafe, the server will prompt the user to restart the problematic module and check the network status, attempting to reconnect automatically.

[0276] An example of a prompt message is: "The app crashed while you were paying at the cafe. Please check your current network status and version information, and attempt to automatically reconnect and fix the module."

[0277] This system enables users of information processing equipment to solve problems efficiently, regardless of their technical skills.

[0278] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0279] Step 1:

[0280] The terminal collects the user's operation actions and the status information of the information processing device. As a specific operation, the terminal acquires data from sensors and input devices and records it. The inputs include the user's touch operations and the status of applications. As an output, these pieces of information are prepared to be sent to the server.

[0281] Step 2:

[0282] The terminal sends the collected operation actions and status information to the server through the communication network. As a specific operation, the terminal encrypts the data and then uses the network protocol to send the data to the server. The input uses the data collected in Step 1. As an output, the data is safely delivered to the server.

[0283] Step 3:

[0284] The server receives the operation actions and status information sent from the terminal. As a specific operation, the server receives the data through the Internet gateway and stores it in a secure storage. The input includes all the data sent from the terminal. As an output, the data that can be analyzed is sorted out.

[0285] Step 4:

[0286] The server analyzes the received data using the generated AI model. As a specific operation, the server inputs the data into the model and conducts problem identification and inference of the optimal solution. The input uses the data in the sorted state in Step 3. As an output, operation instructions and recommended solutions are derived.

[0287] Step 5:

[0288] The server sends the generated operation command to the terminal. Specifically, the server re-encrypts the operation command and sends it to the terminal via the communication network. The operation command obtained in step 4 is used as input. The specific command that the terminal should receive is sent as output.

[0289] Step 6:

[0290] The terminal automatically performs actions according to the received operational commands. Specifically, the terminal may restart applications or adjust settings. The input is commands sent from the server. The output is that the information processing device returns to normal operation and the user is notified accordingly.

[0291] Step 7:

[0292] The user confirms the results through visual notifications from the device. Specifically, the device displays the operation results and problem resolution reports on its screen. The execution results obtained in step 6 are used as input. The output provides information that allows the user to confirm that the problem has been resolved.

[0293] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0294] This invention provides a system that assists in troubleshooting smart devices by combining an emotion engine that recognizes the user's emotions. This system has the function of generating commands for problem solving based on the user's operation information and terminal status information, and further adjusting those commands according to the user's emotional state.

[0295] Users use a dedicated support app to report problems when they occur. The device's built-in emotion engine analyzes the user's facial expressions and voice tone to obtain real-time emotional information. This information plays a crucial role in the troubleshooting process.

[0296] The emotional information acquired by the device, along with operation and state information, is sent to the server. The server analyzes this information and uses an AI model to generate optimal operation commands. Furthermore, based on the emotional information, it is possible to adjust the operation commands, for example, by suggesting more careful and step-by-step procedures if the user is experiencing stress, even for the same solution.

[0297] For example, if a user accidentally deletes an application and feels anxious or worried, the emotion engine recognizes this emotion, and the server generates instructions that, in addition to the usual reinstallation procedure, display guidance messages to reassure the user. This allows the user to confidently follow the instructions and resolve the problem efficiently.

[0298] This system enables flexible and personalized support that takes into account the user's psychological state, providing a supportive environment that is particularly user-friendly for the elderly and those who have anxieties about technology.

[0299] The following describes the processing flow.

[0300] Step 1:

[0301] The user launches the support app on their smart device and checks the problem they are experiencing. The troubleshooting process begins when the user selects the problem within the app or reports it by voice.

[0302] Step 2:

[0303] The terminal collects the user's operation information and the device's status information (e.g., OS version and app installation status). At the same time, the emotion engine analyzes emotion information in real time from the user's facial expressions and voice tones, which is also collected.

[0304] Step 3:

[0305] The terminal sends the collected operation information, status information, and emotion information to the server. This data is encrypted and securely transferred via the communication network.

[0306] Step 4:

[0307] Based on the received information, the server utilizes the AI model to analyze the problem and create an optimal operation command considering the user's emotional state. For example, if the emotion engine determines that the user is feeling stressed, the server will include a reassuring message in the command text.

[0308] Step 5:

[0309] The server sends the generated operation command to the user's terminal. This command includes specific operation procedures for problem-solving and may also include advice to support the user as needed.

[0310] Step 6:

[0311] The terminal executes the solution to the problem according to the received operation command. During the execution process, if confirmation or consent from the user is required, interactive messages will be displayed on the terminal screen. This makes it easier for the user to understand the procedure and proceed with the operation with confidence.

[0312] Step 7:

[0313] The terminal reports to the server that the operation has been completed and further notifies the user that the problem has been solved. If necessary, it provides the user with an option to request further support.

[0314] (Example 2)

[0315] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0316] While there is a need to provide flexible and personalized support that takes into account the user's emotional state when they encounter technical problems while using smart devices, conventional systems have been unable to troubleshoot problems that reflect the user's psychological state. As a result, it has been difficult to provide appropriate and reassuring support, especially for elderly users and those who are technologically insecure.

[0317] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0318] In this invention, the server includes means for inputting user operation information, means for transmitting the input operation information and device status information via a communication network, emotion recognition means for analyzing the user's emotional state based on the transmitted information and generating emotion information, processing means for analyzing the received operation information, device status information and emotion information, generating device operation commands using a generation AI model and adjusting the commands according to the user's emotional state, and means for transmitting the generated operation commands to the device, controlling the device's state, and providing the commands to the user. This enables flexible and effective problem solving that takes into account the user's psychological state.

[0319] "User operation information" refers to input data generated when a user operates a smart device.

[0320] "Device status information" refers to data that shows the current operating status and settings of a smart device.

[0321] A "communication network" is a network infrastructure used to send and receive data from a terminal to a server, or vice versa.

[0322] "Emotion recognition means" refers to technology that analyzes a user's facial expressions and voice data to determine the user's psychological state.

[0323] A "generative AI model" is a framework of artificial intelligence that includes algorithms that derive optimal solutions and instructions based on data.

[0324] "Processing means" refers to a machine or program that analyzes received data, generates necessary commands, and outputs them.

[0325] "Means for controlling the state of the device" refers to technology that adjusts and manages the operation of a smart device based on generated operation commands.

[0326] This invention relates to a smart device troubleshooting system that takes user emotions into consideration. The system is initiated when a user reports a problem using a dedicated support app. The user inputs the specific problem or situation that occurred on the device into the app. The app utilizes the built-in camera and microphone to capture the user's facial expressions and voice tone, and analyzes the user's emotions using emotion recognition means.

[0327] The terminal transmits acquired user operation information, device status information, and emotional information to the server via the communication network. The server uses a generative AI model to analyze the received information and generate optimal operation commands. In doing so, the server takes the user's emotional information into consideration and adjusts the commands accordingly. For example, if the user is feeling stressed, the command will be modified to provide more detailed guidance.

[0328] As a concrete example, consider a situation where a user has accidentally deleted an application and is feeling anxious. In this case, the server can send a reassuring guidance message in addition to the usual reinstallation procedure. The system is designed to provide reassurance to the user by adding a message such as, "The app will be available again soon. Please relax and wait."

[0329] For example, one prompt to input into a generative AI model might be, "Generate steps to take if a user has accidentally deleted an app and is panicking." By generating emotionally sensitive instructions in this way, users can solve problems more efficiently.

[0330] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0331] Step 1:

[0332] When a user experiences a problem with their smart device, they launch a dedicated support app and enter details about the problem and how it occurred. The information entered includes operational details such as the nature of the problem and a description of the circumstances at the time it occurred. Based on this information, the app generates input data to understand the overview of the problem.

[0333] Step 2:

[0334] The device uses its built-in camera and microphone to acquire user facial expression data and voice data. This collects data that reflects the user's emotional state. The device inputs this data into an emotion recognition system and acquires emotional information in real time. As a result of the analysis, emotional information indicating the user's psychological state (e.g., anxiety, worry) is output.

[0335] Step 3:

[0336] The terminal transmits acquired user operation information, device status information, and emotional information to the server via the communication network. This data integration process creates an information package containing the user's operation history, emotional state, and device status. The server uses the received data as the basic input for analysis.

[0337] Step 4:

[0338] The server generates optimal operation commands using a generative AI model based on received operation information, device status information, and emotion information. This process involves data calculations to determine appropriate troubleshooting steps based on data analysis. The output is a set of basic commands for problem solving.

[0339] Step 5:

[0340] The server takes the user's emotional information into account and adjusts the generated instructions accordingly. Specifically, if the user is experiencing stress, the instructions are modified to include more polite and easy-to-understand steps. This emotion-based adjustment process adds reassuring guidance messages. As output, emotion-sensitive instructions are generated.

[0341] Step 6:

[0342] The device provides the user with pre-configured instructions sent from the server. These instructions, displayed on the screen, include details about what the user should do next. For example, a message like, "Please reinstall the app. Please relax and wait," might be displayed as a specific instruction. This allows the user to follow the instructions with reduced stress.

[0343] (Application Example 2)

[0344] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the smart glasses 214 as the "terminal".

[0345] In recent years, smart devices have become more diverse, and the age range and technical proficiency of users have also broadened. As a result, users are increasingly experiencing stress when operating devices or troubleshooting problems. Furthermore, traditional systems tend to provide uniform responses without considering user emotions, which can amplify anxiety and confusion. This poses a particular challenge for elderly and technologically unfamiliar users, as it increases psychological burden and makes efficient problem-solving more difficult.

[0346] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0347] In this invention, the server includes means for acquiring user operation information, means for collecting device status information, and means for acquiring user emotion information using an emotion recognition module. This enables flexible and personalized troubleshooting tailored to the user's emotional state.

[0348] "User operation information" refers to log data of inputs and operations performed by the user on the device.

[0349] "Device status information" refers to data related to the internal operation of the device itself, such as its operating status and error status.

[0350] "Means of transmission via a communication network" refers to systems and protocols for transmitting information to external servers via the internet or other networks.

[0351] An "emotion recognition module" is software or hardware that analyzes a user's facial expressions, tone of voice, etc., to estimate their emotions at that time.

[0352] "Processing means" refers to the software or hardware component that analyzes the collected data and generates commands or feedback based on that analysis.

[0353] "Generating operation commands" refers to the act of creating specific instructions based on acquired data to determine the operation of the device and how to respond to the user.

[0354] A "speech synthesis device" is a device that converts text data into speech and outputs it as speech through a speaker or similar device.

[0355] "Engaging in dialogue" refers to the process by which a user and a machine respond to the user's instructions and questions.

[0356] The system for implementing this invention utilizes operation information obtained from the device's user interface and state information obtained from the device's internal sensors. Furthermore, it collects user emotion information using peripheral devices such as cameras and microphones equipped with emotion recognition modules.

[0357] The server receives this information via the communication network and uses an AI model for data analysis. The analyzed data is used to generate optimal operating instructions to control the device's state. Based on emotional information, if the server determines that the user is experiencing stress, it provides the user with more careful and step-by-step instructions. This process utilizes AI development frameworks such as TensorFlow and PyTorch.

[0358] The terminal performs actions such as changing the screen display and engaging in emotionally sensitive dialogue using a speech synthesis device, based on operation commands sent from the server. Specifically, it can utilize speech synthesis technology to provide navigation messages to the user in a soft tone.

[0359] For example, if a user appears confused while using the device, the device can emit a voice message saying, "You seem to be having trouble, is there anything I can do to help?" Also, if a user accidentally deletes an application, the server will guide them through the reinstallation process and display a reassuring message on the screen.

[0360] An example of a prompt statement is, "Provide guidance on how to generate reassuring messages when the user appears anxious." This allows the program to improve the user experience and provide flexible support tailored to the specific application.

[0361] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0362] Step 1:

[0363] The user operates the device, generating operational information in the process. This operational information includes log data such as touch input, button presses, and application launches and shutdowns. The device's internal sensors acquire device status information, including battery level and memory usage. The input consists of operational and status information, and the output is a dataset of this information.

[0364] Step 2:

[0365] The device uses an emotion recognition module to detect the user's facial expressions and voice tone in real time through the camera and microphone. The input obtained here is the user's image and audio data, and emotional information is output based on this. By analyzing the data using an emotion recognition algorithm, the user's emotional state is understood.

[0366] Step 3:

[0367] The terminal transmits acquired operation information, status information, and emotion information to the server via the communication network. The input is this information, and the output is the appropriate data transfer to the server. The transmitted information becomes an integrated dataset on the server.

[0368] Step 4:

[0369] The server receives an integrated dataset and analyzes it using a generative AI model. The input is the received dataset, and the output is the analysis result. Through data analysis, it generates optimal operational commands. Furthermore, it adjusts the command content based on emotional information to create commands that are appropriate for the user's psychological state.

[0370] Step 5:

[0371] The server sends the generated operation commands to the terminal. The input is the operation commands constructed by the server, and the output is the transmission of commands to the terminal. Based on the transmitted operation commands, the terminal adjusts the user's environment.

[0372] Step 6:

[0373] The terminal executes received operation commands and provides feedback to the user. The input is the received operation command, and the output is the changed state of the device and feedback to the user. This includes specific actions such as changing the screen display and providing user-friendly voice guidance using a speech synthesizer. This allows the user to use the device with confidence.

[0374] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0375] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0376] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0377] [Third Embodiment]

[0378] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0379] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0380] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0381] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0382] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0383] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0384] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0385] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0386] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0387] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0388] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0389] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0390] This invention provides a system for quickly resolving various problems that occur on a user's terminal. This system transmits user operation information and device status information to a server via a communication network, and an AI agent analyzes the problem and generates a solution, thereby executing remote operation commands for the terminal.

[0391] Users use a device with a dedicated support app installed. If a problem occurs, the user launches the app and reports the issue. The device automatically collects this information along with current status information and sends it to the server.

[0392] The server analyzes the received information and uses a trained AI model to derive the optimal solution. For example, if an application is accidentally deleted, the server generates an instruction to reinstall the application. This instruction is immediately sent to the user's device.

[0393] The terminal receives operation commands from the server and automatically executes the commanded procedure. If further confirmation from the user is required during this process, a confirmation message will be displayed on the terminal's screen. If the command is successful, the terminal will notify the user of the operation result and report that the problem has been resolved.

[0394] This system can handle a wide range of issues, from basic to complex, and is particularly effective for users who are unfamiliar with technical operations, such as the elderly. For example, even if multiple apps are crashing for unknown reasons, the server can send commands to optimize the settings of each app, quickly restoring them to a normal state. It can also resolve power supply problems and network connection issues by sending appropriate commands, minimizing user intervention.

[0395] The following describes the processing flow.

[0396] Step 1:

[0397] Users launch a dedicated support app on their device and report the problem they are experiencing by selecting the issue or by voice input. This inputs the user's operation information into the device.

[0398] Step 2:

[0399] The terminal collects operational information related to the reported problem and current terminal status information (OS version, installed applications, network status, etc.) and sends this information to the server.

[0400] Step 3:

[0401] The server analyzes the operation and status information received from the terminal. Using the accumulated database and trained AI models, it identifies the cause of the problem and generates the optimal solution.

[0402] Step 4:

[0403] Based on the generated data, the server generates specific operation commands for the terminal and transmits them to the terminal via the communication network.

[0404] Step 5:

[0405] The terminal interprets the operation commands received from the server and automatically performs the actions in accordance with the instructions. If user confirmation is required, a confirmation message will be displayed on the terminal screen, and the terminal will proceed to the next step only after obtaining the user's consent.

[0406] Step 6:

[0407] The terminal reports the results of the operations performed to the server and simultaneously notifies the user of the results. The user can then confirm that the problem has been successfully resolved and request additional support if necessary.

[0408] (Example 1)

[0409] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0410] Modern information terminals possess diverse functions, allowing users to perform a wide range of operations, but this also increases the likelihood of problems occurring. In particular, there is a lack of means to quickly and appropriately resolve problems when users unfamiliar with technology encounter them. Furthermore, providing support remotely is difficult, as it is challenging to accurately understand the user's terminal status and provide efficient instructions. Therefore, there is a need for methods to reduce the burden on users and enable appropriate troubleshooting.

[0411] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0412] In this invention, the server includes means for analyzing received operation information and status information, and using a generated AI model to identify problems and derive solutions; means for transmitting the generated operation commands to a device, which then automatically executes the operation commands, and displays a confirmation message if user confirmation is required; and means for inputting user operation information. This makes it possible to quickly and appropriately resolve technical problems faced by the user and minimize the user's effort.

[0413] "User operation information" refers to information about the operations and inputs that a user performs on their device.

[0414] "Device status information" refers to information regarding the operating status of the terminal's hardware and software.

[0415] A "communication network" refers to the network infrastructure used for sending and receiving data.

[0416] A "generative AI model" refers to an artificial intelligence model that learns from past data and examples to analyze specific problems and provide solutions.

[0417] An "operation command" refers to a command given to a device in order to perform a specific operation.

[0418] "Encryption technology" refers to the technology used to encode data in order to prevent unauthorized access or tampering during data transmission and reception.

[0419] A "prompt message" refers to instructions or messages that provide information to the user or the system.

[0420] This system is an integrated platform for quickly resolving various technical problems that occur on the user's device. Users install a dedicated support app on their device and report problems through this app when they arise. The device collects user-entered operation information and device status information and transmits it to the server via a communication network.

[0421] The server receives and analyzes transmitted information using a high-performance computing infrastructure. This analysis utilizes a generative AI model, which, having learned from a large amount of trouble data and solution examples, can identify problems with high accuracy and derive optimal solutions. For example, if common problems such as accidental application deletion or configuration errors are detected, the server quickly formulates a solution and generates corresponding operational commands.

[0422] The generated operation commands are transmitted to the user's terminal via the communication network. The terminal receives these commands and automatically performs the necessary operations based on them. Specifically, these operations include reinstalling applications, modifying configuration files, and restarting the system. If user confirmation is required during this process, a confirmation message is displayed through the user interface on the terminal to request the user's consent. After the work is completed, the terminal reports to the user that the problem has been resolved.

[0423] As a concrete example of this system, in cases where multiple applications are experiencing unexplained crashes, the server can create commands to optimize the settings of each application, quickly restoring them to a normal state. Furthermore, it can resolve power supply issues and network connection problems through appropriate commands.

[0424] A good example of a prompt would be, "A specific app on my device crashes frequently. Please tell me the possible causes and solutions." This requires concise and clear instructions that describe the user's specific problem. This allows the generative AI model to provide more accurate solutions.

[0425] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0426] Step 1:

[0427] Users report problems using a dedicated support app installed on their device. During this process, users input specific problems as text through the app's interface. The device also automatically collects status information, such as currently running apps and network connectivity. This input information is combined with user operation data and device status information.

[0428] Step 2:

[0429] The terminal transmits user operation information and device status information to the server via the communication network, protecting the data using encryption technology. During this process, data encryption is performed to ensure that the input data reaches the server securely without security risks. The output consists of encrypted user operation information and device status information.

[0430] Step 3:

[0431] The server receives encrypted data, decrypts it to obtain the original user operation information and device status information. The server then analyzes this information using a generative AI model. Specifically, it performs pattern matching and predictive calculations with a database based on the input data to identify the root cause of the problem. As a result of this process, it outputs the optimal solution for resolving the problem.

[0432] Step 4:

[0433] The server generates specific operational commands based on the solutions derived by the generated AI model. These commands include specific processes and changes to be performed on the terminal. These commands are then transmitted to the user's terminal via the communication network. The output is operational commands in an executable format.

[0434] Step 5:

[0435] The terminal analyzes the operation commands received from the server and automatically executes the necessary steps based on them. Following the commands, the terminal might, for example, reinstall a specific application or change system settings. If user confirmation is required during this process, the terminal displays a confirmation message through the user interface. The output is the result of the performed operation.

[0436] Step 6:

[0437] After all operations are completed, the device notifies the user of the results of the processes performed. This notification includes confirmation that the problem has been resolved and details of the specific processes that were carried out. This allows the user to continue using the device with peace of mind. The output is the notification of the processing results to the user.

[0438] (Application Example 1)

[0439] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0440] Technical problems that occur during the use of information processing devices are a major source of frustration, especially for users unfamiliar with technology. Such problems cause significant inconvenience to users and can lead to inaccurate operations and misunderstandings. In particular, in the context of electronic payments, such malfunctions can result in direct economic losses and a diminished user experience, making these issues urgently resolvable.

[0441] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0442] In this invention, the server includes means for inputting user actions, means for transmitting the input actions and status information of the information processing device via a communication network, and processing means for analyzing the received actions and status information of the information processing device and generating operation commands for the information processing device based on digital data generated using a generation AI model. This makes it possible to quickly and automatically derive solutions when technical problems occur and to maintain the normal operation of the information processing device.

[0443] "User actions" refer to a series of operations and inputs performed by a user during the process of using an information processing device.

[0444] "Status information of an information processing device" refers to data that represents the operating status and settings of an information processing device, and is used for system diagnosis and analysis.

[0445] A "communication network" refers to the network infrastructure used to send and receive data, and includes the internet and dedicated lines.

[0446] A "generative AI model" is a model trained using artificial intelligence technology for a specific purpose, and it analyzes problems and derives solutions based on the data it receives.

[0447] "Digital data" refers to information that is represented electronically and can be used in various forms such as numbers and strings of characters.

[0448] An "operation command for an information processing device" refers to an instruction generated to cause an information processing device to perform a specific operation.

[0449] "Adjusting the visual display" refers to the act of changing or optimizing the content displayed on the screen of an information processing device.

[0450] This invention provides a system for automatically and quickly solving technical problems in information processing equipment. The system is connected to a user's terminal via a communication network and operates in cooperation with a secure server.

[0451] The server receives user actions and information about the state of the information processing device, and uses a generated AI model based on this information to analyze the problem. Suitable AI platforms for this model include commonly used platforms such as TensorFlow and PyTorch. The server analyzes the received digital data, generates optimal action commands, and sends them to the information processing device.

[0452] When a terminal receives an operational command from the server, it automatically responds based on the instruction. This process may include adjusting the visual display. The user receives a notification on the terminal screen about how the problem was addressed. In particular, in the context of electronic payments, the quick and effective handling of problems ensures that the user's operation is completed successfully.

[0453] Server-based analysis requires referencing data from similar past problems. This method allows for more accurate identification of the cause and solution to errors. For example, if the app crashes while paying at a cafe, the server will prompt the user to restart the problematic module and check the network status, attempting to reconnect automatically.

[0454] An example of a prompt message is: "The app crashed while you were paying at the cafe. Please check your current network status and version information, and attempt to automatically reconnect and fix the module."

[0455] This system enables users of information processing equipment to solve problems efficiently, regardless of their technical skills.

[0456] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0457] Step 1:

[0458] The terminal collects user interaction information and status information from the information processing device. Specifically, the terminal acquires and records data from sensors and input devices. Inputs include user touch operations and application status. As output, this information is prepared for transmission to the server.

[0459] Step 2:

[0460] The terminal transmits collected operational behavior and status information to the server via the communication network. Specifically, the terminal encrypts the data before sending it to the server using the network protocol. The input is the data collected in step 1. The output is the data securely delivered to the server.

[0461] Step 3:

[0462] The server receives operational actions and status information transmitted from the terminal. Specifically, the server receives data via an internet gateway and stores it in secure storage. The input includes all data sent from the terminal. The output is organized data ready for analysis.

[0463] Step 4:

[0464] The server analyzes the received data using a generative AI model. Specifically, the server inputs data into the model to identify the problem and infer the optimal solution. The data sorted in step 3 is used as input. The output derives action commands and recommended solutions.

[0465] Step 5:

[0466] The server sends the generated operation command to the terminal. Specifically, the server re-encrypts the operation command and sends it to the terminal via the communication network. The operation command obtained in step 4 is used as input. The specific command that the terminal should receive is sent as output.

[0467] Step 6:

[0468] The terminal automatically performs actions according to the received operational commands. Specifically, the terminal may restart applications or adjust settings. The input is commands sent from the server. The output is that the information processing device returns to normal operation and the user is notified accordingly.

[0469] Step 7:

[0470] The user confirms the results through visual notifications from the device. Specifically, the device displays the operation results and problem resolution reports on its screen. The execution results obtained in step 6 are used as input. The output provides information that allows the user to confirm that the problem has been resolved.

[0471] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0472] This invention provides a system that assists in troubleshooting smart devices by combining an emotion engine that recognizes the user's emotions. This system has the function of generating commands for problem solving based on the user's operation information and terminal status information, and further adjusting those commands according to the user's emotional state.

[0473] Users use a dedicated support app to report problems when they occur. The device's built-in emotion engine analyzes the user's facial expressions and voice tone to obtain real-time emotional information. This information plays a crucial role in the troubleshooting process.

[0474] The emotional information acquired by the device, along with operation and state information, is sent to the server. The server analyzes this information and uses an AI model to generate optimal operation commands. Furthermore, based on the emotional information, it is possible to adjust the operation commands, for example, by suggesting more careful and step-by-step procedures if the user is experiencing stress, even for the same solution.

[0475] For example, if a user accidentally deletes an application and feels anxious or worried, the emotion engine recognizes this emotion, and the server generates instructions that, in addition to the usual reinstallation procedure, display guidance messages to reassure the user. This allows the user to confidently follow the instructions and resolve the problem efficiently.

[0476] This system enables flexible and personalized support that takes into account the user's psychological state, providing a supportive environment that is particularly user-friendly for the elderly and those who have anxieties about technology.

[0477] The following describes the processing flow.

[0478] Step 1:

[0479] The user launches the support app on their smart device and checks the problem they are experiencing. The troubleshooting process begins when the user selects the problem within the app or reports it by voice.

[0480] Step 2:

[0481] The device collects user operation information and device status information (e.g., OS version and app installation status). Simultaneously, an emotion engine analyzes emotional information in real time from the user's facial expressions and tone of voice, and this is also collected.

[0482] Step 3:

[0483] The device transmits collected operation information, status information, and emotion information to the server. This data is encrypted and securely transmitted over the communication network.

[0484] Step 4:

[0485] The server uses an AI model to analyze the problem based on the information it receives and creates optimal instructions that take into account the user's emotional state. For example, if the emotion engine determines that the user is feeling stressed, the server will include a reassuring message in the instructions.

[0486] Step 5:

[0487] The server sends the generated operation command to the user's terminal. This command includes specific steps for resolving the problem, and, if necessary, advice to support the user.

[0488] Step 6:

[0489] The device will follow the received operation commands and perform the necessary steps to resolve the problem. During the process, interactive messages will appear on the device screen if user confirmation or consent is required. This makes it easier for the user to understand the procedure and proceed with confidence.

[0490] Step 7:

[0491] The terminal reports to the server that the operation is complete and also notifies the user that the problem has been resolved. If necessary, it provides the user with the option to request further support.

[0492] (Example 2)

[0493] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0494] While there is a need to provide flexible and personalized support that takes into account the user's emotional state when they encounter technical problems while using smart devices, conventional systems have been unable to troubleshoot problems that reflect the user's psychological state. As a result, it has been difficult to provide appropriate and reassuring support, especially for elderly users and those who are technologically insecure.

[0495] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0496] In this invention, the server includes means for inputting user operation information, means for transmitting the input operation information and device status information via a communication network, emotion recognition means for analyzing the user's emotional state based on the transmitted information and generating emotion information, processing means for analyzing the received operation information, device status information and emotion information, generating device operation commands using a generation AI model and adjusting the commands according to the user's emotional state, and means for transmitting the generated operation commands to the device, controlling the device's state, and providing the commands to the user. This enables flexible and effective problem solving that takes into account the user's psychological state.

[0497] "User operation information" refers to input data generated when a user operates a smart device.

[0498] "Device status information" refers to data that shows the current operating status and settings of a smart device.

[0499] A "communication network" is a network infrastructure used to send and receive data from a terminal to a server, or vice versa.

[0500] "Emotion recognition means" refers to technology that analyzes a user's facial expressions and voice data to determine the user's psychological state.

[0501] A "generative AI model" is a framework of artificial intelligence that includes algorithms that derive optimal solutions and instructions based on data.

[0502] "Processing means" refers to a machine or program that analyzes received data, generates necessary commands, and outputs them.

[0503] "Means for controlling the state of the device" refers to technology that adjusts and manages the operation of a smart device based on generated operation commands.

[0504] This invention relates to a smart device troubleshooting system that takes user emotions into consideration. The system is initiated when a user reports a problem using a dedicated support app. The user inputs the specific problem or situation that occurred on the device into the app. The app utilizes the built-in camera and microphone to capture the user's facial expressions and voice tone, and analyzes the user's emotions using emotion recognition means.

[0505] The terminal transmits acquired user operation information, device status information, and emotional information to the server via the communication network. The server uses a generative AI model to analyze the received information and generate optimal operation commands. In doing so, the server takes the user's emotional information into consideration and adjusts the commands accordingly. For example, if the user is feeling stressed, the command will be modified to provide more detailed guidance.

[0506] As a concrete example, consider a situation where a user has accidentally deleted an application and is feeling anxious. In this case, the server can send a reassuring guidance message in addition to the usual reinstallation procedure. The system is designed to provide reassurance to the user by adding a message such as, "The app will be available again soon. Please relax and wait."

[0507] For example, one prompt to input into a generative AI model might be, "Generate steps to take if a user has accidentally deleted an app and is panicking." By generating emotionally sensitive instructions in this way, users can solve problems more efficiently.

[0508] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0509] Step 1:

[0510] When a user experiences a problem with their smart device, they launch a dedicated support app and enter details about the problem and how it occurred. The information entered includes operational details such as the nature of the problem and a description of the circumstances at the time it occurred. Based on this information, the app generates input data to understand the overview of the problem.

[0511] Step 2:

[0512] The device uses its built-in camera and microphone to acquire user facial expression data and voice data. This collects data that reflects the user's emotional state. The device inputs this data into an emotion recognition system and acquires emotional information in real time. As a result of the analysis, emotional information indicating the user's psychological state (e.g., anxiety, worry) is output.

[0513] Step 3:

[0514] The terminal transmits acquired user operation information, device status information, and emotional information to the server via the communication network. This data integration process creates an information package containing the user's operation history, emotional state, and device status. The server uses the received data as the basic input for analysis.

[0515] Step 4:

[0516] The server generates optimal operation commands using a generative AI model based on received operation information, device status information, and emotion information. This process involves data calculations to determine appropriate troubleshooting steps based on data analysis. The output is a set of basic commands for problem solving.

[0517] Step 5:

[0518] The server takes the user's emotional information into account and adjusts the generated instructions accordingly. Specifically, if the user is experiencing stress, the instructions are modified to include more polite and easy-to-understand steps. This emotion-based adjustment process adds reassuring guidance messages. As output, emotion-sensitive instructions are generated.

[0519] Step 6:

[0520] The device provides the user with pre-configured instructions sent from the server. These instructions, displayed on the screen, include details about what the user should do next. For example, a message like, "Please reinstall the app. Please relax and wait," might be displayed as a specific instruction. This allows the user to follow the instructions with reduced stress.

[0521] (Application Example 2)

[0522] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0523] In recent years, smart devices have become more diverse, and the age range and technical proficiency of users have also broadened. As a result, users are increasingly experiencing stress when operating devices or troubleshooting problems. Furthermore, traditional systems tend to provide uniform responses without considering user emotions, which can amplify anxiety and confusion. This poses a particular challenge for elderly and technologically unfamiliar users, as it increases psychological burden and makes efficient problem-solving more difficult.

[0524] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0525] In this invention, the server includes means for acquiring user operation information, means for collecting device status information, and means for acquiring user emotion information using an emotion recognition module. This enables flexible and personalized troubleshooting tailored to the user's emotional state.

[0526] "User operation information" refers to log data of inputs and operations performed by the user on the device.

[0527] "Device status information" refers to data related to the internal operation of the device itself, such as its operating status and error status.

[0528] "Means of transmission via a communication network" refers to systems and protocols for transmitting information to external servers via the internet or other networks.

[0529] An "emotion recognition module" is software or hardware that analyzes a user's facial expressions, tone of voice, etc., to estimate their emotions at that time.

[0530] "Processing means" refers to the software or hardware component that analyzes the collected data and generates commands or feedback based on that analysis.

[0531] "Generating operation commands" refers to the act of creating specific instructions based on acquired data to determine the operation of the device and how to respond to the user.

[0532] A "speech synthesis device" is a device that converts text data into speech and outputs it as speech through a speaker or similar device.

[0533] "Engaging in dialogue" refers to the process by which a user and a machine respond to the user's instructions and questions.

[0534] The system for implementing this invention utilizes operation information obtained from the device's user interface and state information obtained from the device's internal sensors. Furthermore, it collects user emotion information using peripheral devices such as cameras and microphones equipped with emotion recognition modules.

[0535] The server receives this information via the communication network and uses an AI model for data analysis. The analyzed data is used to generate optimal operating instructions to control the device's state. Based on emotional information, if the server determines that the user is experiencing stress, it provides the user with more careful and step-by-step instructions. This process utilizes AI development frameworks such as TensorFlow and PyTorch.

[0536] The terminal performs actions such as changing the screen display and engaging in emotionally sensitive dialogue using a speech synthesis device, based on operation commands sent from the server. Specifically, it can utilize speech synthesis technology to provide navigation messages to the user in a soft tone.

[0537] For example, if a user appears confused while using the device, the device can emit a voice message saying, "You seem to be having trouble, is there anything I can do to help?" Also, if a user accidentally deletes an application, the server will guide them through the reinstallation process and display a reassuring message on the screen.

[0538] An example of a prompt statement is, "Provide guidance on how to generate reassuring messages when the user appears anxious." This allows the program to improve the user experience and provide flexible support tailored to the specific application.

[0539] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0540] Step 1:

[0541] The user operates the device, generating operational information in the process. This operational information includes log data such as touch input, button presses, and application launches and shutdowns. The device's internal sensors acquire device status information, including battery level and memory usage. The input consists of operational and status information, and the output is a dataset of this information.

[0542] Step 2:

[0543] The device uses an emotion recognition module to detect the user's facial expressions and voice tone in real time through the camera and microphone. The input obtained here is the user's image and audio data, and emotional information is output based on this. By analyzing the data using an emotion recognition algorithm, the user's emotional state is understood.

[0544] Step 3:

[0545] The terminal transmits acquired operation information, status information, and emotion information to the server via the communication network. The input is this information, and the output is the appropriate data transfer to the server. The transmitted information becomes an integrated dataset on the server.

[0546] Step 4:

[0547] The server receives an integrated dataset and analyzes it using a generative AI model. The input is the received dataset, and the output is the analysis result. Through data analysis, it generates optimal operational commands. Furthermore, it adjusts the command content based on emotional information to create commands that are appropriate for the user's psychological state.

[0548] Step 5:

[0549] The server sends the generated operation commands to the terminal. The input is the operation commands constructed by the server, and the output is the transmission of commands to the terminal. Based on the transmitted operation commands, the terminal adjusts the user's environment.

[0550] Step 6:

[0551] The terminal executes received operation commands and provides feedback to the user. The input is the received operation command, and the output is the changed state of the device and feedback to the user. This includes specific actions such as changing the screen display and providing user-friendly voice guidance using a speech synthesizer. This allows the user to use the device with confidence.

[0552] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0553] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0554] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[0555] [Fourth Embodiment]

[0556] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0557] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0558] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0559] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[0560] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0561] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0562] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0563] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0564] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0565] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0566] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0567] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0568] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0569] This invention provides a system for quickly resolving various problems that occur on a user's terminal. This system transmits user operation information and device status information to a server via a communication network, and an AI agent analyzes the problem and generates a solution, thereby executing remote operation commands for the terminal.

[0570] Users use a device with a dedicated support app installed. If a problem occurs, the user launches the app and reports the issue. The device automatically collects this information along with current status information and sends it to the server.

[0571] The server analyzes the received information and uses a trained AI model to derive the optimal solution. For example, if an application is accidentally deleted, the server generates an instruction to reinstall the application. This instruction is immediately sent to the user's device.

[0572] The terminal receives operation commands from the server and automatically executes the commanded procedure. If further confirmation from the user is required during this process, a confirmation message will be displayed on the terminal's screen. If the command is successful, the terminal will notify the user of the operation result and report that the problem has been resolved.

[0573] This system can handle a wide range of issues, from basic to complex, and is particularly effective for users who are unfamiliar with technical operations, such as the elderly. For example, even if multiple apps are crashing for unknown reasons, the server can send commands to optimize the settings of each app, quickly restoring them to a normal state. It can also resolve power supply problems and network connection issues by sending appropriate commands, minimizing user intervention.

[0574] The following describes the processing flow.

[0575] Step 1:

[0576] Users launch a dedicated support app on their device and report the problem they are experiencing by selecting the issue or by voice input. This inputs the user's operation information into the device.

[0577] Step 2:

[0578] The terminal collects operational information related to the reported problem and current terminal status information (OS version, installed applications, network status, etc.) and sends this information to the server.

[0579] Step 3:

[0580] The server analyzes the operation and status information received from the terminal. Using the accumulated database and trained AI models, it identifies the cause of the problem and generates the optimal solution.

[0581] Step 4:

[0582] Based on the generated data, the server generates specific operation commands for the terminal and transmits them to the terminal via the communication network.

[0583] Step 5:

[0584] The terminal interprets the operation commands received from the server and automatically performs the actions in accordance with the instructions. If user confirmation is required, a confirmation message will be displayed on the terminal screen, and the terminal will proceed to the next step only after obtaining the user's consent.

[0585] Step 6:

[0586] The terminal reports the results of the operations performed to the server and simultaneously notifies the user of the results. The user can then confirm that the problem has been successfully resolved and request additional support if necessary.

[0587] (Example 1)

[0588] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0589] Modern information terminals possess diverse functions, allowing users to perform a wide range of operations, but this also increases the likelihood of problems occurring. In particular, there is a lack of means to quickly and appropriately resolve problems when users unfamiliar with technology encounter them. Furthermore, providing support remotely is difficult, as it is challenging to accurately understand the user's terminal status and provide efficient instructions. Therefore, there is a need for methods to reduce the burden on users and enable appropriate troubleshooting.

[0590] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0591] In this invention, the server includes means for analyzing received operation information and status information, and using a generated AI model to identify problems and derive solutions; means for transmitting the generated operation commands to a device, which then automatically executes the operation commands, and displays a confirmation message if user confirmation is required; and means for inputting user operation information. This makes it possible to quickly and appropriately resolve technical problems faced by the user and minimize the user's effort.

[0592] "User operation information" refers to information about the operations and inputs that a user performs on their device.

[0593] "Device status information" refers to information regarding the operating status of the terminal's hardware and software.

[0594] A "communication network" refers to the network infrastructure used for sending and receiving data.

[0595] A "generative AI model" refers to an artificial intelligence model that learns from past data and examples to analyze specific problems and provide solutions.

[0596] An "operation command" refers to a command given to a device in order to perform a specific operation.

[0597] "Encryption technology" refers to the technology used to encode data in order to prevent unauthorized access or tampering during data transmission and reception.

[0598] A "prompt message" refers to instructions or messages that provide information to the user or the system.

[0599] This system is an integrated platform for quickly resolving various technical problems that occur on the user's device. Users install a dedicated support app on their device and report problems through this app when they arise. The device collects user-entered operation information and device status information and transmits it to the server via a communication network.

[0600] The server receives and analyzes transmitted information using a high-performance computing infrastructure. This analysis utilizes a generative AI model, which, having learned from a large amount of trouble data and solution examples, can identify problems with high accuracy and derive optimal solutions. For example, if common problems such as accidental application deletion or configuration errors are detected, the server quickly formulates a solution and generates corresponding operational commands.

[0601] The generated operation commands are transmitted to the user's terminal via the communication network. The terminal receives these commands and automatically performs the necessary operations based on them. Specifically, these operations include reinstalling applications, modifying configuration files, and restarting the system. If user confirmation is required during this process, a confirmation message is displayed through the user interface on the terminal to request the user's consent. After the work is completed, the terminal reports to the user that the problem has been resolved.

[0602] As a concrete example of this system, in cases where multiple applications are experiencing unexplained crashes, the server can create commands to optimize the settings of each application, quickly restoring them to a normal state. Furthermore, it can resolve power supply issues and network connection problems through appropriate commands.

[0603] A good example of a prompt would be, "A specific app on my device crashes frequently. Please tell me the possible causes and solutions." This requires concise and clear instructions that describe the user's specific problem. This allows the generative AI model to provide more accurate solutions.

[0604] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0605] Step 1:

[0606] Users report problems using a dedicated support app installed on their device. During this process, users input specific problems as text through the app's interface. The device also automatically collects status information, such as currently running apps and network connectivity. This input information is combined with user operation data and device status information.

[0607] Step 2:

[0608] The terminal transmits user operation information and device status information to the server via the communication network, protecting the data using encryption technology. During this process, data encryption is performed to ensure that the input data reaches the server securely without security risks. The output consists of encrypted user operation information and device status information.

[0609] Step 3:

[0610] The server receives encrypted data, decrypts it to obtain the original user operation information and device status information. The server then analyzes this information using a generative AI model. Specifically, it performs pattern matching and predictive calculations with a database based on the input data to identify the root cause of the problem. As a result of this process, it outputs the optimal solution for resolving the problem.

[0611] Step 4:

[0612] The server generates specific operational commands based on the solutions derived by the generated AI model. These commands include specific processes and changes to be performed on the terminal. These commands are then transmitted to the user's terminal via the communication network. The output is operational commands in an executable format.

[0613] Step 5:

[0614] The terminal analyzes the operation commands received from the server and automatically executes the necessary steps based on them. Following the commands, the terminal might, for example, reinstall a specific application or change system settings. If user confirmation is required during this process, the terminal displays a confirmation message through the user interface. The output is the result of the performed operation.

[0615] Step 6:

[0616] After all operations are completed, the device notifies the user of the results of the processes performed. This notification includes confirmation that the problem has been resolved and details of the specific processes that were carried out. This allows the user to continue using the device with peace of mind. The output is the notification of the processing results to the user.

[0617] (Application Example 1)

[0618] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0619] Technical problems that occur during the use of information processing devices are a major source of frustration, especially for users unfamiliar with technology. Such problems cause significant inconvenience to users and can lead to inaccurate operations and misunderstandings. In particular, in the context of electronic payments, such malfunctions can result in direct economic losses and a diminished user experience, making these issues urgently resolvable.

[0620] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0621] In this invention, the server includes means for inputting user actions, means for transmitting the input actions and status information of the information processing device via a communication network, and processing means for analyzing the received actions and status information of the information processing device and generating operation commands for the information processing device based on digital data generated using a generation AI model. This makes it possible to quickly and automatically derive solutions when technical problems occur and to maintain the normal operation of the information processing device.

[0622] "User actions" refer to a series of operations and inputs performed by a user during the process of using an information processing device.

[0623] "Status information of an information processing device" refers to data that represents the operating status and settings of an information processing device, and is used for system diagnosis and analysis.

[0624] A "communication network" refers to the network infrastructure used to send and receive data, and includes the internet and dedicated lines.

[0625] A "generative AI model" is a model trained using artificial intelligence technology for a specific purpose, and it analyzes problems and derives solutions based on the data it receives.

[0626] "Digital data" refers to information that is represented electronically and can be used in various forms such as numbers and strings of characters.

[0627] An "operation command for an information processing device" refers to an instruction generated to cause an information processing device to perform a specific operation.

[0628] "Adjusting the visual display" refers to the act of changing or optimizing the content displayed on the screen of an information processing device.

[0629] This invention provides a system for automatically and quickly solving technical problems in information processing equipment. The system is connected to a user's terminal via a communication network and operates in cooperation with a secure server.

[0630] The server receives user actions and information about the state of the information processing device, and uses a generated AI model based on this information to analyze the problem. Suitable AI platforms for this model include commonly used platforms such as TensorFlow and PyTorch. The server analyzes the received digital data, generates optimal action commands, and sends them to the information processing device.

[0631] When a terminal receives an operational command from the server, it automatically responds based on the instruction. This process may include adjusting the visual display. The user receives a notification on the terminal screen about how the problem was addressed. In particular, in the context of electronic payments, the quick and effective handling of problems ensures that the user's operation is completed successfully.

[0632] Server-based analysis requires referencing data from similar past problems. This method allows for more accurate identification of the cause and solution to errors. For example, if the app crashes while paying at a cafe, the server will prompt the user to restart the problematic module and check the network status, attempting to reconnect automatically.

[0633] An example of a prompt message is: "The app crashed while you were paying at the cafe. Please check your current network status and version information, and attempt to automatically reconnect and fix the module."

[0634] This system enables users of information processing equipment to solve problems efficiently, regardless of their technical skills.

[0635] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0636] Step 1:

[0637] The terminal collects user interaction information and status information from the information processing device. Specifically, the terminal acquires and records data from sensors and input devices. Inputs include user touch operations and application status. As output, this information is prepared for transmission to the server.

[0638] Step 2:

[0639] The terminal transmits collected operational behavior and status information to the server via the communication network. Specifically, the terminal encrypts the data before sending it to the server using the network protocol. The input is the data collected in step 1. The output is the data securely delivered to the server.

[0640] Step 3:

[0641] The server receives operational actions and status information transmitted from the terminal. Specifically, the server receives data via an internet gateway and stores it in secure storage. The input includes all data sent from the terminal. The output is organized data ready for analysis.

[0642] Step 4:

[0643] The server analyzes the received data using a generative AI model. Specifically, the server inputs data into the model to identify the problem and infer the optimal solution. The data sorted in step 3 is used as input. The output derives action commands and recommended solutions.

[0644] Step 5:

[0645] The server sends the generated operation command to the terminal. Specifically, the server re-encrypts the operation command and sends it to the terminal via the communication network. The operation command obtained in step 4 is used as input. The specific command that the terminal should receive is sent as output.

[0646] Step 6:

[0647] The terminal automatically performs actions according to the received operational commands. Specifically, the terminal may restart applications or adjust settings. The input is commands sent from the server. The output is that the information processing device returns to normal operation and the user is notified accordingly.

[0648] Step 7:

[0649] The user confirms the results through visual notifications from the device. Specifically, the device displays the operation results and problem resolution reports on its screen. The execution results obtained in step 6 are used as input. The output provides information that allows the user to confirm that the problem has been resolved.

[0650] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0651] This invention provides a system that assists in troubleshooting smart devices by combining an emotion engine that recognizes the user's emotions. This system has the function of generating commands for problem solving based on the user's operation information and terminal status information, and further adjusting those commands according to the user's emotional state.

[0652] Users use a dedicated support app to report problems when they occur. The device's built-in emotion engine analyzes the user's facial expressions and voice tone to obtain real-time emotional information. This information plays a crucial role in the troubleshooting process.

[0653] The emotional information acquired by the device, along with operation and state information, is sent to the server. The server analyzes this information and uses an AI model to generate optimal operation commands. Furthermore, based on the emotional information, it is possible to adjust the operation commands, for example, by suggesting more careful and step-by-step procedures if the user is experiencing stress, even for the same solution.

[0654] For example, if a user accidentally deletes an application and feels anxious or worried, the emotion engine recognizes this emotion, and the server generates instructions that, in addition to the usual reinstallation procedure, display guidance messages to reassure the user. This allows the user to confidently follow the instructions and resolve the problem efficiently.

[0655] This system enables flexible and personalized support that takes into account the user's psychological state, providing a supportive environment that is particularly user-friendly for the elderly and those who have anxieties about technology.

[0656] The following describes the processing flow.

[0657] Step 1:

[0658] The user launches the support app on their smart device and checks the problem they are experiencing. The troubleshooting process begins when the user selects the problem within the app or reports it by voice.

[0659] Step 2:

[0660] The device collects user operation information and device status information (e.g., OS version and app installation status). Simultaneously, an emotion engine analyzes emotional information in real time from the user's facial expressions and tone of voice, and this is also collected.

[0661] Step 3:

[0662] The device transmits collected operation information, status information, and emotion information to the server. This data is encrypted and securely transmitted over the communication network.

[0663] Step 4:

[0664] The server uses an AI model to analyze the problem based on the information it receives and creates optimal instructions that take into account the user's emotional state. For example, if the emotion engine determines that the user is feeling stressed, the server will include a reassuring message in the instructions.

[0665] Step 5:

[0666] The server sends the generated operation command to the user's terminal. This command includes specific steps for resolving the problem, and, if necessary, advice to support the user.

[0667] Step 6:

[0668] The device will follow the received operation commands and perform the necessary steps to resolve the problem. During the process, interactive messages will appear on the device screen if user confirmation or consent is required. This makes it easier for the user to understand the procedure and proceed with confidence.

[0669] Step 7:

[0670] The terminal reports to the server that the operation is complete and also notifies the user that the problem has been resolved. If necessary, it provides the user with the option to request further support.

[0671] (Example 2)

[0672] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0673] While there is a need to provide flexible and personalized support that takes into account the user's emotional state when they encounter technical problems while using smart devices, conventional systems have been unable to troubleshoot problems that reflect the user's psychological state. As a result, it has been difficult to provide appropriate and reassuring support, especially for elderly users and those who are technologically insecure.

[0674] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0675] In this invention, the server includes means for inputting user operation information, means for transmitting the input operation information and device status information via a communication network, emotion recognition means for analyzing the user's emotional state based on the transmitted information and generating emotion information, processing means for analyzing the received operation information, device status information and emotion information, generating device operation commands using a generation AI model and adjusting the commands according to the user's emotional state, and means for transmitting the generated operation commands to the device, controlling the device's state, and providing the commands to the user. This enables flexible and effective problem solving that takes into account the user's psychological state.

[0676] "User operation information" refers to input data generated when a user operates a smart device.

[0677] "Device status information" refers to data that shows the current operating status and settings of a smart device.

[0678] A "communication network" is a network infrastructure used to send and receive data from a terminal to a server, or vice versa.

[0679] "Emotion recognition means" refers to technology that analyzes a user's facial expressions and voice data to determine the user's psychological state.

[0680] A "generative AI model" is a framework of artificial intelligence that includes algorithms that derive optimal solutions and instructions based on data.

[0681] "Processing means" refers to a machine or program that analyzes received data, generates necessary commands, and outputs them.

[0682] "Means for controlling the state of the device" refers to technology that adjusts and manages the operation of a smart device based on generated operation commands.

[0683] This invention relates to a smart device troubleshooting system that takes user emotions into consideration. The system is initiated when a user reports a problem using a dedicated support app. The user inputs the specific problem or situation that occurred on the device into the app. The app utilizes the built-in camera and microphone to capture the user's facial expressions and voice tone, and analyzes the user's emotions using emotion recognition means.

[0684] The terminal transmits acquired user operation information, device status information, and emotional information to the server via the communication network. The server uses a generative AI model to analyze the received information and generate optimal operation commands. In doing so, the server takes the user's emotional information into consideration and adjusts the commands accordingly. For example, if the user is feeling stressed, the command will be modified to provide more detailed guidance.

[0685] As a concrete example, consider a situation where a user has accidentally deleted an application and is feeling anxious. In this case, the server can send a reassuring guidance message in addition to the usual reinstallation procedure. The system is designed to provide reassurance to the user by adding a message such as, "The app will be available again soon. Please relax and wait."

[0686] For example, one prompt to input into a generative AI model might be, "Generate steps to take if a user has accidentally deleted an app and is panicking." By generating emotionally sensitive instructions in this way, users can solve problems more efficiently.

[0687] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0688] Step 1:

[0689] When a user experiences a problem with their smart device, they launch a dedicated support app and enter details about the problem and how it occurred. The information entered includes operational details such as the nature of the problem and a description of the circumstances at the time it occurred. Based on this information, the app generates input data to understand the overview of the problem.

[0690] Step 2:

[0691] The device uses its built-in camera and microphone to acquire user facial expression data and voice data. This collects data that reflects the user's emotional state. The device inputs this data into an emotion recognition system and acquires emotional information in real time. As a result of the analysis, emotional information indicating the user's psychological state (e.g., anxiety, worry) is output.

[0692] Step 3:

[0693] The terminal transmits acquired user operation information, device status information, and emotional information to the server via the communication network. This data integration process creates an information package containing the user's operation history, emotional state, and device status. The server uses the received data as the basic input for analysis.

[0694] Step 4:

[0695] The server generates optimal operation commands using a generative AI model based on received operation information, device status information, and emotion information. This process involves data calculations to determine appropriate troubleshooting steps based on data analysis. The output is a set of basic commands for problem solving.

[0696] Step 5:

[0697] The server takes the user's emotional information into account and adjusts the generated instructions accordingly. Specifically, if the user is experiencing stress, the instructions are modified to include more polite and easy-to-understand steps. This emotion-based adjustment process adds reassuring guidance messages. As output, emotion-sensitive instructions are generated.

[0698] Step 6:

[0699] The device provides the user with pre-configured instructions sent from the server. These instructions, displayed on the screen, include details about what the user should do next. For example, a message like, "Please reinstall the app. Please relax and wait," might be displayed as a specific instruction. This allows the user to follow the instructions with reduced stress.

[0700] (Application Example 2)

[0701] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0702] In recent years, smart devices have become more diverse, and the age range and technical proficiency of users have also broadened. As a result, users are increasingly experiencing stress when operating devices or troubleshooting problems. Furthermore, traditional systems tend to provide uniform responses without considering user emotions, which can amplify anxiety and confusion. This poses a particular challenge for elderly and technologically unfamiliar users, as it increases psychological burden and makes efficient problem-solving more difficult.

[0703] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0704] In this invention, the server includes means for acquiring user operation information, means for collecting device status information, and means for acquiring user emotion information using an emotion recognition module. This enables flexible and personalized troubleshooting tailored to the user's emotional state.

[0705] "User operation information" refers to log data of inputs and operations performed by the user on the device.

[0706] "Device status information" refers to data related to the internal operation of the device itself, such as its operating status and error status.

[0707] "Means of transmission via a communication network" refers to systems and protocols for transmitting information to external servers via the internet or other networks.

[0708] An "emotion recognition module" is software or hardware that analyzes a user's facial expressions, tone of voice, etc., to estimate their emotions at that time.

[0709] "Processing means" refers to the software or hardware component that analyzes the collected data and generates commands or feedback based on that analysis.

[0710] "Generating operation commands" refers to the act of creating specific instructions based on acquired data to determine the operation of the device and how to respond to the user.

[0711] A "speech synthesis device" is a device that converts text data into speech and outputs it as speech through a speaker or similar device.

[0712] "Engaging in dialogue" refers to the process by which a user and a machine respond to the user's instructions and questions.

[0713] The system for implementing this invention utilizes operation information obtained from the device's user interface and state information obtained from the device's internal sensors. Furthermore, it collects user emotion information using peripheral devices such as cameras and microphones equipped with emotion recognition modules.

[0714] The server receives this information via the communication network and uses an AI model for data analysis. The analyzed data is used to generate optimal operating instructions to control the device's state. Based on emotional information, if the server determines that the user is experiencing stress, it provides the user with more careful and step-by-step instructions. This process utilizes AI development frameworks such as TensorFlow and PyTorch.

[0715] The terminal performs actions such as changing the screen display and engaging in emotionally sensitive dialogue using a speech synthesis device, based on operation commands sent from the server. Specifically, it can utilize speech synthesis technology to provide navigation messages to the user in a soft tone.

[0716] For example, if a user appears confused while using the device, the device can emit a voice message saying, "You seem to be having trouble, is there anything I can do to help?" Also, if a user accidentally deletes an application, the server will guide them through the reinstallation process and display a reassuring message on the screen.

[0717] An example of a prompt statement is, "Provide guidance on how to generate reassuring messages when the user appears anxious." This allows the program to improve the user experience and provide flexible support tailored to the specific application.

[0718] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0719] Step 1:

[0720] The user operates the device, generating operational information in the process. This operational information includes log data such as touch input, button presses, and application launches and shutdowns. The device's internal sensors acquire device status information, including battery level and memory usage. The input consists of operational and status information, and the output is a dataset of this information.

[0721] Step 2:

[0722] The device uses an emotion recognition module to detect the user's facial expressions and voice tone in real time through the camera and microphone. The input obtained here is the user's image and audio data, and emotional information is output based on this. By analyzing the data using an emotion recognition algorithm, the user's emotional state is understood.

[0723] Step 3:

[0724] The terminal transmits acquired operation information, status information, and emotion information to the server via the communication network. The input is this information, and the output is the appropriate data transfer to the server. The transmitted information becomes an integrated dataset on the server.

[0725] Step 4:

[0726] The server receives an integrated dataset and analyzes it using a generative AI model. The input is the received dataset, and the output is the analysis result. Through data analysis, it generates optimal operational commands. Furthermore, it adjusts the command content based on emotional information to create commands that are appropriate for the user's psychological state.

[0727] Step 5:

[0728] The server sends the generated operation commands to the terminal. The input is the operation commands constructed by the server, and the output is the transmission of commands to the terminal. Based on the transmitted operation commands, the terminal adjusts the user's environment.

[0729] Step 6:

[0730] The terminal executes received operation commands and provides feedback to the user. The input is the received operation command, and the output is the changed state of the device and feedback to the user. This includes specific actions such as changing the screen display and providing user-friendly voice guidance using a speech synthesizer. This allows the user to use the device with confidence.

[0731] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0732] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0733] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[0734] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0735] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[0736] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[0737] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[0738] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[0739] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[0740] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[0741] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[0742] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[0743] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0744] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0745] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0746] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[0747] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0748] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0749] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0750] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0751] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[0752] The following is further disclosed regarding the embodiments described above.

[0753] (Claim 1)

[0754] A means of inputting user operation information,

[0755] Means for transmitting input operation information and device status information via a communication network,

[0756] A processing means that analyzes received operation information and device status information, and generates device operation commands based on the generated data,

[0757] A means for transmitting the generated operation commands to the device and controlling the state of the device,

[0758] A system that includes this.

[0759] (Claim 2)

[0760] The system according to claim 1, characterized in that the operation command includes an instruction to reinstall the application.

[0761] (Claim 3)

[0762] The system according to claim 1, characterized in that the device changes the screen display based on the received operation command.

[0763] "Example 1"

[0764] (Claim 1)

[0765] A means of inputting user operation information,

[0766] Means for transmitting input operation information and device status information via a communication network,

[0767] A processing means that analyzes received operation information and state information, and uses a generated AI model to identify problems and derive solutions,

[0768] A means for transmitting generated operation commands to a device, for the device to automatically execute the operation commands, and for displaying a confirmation message if user confirmation is required.

[0769] A system that includes this.

[0770] (Claim 2)

[0771] The system according to claim 1, characterized in that the operation command executes an instruction including the reinstallation of the application.

[0772] (Claim 3)

[0773] The system according to claim 1, characterized in that the device changes the screen display and provides notifications based on the received operation command.

[0774] "Application Example 1"

[0775] (Claim 1)

[0776] A means of inputting user actions,

[0777] Means for transmitting input operation actions and status information of the information processing device via a communication network,

[0778] A processing means that analyzes the received operation actions and state information of the information processing device, and generates operation commands for the information processing device based on the digital data generated using a generation AI model,

[0779] A means for transmitting the generated operation command to an information processing device and controlling the state of the information processing device,

[0780] A system that includes this.

[0781] (Claim 2)

[0782] The system according to claim 1, characterized in that the operation command includes an instruction to reinstall the digital program.

[0783] (Claim 3)

[0784] The system according to claim 1, characterized in that the information processing device adjusts the visual display based on the received operation command.

[0785] "Example 2 of combining an emotion engine"

[0786] (Claim 1)

[0787] A means of inputting user operation information,

[0788] Means for transmitting input operation information and device status information via a communication network,

[0789] An emotion recognition means that analyzes the user's emotional state based on transmitted information and generates emotional information,

[0790] A processing means that analyzes received operation information, device status information, and emotional information, generates device operation commands using a generated AI model, and adjusts the commands according to the user's emotional state,

[0791] A means for transmitting generated operation commands to the device, controlling the device's state, and providing commands to the user,

[0792] A system that includes this.

[0793] (Claim 2)

[0794] The system according to claim 1, characterized in that the operation command includes a command to reinstall the application, as well as a guidance message to reassure the user.

[0795] (Claim 3)

[0796] The system according to claim 1, characterized in that the device adjusts the display content based on received operation commands and emotional information, and changes the screen display so that the user does not feel stressed.

[0797] "Application example 2 when combining with an emotional engine"

[0798] (Claim 1)

[0799] A means of inputting user operation information,

[0800] Means for transmitting input operation information and device status information via a communication network,

[0801] A means of acquiring user emotional information using an emotion recognition module installed in the device,

[0802] A processing means that analyzes received operation information, device status information, and emotion information, and generates operation commands for the device based on the generated data,

[0803] A means for transmitting generated operation commands to the device and controlling the device's state according to the user's emotional state,

[0804] A system that includes this.

[0805] (Claim 2)

[0806] The system according to claim 1, characterized in that the operation commands include an instruction to reinstall the application, and also generate instructions to enhance the user's sense of security.

[0807] (Claim 3)

[0808] The system according to claim 1, characterized in that the device changes the screen display based on the received operation command and further engages in dialogue according to the emotional state using a speech synthesis device. [Explanation of Symbols]

[0809] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. A means of inputting user actions, Means for transmitting input operation actions and status information of the information processing device via a communication network, A processing means that analyzes the received operation actions and state information of the information processing device, and generates operation commands for the information processing device based on the digital data generated using a generation AI model, A means for transmitting the generated operation command to an information processing device and controlling the state of the information processing device, A system that includes this.

2. The system according to claim 1, characterized in that the operation command includes an instruction to reinstall the digital program.

3. The system according to claim 1, characterized in that the information processing device adjusts the visual display based on the received operation command.

Images