System

Abstract

A system is provided.SOLUTION: A system comprising: means for inputting a response result of an apparatus; means for analyzing the response result and confirming a deviation from a target procedure; means for investigating a method for eliminating the deviation and generating a procedure to be executed next; and means for displaying the generated procedure.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Traditionally, work procedure manuals have been created manually, and if an unexpected event occurs during work, the work must be stopped, requiring clarification and readjustment, which requires a lot of time and effort. This reduces work efficiency and wastes time and effort. Furthermore, it is difficult to flexibly change procedures depending on the situation on-site, and they lack the flexibility to deal with a variety of irregular events. To solve these problems and allow work to proceed efficiently and flexibly, there is a need for the development of a system that can flexibly present the next required commands and procedures based on device responses. [Means for solving the problem]

[0005] The present invention provides a system including a means for inputting device response results, a means for analyzing the response results and checking for deviations from a target procedure, a means for investigating ways to resolve the deviations and generating a next procedure to be performed, and a means for displaying the generated procedure. This allows the next procedure to be presented in real time based on the device response results, eliminating the need to interrupt work and review the procedure manual as in the past. Furthermore, by automatically investigating the optimal way to resolve the deviations and providing specific commands and procedures, the system can flexibly respond to a variety of irregular events. Furthermore, this system achieves more efficient work management by creating a cycle in which the generated procedure is displayed to the user and the user re-enters the results of their execution.

[0006] "Equipment" is a general term for machines and equipment designed to perform a specific task or process.

[0007] "Response result" refers to the reaction or output that a device shows in response to a specific input or condition.

[0008] "Means" refers to a method or device used to achieve a particular purpose.

[0009] A "server" is a networked computer system that acts as a central location for storing, processing, and sharing data.

[0010] "Terminal" refers to a computer or device that a user can directly operate to input and output information.

[0011] "User" refers to a person or organization that operates a device or system to perform a specific task or process.

[0012] "Analysis" refers to examining the data and information obtained in detail and clarifying its structure, characteristics, and meaning.

[0013] "Deviation" refers to a deviation or deviation from a goal or standard.

[0014] "Research" refers to the process of collecting and analyzing information about a specific problem or issue.

[0015] A "command" refers to an instruction that instructs a computer or device to perform a specific operation.

[0016] A "procedure" refers to the sequence of operations or actions required to perform a particular task.

[0017] "Display" means showing information visually to a user, often through a screen or monitor. [Brief explanation of the drawings]

[0018] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0019] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0020] First, the terms used in the following description will be explained.

[0021] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0022] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0023] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0024] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0025] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0026] [First embodiment]

[0027] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0028] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0029] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0030] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0031] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0032] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0033] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0034] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0035] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0036] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0037] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0038] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0039] This invention is a system that inputs the response results of a device and generates and presents the next required commands and procedures based on them. Below, the program processing of this system is explained in natural language and in detail with concrete examples.

[0040] Program Processing Overview

[0041] 1. The user inputs the device response results into the server.

[0042] The user inputs the response results obtained from the device into the terminal while working. For example, when the user inputs the reading of a temperature sensor device, the user inputs that the temperature is "85°C" into the terminal.

[0043] The terminal transmits this input data to the server.

[0044] 2. The server analyzes the response

[0045] The server analyzes the response and checks whether it deviates from the set value or the expected normal value. For example, the server detects that the temperature "85°C" exceeds the set value "70°C."

[0046] The server records logs based on the analysis results and stores them for future analysis.

[0047] 3. The server checks for discrepancies

[0048] Based on the analysis results, the server determines whether the response deviates from the target procedure. The server determines that the temperature of 85°C is abnormal because it exceeds the normal range of 70°C.

[0049] Based on the result of this determination, the next step is to determine the necessary action.

[0050] 4. Investigate how the server resolves the discrepancy

[0051] The server's AI module investigates the best way to resolve the discrepancy. If the temperature exceeds the set point, the server's AI determines that activating the cooling system would be effective.

[0052] Validate the findings and generate specific implementation steps.

[0053] 5. The server generates the next command and steps to be executed

[0054] Based on the results of the investigation and verification, the server generates the next command and procedure to be performed, such as "switch on the cooling device" or "restart the device."

[0055] The generated procedure is formatted so that it is easy for the user to understand, and is sent to the terminal.

[0056] 6. Present the procedure for creating the device to the user

[0057] The device will then display the new instructions received from the server to the user, for example, "Please turn on the cooling device."

[0058] After the user has performed the procedure, he is prompted to enter the device's response again.

[0059] Specific examples

[0060] 1. The user inputs the device response results into the server.

[0061] Example: A user enters a temperature sensor reading of "85°C" into a terminal and presses the send button.

[0062] 2. The server analyzes the response

[0063] Example: The server analyzes the input data "85°C" and verifies that the expected normal value is "up to 70°C." The server records this information in a log and determines that there is a discrepancy.

[0064] 3. The server checks for discrepancies

[0065] Example: The server determines that the temperature "85°C" is abnormal because it exceeds the set value "70°C."

[0066] 4. Investigate how the server resolves the discrepancy

[0067] Example: A server's AI investigates whether operating cooling equipment is an effective countermeasure when temperatures exceed the limit, and verifies the procedure for operating the cooling system.

[0068] 5. The server generates the next command and steps to be executed

[0069] Example: The server generates the procedure "Turn on the cooling system and set the internal fan to maximum output."

[0070] 6. Present the procedure for creating the device to the user

[0071] Example: The terminal will prompt the user to "Please turn on the cooling system and set the internal fan to maximum power." After the user performs the steps, they will be prompted to enter the device's response again.

[0072] This system allows users to receive appropriate response procedures in real time based on the device's response results, enabling them to respond quickly, preventing work interruptions and realizing efficient and flexible work.

[0073] The processing flow will be explained below.

[0074] Step 1:

[0075] The user inputs the device response results into the terminal. The user inputs the device response results obtained during work into the terminal's input field and presses the send button. For example, the user inputs the reading from the temperature sensor, "85°C."

[0076] Step 2:

[0077] The terminal transmits the input response result to the server. The terminal converts the data input by the user into a packet format and transmits it to the server.

[0078] Step 3:

[0079] The server analyzes the response it receives. The server passes the received data to the analysis module, which analyzes the contents of the response. For example, the server detects that the temperature "85°C" deviates from the set value "70°C."

[0080] Step 4:

[0081] The server checks for any deviations. Based on the analysis results, the server determines whether there is any deviation from the target procedure. Because the temperature "85°C" exceeds the normal range of "70°C," the server determines that there is an "abnormality."

[0082] Step 5:

[0083] The server investigates ways to resolve the discrepancy. The server's AI module runs and investigates the best way to resolve the discrepancy. For example, it determines that turning on the cooling system is the best option.

[0084] Step 6:

[0085] The server verifies the investigation results. Based on the investigation results, the server verifies whether the generated response method is effective using simulations and past data.

[0086] Step 7:

[0087] The server generates the next command and procedure to be performed. Based on the verified method, the server generates the specific command and procedure to be performed next. For example, it creates a specific procedure such as "turn on the cooling system and set the fan to maximum output."

[0088] Step 8:

[0089] The server transmits the generated procedure to the terminal, converts the generated procedure into a packet format, and transmits it to the terminal.

[0090] Step 9:

[0091] The terminal presents the instructions to the user. The terminal displays the received instructions in an easy-to-read format for the user. For example, it displays "Please turn on the cooling system and set the internal fan to maximum output."

[0092] Step 10:

[0093] The user follows the instructions displayed on the device, powering on the cooling system and setting the fans to maximum power as instructed.

[0094] Step 11:

[0095] The user again inputs the device's response result into the terminal and sends it. The user checks the device status after the procedure is executed, inputs a new response result into the terminal, and sends it again.

[0096] Step 12:

[0097] The device sends a new response to the server. The device converts the received data back into packets and sends them to the server. The cycle begins again, generating and submitting new instructions as needed.

[0098] Example 1

[0099] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0100] Conventionally, the generation and presentation of procedures and commands based on device response results has often been done manually, resulting in problems such as reduced work efficiency and accuracy. Furthermore, manual analysis and response takes time, making it difficult to respond in real time. The objective of this invention is to provide a system that automatically analyzes device response results and quickly and accurately presents the necessary procedures and commands to the user.

[0101] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0102] In this invention, the server includes a means for inputting the response results of the device, a means for analyzing the response results and checking for deviations from the target procedure, and a means for investigating a method for eliminating the deviation using a generative AI model and generating the procedure to be performed next, thereby enabling the user to receive the appropriate response procedure in real time and respond quickly.

[0103] The "means for inputting the response results of the device" is a function that allows the user to input data or information obtained from the device into a terminal or the like and transmit that data to the server.

[0104] "Means of analysis" refers to the process by which the server compares the data it receives with set values ​​and expected normal values ​​to determine any deviations or abnormalities in the data.

[0105] The "means for checking deviation from the target procedure" is a function that determines, based on the analyzed data, how much the actual results deviate from the set target procedure.

[0106] "Means of investigating using a generative AI model and generating the next steps to be taken" refers to the process of using an AI module in the server to investigate the optimal method for resolving any discrepancies or anomalies that have occurred and generate specific response steps.

[0107] The "means for displaying procedures" is a function that visualizes and presents the generated response procedures and commands to the user.

[0108] "Means for inputting to a data management device" refers to an interface or operating procedure for inputting and sending the generated procedures and commands to a data management device such as a server.

[0109] The "means for generating specific operational instructions" is a function that generates specific commands and operational procedures to be executed for a specific device or system based on the analysis results.

[0110] The "means for verifying whether the operation instruction is valid" is a process for checking and confirming whether the generated specific operation instruction actually has an effect.

[0111] The present invention is a system that inputs a device response result and generates and presents the next required command or procedure based on that result. The system includes a means for inputting the device response result, a means for analyzing it, a means for checking for discrepancies, a means for generating the next procedure using a generative AI model, and a means for displaying the procedure.

[0112] Hardware and software used

[0113] Device: The computing device (smartphone, tablet, PC) operated by the user.

[0114] Server: A server that analyzes data, records logs, generates next steps, etc. It is built using a database or cloud infrastructure (e.g., Amazon Web Services or Microsoft Azure).

[0115] Generative AI model: An AI module (e.g., TensorFlow, PyTorch, etc.) used for data analysis and procedure generation.

[0116] Natural language explanation of program processing

[0117] 1. The user enters the device's response into the terminal and sends it to the server.

[0118] The user inputs the response obtained from the device into the terminal. For example, the user inputs "85°C" as the temperature sensor reading.

[0119] The terminal transmits the input data to the server.

[0120] 2. The server receives the response and analyzes the data.

[0121] The server receives the data sent from the device and analyzes it by comparing it with the set value and expected normal value. It checks whether the temperature "85°C" exceeds the set value "70°C".

[0122] The server records the analysis results as a log and detects any deviations.

[0123] 3. The server determines whether there is a discrepancy

[0124] Based on the analysis results, the server determines how much the input data deviates from the set value. Since the temperature "85°C" exceeds the set value "70°C," it is judged to be "abnormal."

[0125] Based on this decision, the server determines the next action to be taken.

[0126] 4. The server uses a generative AI model to investigate and select a method to resolve the discrepancy.

[0127] The server uses a generative AI model to investigate the best way to resolve the anomaly. If the temperature is high, the investigation confirms that turning on a cooling device would be beneficial.

[0128] Generate specific steps based on the survey results.

[0129] 5. The server generates the next steps to be taken and sends them to the device.

[0130] The server then formats the generated instructions into a format that is easy for the user to understand, such as "Please turn on the cooling device."

[0131] This procedure is sent to the terminal.

[0132] 6. Present the procedure for creating the device to the user

[0133] The device displays the instructions received from the server to the user, for example, "Please turn on the cooling device."

[0134] The user acts according to the presented procedure and inputs the response results of the device again.

[0135] Examples of specific examples and prompts

[0136] Specific examples

[0137] 1. The user inputs the device's response results.

[0138] Example: A user enters the temperature sensor reading "85°C" into the terminal input form and presses the "Submit" button.

[0139] 2. The server analyzes the response

[0140] Example: The server receives the data "85°C" and compares it with the set value "70°C" to detect an abnormality.

[0141] 3. The server checks for discrepancies

[0142] Example: The server temperature "85°C" exceeds the set value "70°C", so it is logged as an abnormality.

[0143] 4. Investigate how the server resolves the discrepancy

[0144] Example: Using a generative AI model, determine that running a cooling unit is the optimal solution.

[0145] 5. The server generates the next steps to be taken

[0146] Example: The server generates the procedure "Switch on the cooling device and set the internal fan to maximum power."

[0147] 6. Present the procedure for creating the device to the user

[0148] Example: The device displays to the user, "Please turn on the cooling device."

[0149] Prompt Sentence Examples

[0150] Generate a procedure for what to do if the temperature is too high.

[0151] Generate instructions to set the cooling system to maximum power.

[0152] Please indicate the procedure to be followed in response to an abnormal value of 85°C.

[0153] As described above, the present invention aims to provide the user with an appropriate response procedure in real time based on the response results of the device, thereby realizing a quick response.

[0154] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0155] Step 1:

[0156] The user inputs the device's response into the terminal and sends it to the server.

[0157] Specific operation: The user enters the response obtained from the device, for example, the temperature sensor reading "85°C," into the terminal and presses the send button.

[0158] Input: Temperature sensor reading "85°C".

[0159] Output: Sending data from the device to the server.

[0160] Step 2:

[0161] The server receives the response result

[0162] Specific operation: The server receives data sent from the device via HTTP request or WebSocket.

[0163] Input: Temperature data sent from the device: "85°C".

[0164] Output: The temperature data received by the server.

[0165] Step 3:

[0166] The server analyzes the response

[0167] Specific operation: The server compares the received data with the set value or expected normal value to check for deviations. For example, it checks whether the temperature "85°C" exceeds the set value "70°C".

[0168] Input: Temperature data received by the server is "85°C", set value is "70°C".

[0169] Output: Anomaly detection results.

[0170] Step 4:

[0171] The server determines whether there is a discrepancy

[0172] Specific operation: The server determines how much the temperature deviates from the set value based on the analysis results. Since the temperature "85°C" exceeds the set value "70°C", it is determined to be abnormal.

[0173] Input: Anomaly detection results.

[0174] Output: Log of abnormality detection.

[0175] Step 5:

[0176] The server uses generative AI models to explore solutions

[0177] Specific behavior: The generative AI model in the server investigates the best way to resolve the anomaly, for example, investigating whether running a cooling system is effective.

[0178] Input: Abnormality determination log, abnormal value "85°C".

[0179] Output: The optimal solution and its validation results.

[0180] Step 6:

[0181] The server generates the next steps and sends them to the device.

[0182] Specific operation: The server formats the generated instructions into a format that is easy for the user to understand and sends it to the terminal. For example, it creates and sends an instruction such as "Please turn on the cooling device."

[0183] Input: Validation results of the optimal solution.

[0184] Output: Generates specific operating instructions and sends them to the terminal.

[0185] Step 7:

[0186] Present the procedure for the device to be generated to the user

[0187] Specific behavior: The device displays the instructions sent from the server to the user, such as "Please turn on the cooling device."

[0188] Input: Specific operating instructions received from the server.

[0189] Output: The instructions presented to the user.

[0190] Step 8:

[0191] The user performs the procedure and enters the response again.

[0192] Specific operation: The user follows the presented procedure, for example, turning on the cooling device, and then inputs the device's response again.

[0193] Input: Procedures presented by the server, and the results of the user's execution.

[0194] Output: The response result of the input device again.

[0195] (Application example 1)

[0196] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0197] Picking work at logistics centers requires a lot of manpower and time, so there is a need to improve work efficiency. It is also important to quickly correct errors that occur during work and achieve accurate picking. In response to these challenges, the present invention provides a system that enables picking robots to receive optimal work procedures in real time and perform work efficiently and accurately.

[0198] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0199] In this invention, the server includes means for recalculating the picking procedure for the logistics robot based on the response result of the device and generating a corrected command, means for displaying the generated procedure to the user and inputting the response result to the server, means for presenting an appropriate procedure in real time in the logistics process, means for generating specific picking commands, and means for verifying whether the generated commands are accurate and optimal, thereby enabling improved efficiency and accuracy of picking work in logistics centers.

[0200] "Device response result" refers to the result information that a device returns in response to some input or operation.

[0201] A "procedure" refers to the sequence of tasks or operations that are used to achieve a particular purpose.

[0202] A "server" is a computer system that stores, manages, and processes data over a network.

[0203] A "logistics robot" is a robot that automates the movement and picking of goods at logistics centers and other locations.

[0204] "Real-time" refers to a method in which data input, processing, and response occur within an extremely short period of time.

[0205] "Picking procedure" refers to the order and method for removing specified products from shelves at a logistics center.

[0206] A "command" refers to an instruction that instructs a computer or robot to perform a specific action.

[0207] "Analysis" refers to the scientific process of breaking down data to understand its structure and relationships.

[0208] Deviation refers to a situation in which actual procedures or results deviate from goals or expectations.

[0209] "User" refers to a person who operates a system or device and uses it to achieve a purpose.

[0210] A "process" is a sequence of tasks or steps to achieve a specific goal.

[0211] "Generation" refers to the act of creating new data or procedures.

[0212] "Verification" refers to the process of verifying the accuracy of the results and commands generated.

[0213] This invention is a system that inputs the response results of a device and generates and presents the next necessary commands and procedures based on them. This system is particularly useful for operating picking robots in logistics centers.

[0214] System Program

[0215] A system for a picking robot used in a logistics center consists of the following elements:

[0216] Hardware: Picking robot, smartphone / tablet

[0217] Software: Robot control software, cloud-based AI analysis tools (e.g., AWS SageMaker), databases (e.g., AWS RDS)

[0218] Program processing description

[0219] Input of device response results

[0220] Users input the barcode information read by the robot and their current location information using their smartphone or tablet, and this data is sent to a cloud server.

[0221] Analysis of response results

[0222] The server analyzes the response and compares it with the product information and picking procedures in the database. For example, if an item is picked from the wrong shelf or the picking order is incorrect, the information is checked against the database.

[0223] Check for deviations

[0224] Based on the analysis results, the server determines whether the input response results deviate from the set target procedure. If a deviation is found, it is deemed to be a deviation.

[0225] Investigating ways to resolve discrepancies

[0226] The AI ​​analysis tool investigates and recalculates the optimal method to eliminate the discrepancy. Specifically, it recalculates and proposes the correct picking order and procedures.

[0227] Generate and view steps

[0228] The server generates specific steps and commands to resolve the discrepancy and displays them on the tablet or smartphone device, instructing the user or robot on the next action to be taken.

[0229] Specific examples

[0230] Data entry examples

[0231] The user enters the barcode information of the product "Product 12345" that the picking robot reads with its sensor into the terminal along with the user's current location "Section A" and sends it to the cloud server.

[0232] Specific examples of data analysis

[0233] The server checks the database to see that the received "item 12345" should be in section B, and determines that it was picked from section A by mistake.

[0234] Specific examples of checking discrepancies

[0235] The server determines that "picking from section A" is a deviation and records this.

[0236] Specific examples of methods for eliminating discrepancies

[0237] The AI ​​analysis tool recalculates the correct picking order and suggests the correct procedure: "Pick item 12345 from section B."

[0238] Example of procedure generation and display

[0239] The server generates instructions such as "Please pick product 12345 from section B" and displays them on the smartphone / tablet.

[0240] Prompt Sentence Examples

[0241] Prompt statement:

[0242] "Which section is product ID "12345" currently in and what is the correct picking procedure?"

[0243] This enables picking operations at logistics centers to be carried out efficiently and accurately.

[0244] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0245] Step 1:

[0246] The user inputs the barcode information read by the picking robot and their current location information on their smartphone or tablet. For example, they input the product ID "12345" and their current location "Section A," and send them to the cloud server. This input is done using a dedicated input form provided by the device.

[0247] Input: Product ID "12345", current location "Section A"

[0248] Output: Sending input data to the cloud server

[0249] Step 2:

[0250] The server analyzes the response received on the cloud (product ID "12345" and current location "Section A") and compares this information with the product information and picking procedures in the database.

[0251] Input: Product ID "12345", current location "Section A"

[0252] Output: Matching result (correct position or deviation information)

[0253] Step 3:

[0254] The server determines whether the input response result deviates from the set target procedure based on the collation result. If so, it determines that there is a deviation and records that information.

[0255] Input: Matching result

[0256] Output: Check result for discrepancy

[0257] Step 4:

[0258] The server's AI analysis tool investigates the optimal method for resolving the discrepancy. Specifically, it recalculates the correct picking order and procedures. For example, it recalculates the procedure for picking "product 12345" from the correct location (section B).

[0259] Input: Deviation check result

[0260] Output: Recalculation result of correct procedure

[0261] Step 5:

[0262] The server generates specific instructions and commands, such as "Please pick product 12345 from section B."

[0263] Input: Recalculated steps

[0264] Output: The specific steps or commands generated

[0265] Step 6:

[0266] The terminal displays the new instructions received from the server to the user. For example, the smartphone or tablet screen displays, "Please pick product 12345 from section B."

[0267] Input: The specific steps or commands generated

[0268] Output: On-screen instructions

[0269] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0270] This invention combines a system that inputs the response results of a device and generates and presents the next necessary commands and procedures based on that, with an emotion engine that recognizes the user's emotions. Below, the program processing of this system is explained in natural language and in detail with concrete examples.

[0271] Program Processing Overview

[0272] 1. The user inputs the device's response results.

[0273] The user inputs the response results obtained from the device into the terminal while working. For example, when the user inputs the reading of a temperature sensor device, the user inputs that the temperature is "85°C" into the terminal.

[0274] 2. The device sends the input response to the server.

[0275] The terminal sends this input data to the server, which converts the input data into a packet format and sends it to the server.

[0276] 3. The server analyzes the response

[0277] The server analyzes the response and checks whether it deviates from the set value or the expected normal value. For example, the server detects that the temperature "85°C" deviates from the set value "70°C".

[0278] 4. The server checks for discrepancies

[0279] The server uses the analysis results to determine whether there is a deviation from the target procedure. Because the temperature "85°C" exceeds the normal range of "70°C," the server determines that there is an "abnormality."

[0280] 5. Investigate how the server resolves the discrepancy

[0281] The server's AI module then goes to work and investigates the best way to resolve the discrepancy. For example, it may determine that turning on the cooling system is the best solution.

[0282] 6. The server verifies the findings

[0283] Based on the investigation results, the server verifies whether the generated response method is effective using simulations and past data.

[0284] 7. The server generates the next command and steps to be executed

[0285] Based on the verified method, the server generates specific commands and steps to be taken next, such as "turn on the cooling system and set the fans to maximum power."

[0286] 8. Recognize user emotions

[0287] The emotion engine recognizes the user's emotions, for example by using a camera and microphone to analyze the user's facial expressions and tone of voice to determine whether the user is feeling anxious or stressed.

[0288] 9. The server adjusts procedures based on emotions

[0289] The server adjusts the presented instructions based on the user's emotional data obtained from the emotion engine. For example, if the user feels anxious, the instructions may be made more detailed or more concise.

[0290] 10. The server sends the generated instructions to the device

[0291] The server converts the generated procedure into a packet format and transmits it to the terminal.

[0292] 11. The device presents instructions to the user

[0293] The device then displays the received instructions in a user-friendly format, for example, "Please turn on the cooling system and set the internal fan to maximum output."

[0294] 12. The user follows the steps provided

[0295] The user follows the instructions displayed on the terminal, turning on the cooling system and setting the fan to maximum output as instructed.

[0296] 13. The user enters the device's response again and sends it.

[0297] The user checks the device status after the procedure is executed, enters a new response result into the terminal, and transmits it again, starting the cycle again, generating and presenting new procedures as necessary.

[0298] Specific examples

[0299] 1. The user inputs the device's response results.

[0300] Example: A user enters a temperature sensor reading of "85°C" into a terminal and presses the send button.

[0301] 2. The device sends the input response to the server.

[0302] Example: The terminal sends the data "85°C" in packet format to the server.

[0303] 3. The server analyzes the response

[0304] Example: The server parses the input data "85°C" and verifies that the expected normal value is "up to 70°C."

[0305] 4. The server checks for discrepancies

[0306] Example: The server determines that the temperature "85°C" is abnormal because it exceeds the set value "70°C."

[0307] 5. Investigate how the server resolves the discrepancy

[0308] Example: A server's AI investigates whether operating cooling equipment is an effective countermeasure when temperatures exceed the limit, and verifies the procedure for operating the cooling system.

[0309] 6. The server generates the next command and procedure to be executed

[0310] Example: The server generates the procedure "Turn on the cooling system and set the internal fan to maximum output."

[0311] 7. Recognize user emotions

[0312] Example: An emotion engine analyzes a user's facial expression and tone of voice and determines that the user is feeling anxious.

[0313] 8. The server adjusts procedures based on emotions

[0314] Example: The server notices user anxiety and adjusts the procedure by adding more detailed instructions.

[0315] 9. The server sends the generated instructions to the device

[0316] Example: The server sends the adjusted instructions to the terminal.

[0317] 10. The device presents instructions to the user

[0318] Example: The terminal displays "Please turn on the cooling system and set the internal fan to maximum power" along with detailed instructions.

[0319] This system allows users to receive appropriate response procedures in real time based on the device's response results, enabling them to respond quickly. Furthermore, by using an emotion engine, it is possible to present appropriate procedures according to the user's emotional state, improving work efficiency and safety.

[0320] The processing flow will be explained below.

[0321] Step 1:

[0322] The user inputs the response from the device. The user inputs the response obtained from the device into the input field of the terminal and presses the send button. For example, the user inputs that the temperature sensor reading is 85°C.

[0323] Step 2:

[0324] The terminal sends the input response result to the server. The terminal converts the data input by the user into a packet format and transmits it to the server.

[0325] Step 3:

[0326] The server analyzes the response it receives. The server passes the received data to the analysis module, which compares the "Temperature 85°C" with the set value "70°C." Based on this analysis result, the server determines that the temperature is too high.

[0327] Step 4:

[0328] The server checks for any discrepancies. Based on the analysis results, the server determines that the temperature "85°C" is "abnormal" because it exceeds the set value "70°C." The server records this information in a log and proceeds to the next step.

[0329] Step 5:

[0330] The server investigates ways to resolve the discrepancy. The server's AI module searches for the optimal way to resolve the discrepancy. For example, it determines that running the cooling system would be effective.

[0331] Step 6:

[0332] The server verifies the investigation results. The server verifies the results of the AI ​​module using simulations and past data to confirm their reliability. This confirms that the generated response measures are effective.

[0333] Step 7:

[0334] The server generates the next command and procedure to be performed. Based on the verification results, the server generates specific commands and procedures, such as "turn on the cooling system and set the fan to maximum output."

[0335] Step 8:

[0336] The emotion engine recognizes the user's emotions. The emotion engine uses the device's built-in camera and microphone to analyze the user's facial expressions and tone of voice to determine whether the user is feeling anxious or stressed.

[0337] Step 9:

[0338] The server adjusts the steps based on the user's emotions. The server adjusts the content of the steps and the level of detail of the explanation based on the user's emotional data obtained from the emotion engine. For example, if the server determines that the user is feeling anxious, it will provide more detailed explanations of the steps.

[0339] Step 10:

[0340] The server sends the generated procedure to the terminal, converts the adjusted procedure into a packet format, and transmits the data to the terminal.

[0341] Step 11:

[0342] The terminal presents the instructions to the user. The terminal displays the received instructions in an easy-to-read format for the user. For example, it displays specific instructions such as "Turn on the cooling system and set the internal fan to maximum output" along with detailed instructions.

[0343] Step 12:

[0344] The user follows the instructions displayed on the device, powering on the cooling system and setting the fans to maximum power.

[0345] Step 13:

[0346] The user again inputs the response result of the device and sends it. The user checks the device status after the procedure is executed, inputs the new response result into the terminal, and sends it again. This process regenerates the necessary procedure.

[0347] This series of processes allows users to receive appropriate response procedures in real time, enabling them to take action quickly.In addition, the emotion engine presents procedures that take the user's emotional state into consideration, allowing them to work efficiently while reducing stress and anxiety.

[0348] Example 2

[0349] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0350] In conventional systems, when generating the next steps to be taken based on the device's response results, a uniform procedure was often presented without considering the user's emotions. As a result, even when the user was in a situation where they felt anxious or stressed, appropriate follow-up was not provided, which could lead to a decrease in work efficiency and safety. Another issue was that the generated procedures were sometimes insufficiently verified to be effective, resulting in the inability to take effective countermeasures.

[0351] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0352] In this invention, the server includes means for inputting the response results of the devices, means for analyzing the response results and checking for deviations from a target procedure, means for investigating ways to resolve the deviations and generating a procedure to be performed next, means for displaying the generated procedure, means for recognizing the user's emotion, and means for adjusting the procedure based on the recognized emotion. This makes it possible to provide an appropriate procedure according to the user's emotional state, thereby improving work efficiency and safety.

[0353] "Device" is a general term for hardware or software that has a specific function and outputs the results of its operation.

[0354] "Response results" refer to data such as measurement values ​​and operation results obtained from a device.

[0355] "Input means" refers to an interface or device that allows a user to input the response results of an apparatus into the system.

[0356] "Means for analyzing" refers to software modules or algorithms for processing the input response results and analyzing their contents.

[0357] "Target procedure" refers to the ideal work procedure or set values ​​that are predetermined by the system.

[0358] "Deviation" refers to the difference or discrepancy between the actual response result and the target procedure or set value.

[0359] "Investigative measures" refers to methods and algorithms for finding optimal solutions to resolve discrepancies.

[0360] "Means for generating" refers to a software module that has the function of generating specific steps or commands to be performed next based on the results of the investigation.

[0361] "Means for displaying" refers to a display or user interface for informing the user of the generated procedures or commands.

[0362] "User" refers to a person who operates this system and whose role is to input the response results of the device and carry out work according to the generated procedure.

[0363] "Means for recognizing emotions" refers to software or hardware that analyzes a user's facial expressions, tone of voice, etc. to determine their emotional state.

[0364] "Adjustment means" refers to a software module that has the functionality to change or adjust the content of the procedures or commands presented based on the recognized emotions.

[0365] This invention is a system in which the user inputs the device's response results into a terminal, and the server generates and presents the next required command or procedure based on the results. By combining this process with an emotion engine that recognizes the user's emotions, we have created a system that supports appropriate responses according to the user.

[0366] System configuration

[0367] Hardware and software used

[0368] 1. Terminal

[0369] Hardware: PCs, tablets, smartphones, etc.

[0370] Software: Web browser, input interface, communication module

[0371] 2. Server

[0372] Hardware: High-performance server

[0373] Software: AI module, data analysis algorithm, emotion recognition engine, communication module

[0374] 3. Emotion Recognition Engine

[0375] Hardware: Camera, microphone

[0376] Software: Image analysis software, audio analysis software

[0377] System Operation

[0378] This system operates in the following specific steps:

[0379] 1. The user inputs the device's response results.

[0380] The user inputs the response obtained from the device into the terminal, for example, entering data such as the temperature read from the temperature sensor (85°C), and presses the "Send" button.

[0381] 2. The device sends the input response to the server.

[0382] The terminal sends the input data to the server in packet format, using protocols such as HTTP.

[0383] 3. The server analyzes the response

[0384] The server analyzes the received data and compares it with expected normal values ​​or set values, for example, a set value of "70°C" stored in a database, to determine whether there is any deviation.

[0385] 4. The server investigates how to resolve the discrepancy and generates a procedure

[0386] The server's AI module investigates the best way to resolve the discrepancy and generates the next steps to take, such as turning on the cooling system.

[0387] 5. The server verifies the findings

[0388] The server verifies whether the investigation results are valid using simulation tools and past data logs.

[0389] 6. The server generates and sends the next command and procedure to be performed.

[0390] For example, it generates specific instructions such as "turn on the cooling system and set the fan to maximum output" and sends them to the terminal.

[0391] 7. Recognize user emotions

[0392] The emotion recognition engine analyzes the user's facial expressions and tone of voice to recognize emotions such as anxiety and stress.

[0393] 8. The server adjusts procedures based on emotions

[0394] The generated instructions are adjusted based on the recognized emotion data, for example, providing more detailed instructions if the user is feeling anxious.

[0395] 9. The device presents instructions to the user

[0396] The terminal displays the procedure received from the server to the user in a format that is easy for the user to understand.

[0397] 10. The user follows the steps provided

[0398] The user follows the displayed instructions, for example, turning on the cooling system and setting the fans to maximum power.

[0399] Examples of concrete examples and prompts

[0400] For example, consider a situation where a user inputs a temperature sensor reading of "85°C" into a terminal, and the server generates and presents a procedure to "turn on the cooling system and set the fan to maximum output" based on that. In this case, the following prompt sentence is used:

[0401] Sample prompt: "Generate the next steps based on the following device response result: '85°C'. The user is feeling anxious."

[0402] This system allows users to receive appropriate response procedures in real time, enabling them to respond quickly.In addition, by using an emotion recognition engine, it provides appropriate procedures according to the user's emotional state, improving work efficiency and safety.

[0403] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0404] Step 1:

[0405] The user inputs the response result of the device.

[0406] The user inputs the response obtained from the device into the terminal. For example, the reading from the temperature sensor is "85°C." The input data is the raw data from the temperature sensor (85°C), and the output is a data packet containing this data.

[0407] Step 2:

[0408] The terminal sends the input response results to the server.

[0409] The terminal converts the input data into a packet format and sends it to the server. Specifically, it sends the data using the HTTP protocol. The input is a data packet from the user, and the output is a data packet sent to the server.

[0410] Step 3:

[0411] The server analyzes the response

[0412] The server analyzes the received data (85°C). Specifically, it compares it with the set value "70°C" stored in the database. The input is the raw data (85°C) sent from the terminal, and the output is the analysis result: "85°C exceeds the set value of 70°C."

[0413] Step 4:

[0414] The server checks for discrepancies

[0415] Based on the analysis results, the server determines that the actual temperature exceeds the set value, indicating an abnormality. The input is the analysis result, and the output is the judgment on whether or not there is an abnormality (abnormality).

[0416] Step 5:

[0417] Investigate how the server resolves the discrepancy

[0418] The server's AI module runs and investigates the optimal solution. For example, it suggests turning on the cooling system. The input is the anomaly detection result, and the output is a candidate for the optimal solution (turning on the cooling system).

[0419] Step 6:

[0420] The server verifies the findings

[0421] The server verifies the effectiveness of the investigation results using simulations and past data logs. Specifically, it uses a simulation tool. The input is the candidate optimal countermeasures, and the output is the verification result (validity).

[0422] Step 7:

[0423] The server generates the next command and steps to be executed.

[0424] Based on the verification results, the server generates specific steps to be taken next, such as "turn on the cooling system and set the fan to maximum output." The input is the verification results, and the output is the specific steps.

[0425] Step 8:

[0426] Recognize user emotions

[0427] The emotion recognition engine analyzes the user's facial expressions and tone of voice to determine whether the user is feeling anxious or stressed. Specifically, it uses a camera and microphone. The input is the user's real-time voice and video data, and the output is the user's emotional state (anxiety).

[0428] Step 9:

[0429] The server adjusts the procedure based on emotions.

[0430] The server adjusts the detail and explanation of the procedure based on the emotional data. For example, if the user feels anxious, the server will provide more detailed instructions. The input is the emotional state data, and the output is the adjusted procedure.

[0431] Step 10:

[0432] The server sends the generated instructions to the device.

[0433] The server sends the generated instructions to the terminal in the form of packets, where the input is the adjusted instructions and the output is the data packets sent to the terminal.

[0434] Step 11:

[0435] The terminal presents instructions to the user

[0436] The terminal displays the instructions received from the server to the user. Specifically, it displays on the screen, "Please turn on the cooling system and set the fan to maximum output." The input is the instructions sent from the server, and the output is the display to the user.

[0437] Step 12:

[0438] The user follows the steps provided

[0439] The user follows the instructions displayed on the terminal to perform physical and software tasks, such as turning on the power to a cooling system and setting the internal fan to maximum output. The input is the displayed instructions, and the output is the result of executing the instructions.

[0440] Step 13:

[0441] The user re-enters the device's response and sends it.

[0442] The user checks the device status after the procedure is executed and inputs a new response result (e.g., temperature "75°C") into the terminal. Then, the user presses the "Send" button again to send the data to the server. The input is the new response result, and the output is the resent data packet.

[0443] (Application example 2)

[0444] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0445] While conventional device monitoring systems can generate procedures based on device response results, they are unable to adjust procedures to take into account the user's emotional state. This has led to issues such as users becoming stressed, especially in emergency situations or complex situations, and performance degradation due to operational errors and tension. This can compromise efficiency and safety in scenarios that require rapid and accurate responses.

[0446] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes a means for inputting the device's response result, a means for analyzing the response result and checking for deviation from a target procedure, a means for investigating a method for eliminating the deviation and generating a procedure to be performed next, a means for recognizing the user's emotion, and a means for adjusting the procedure based on the recognized user emotion. This makes it possible to generate an appropriate procedure based on the device's response result and further provide detailed guidance according to the user's emotion. This reduces user stress and enables a quick and accurate response.

[0447] "Device response results" are data or output information generated by a device during operation.

[0448] A "means" is a method, process, or device used to accomplish a particular purpose.

[0449] "Target procedures" are procedures or standard values ​​that are set in advance to achieve normal operation or a goal.

[0450] "Deviation" means a deviation or abnormal variation from an expected normal range or set point.

[0451] "Research" is the process of collecting and analyzing information to find appropriate methods and procedures for solving a problem.

[0452] "Generation" is the act of creating new procedures or commands based on specific conditions or data.

[0453] "Display" refers to the act of visually presenting information to a user.

[0454] "Emotion recognition" is a technology that analyzes a user's facial expressions, tone of voice, etc. to determine their emotional state.

[0455] "Adjustment" is the act of changing or modifying procedures or instructions to suit specific conditions or circumstances.

[0456] A "system" is a collection of integrated devices or processes in which multiple elements or means work together to perform a single function.

[0457] "Device" refers to an electronic device or terminal used for a specific purpose.

[0458] A "server" is a device that stores and processes data on a network and provides services to other devices and users.

[0459] "Input of response results" refers to the act of a user manually or automatically inputting data or results output by a device into the system.

[0460] "Data analysis" is the process of analyzing collected data to find meaningful information and patterns.

[0461] "Emotion-based procedure adjustment" refers to the act of appropriately changing the procedures and guidance presented to the user in consideration of the user's emotional state.

[0462] This invention relates to a system that generates and presents the next steps required based on the response results of the device, and also recognizes and adjusts the user's emotions. This system is intended to monitor and manage devices and robots used primarily in factories.

[0463] The system includes the following major components:

[0464] 1. User Device

[0465] The user device inputs the response results from the device and sends the data to the server. The device is typically a smartphone or tablet, and is equipped with a camera and microphone. This allows it to recognize the user's emotions.

[0466] 2. Server

[0467] The server is the central processing unit that analyzes the data, generates procedures as needed, and presents them to the user. The server includes the following functions:

[0468] Data Analysis Module

[0469] The server receives and analyzes the device response results sent from the user terminal, checks for deviations from the normal range, and investigates and determines the optimal method to resolve the deviations.

[0470] Procedure Generation Module

[0471] Based on the analysis results, the next steps to be taken are generated and verified to see if they are effective.

[0472] Emotion Recognition Module

[0473] It analyzes the video and audio data sent from the user's device to determine the user's emotional state (anxiety, stress, etc.).

[0474] Procedure Coordination Module

[0475] Based on the results of the emotion recognition module, the generated instructions are adjusted, specifically determining whether to provide detailed or concise instructions depending on the emotional state.

[0476] 3. Presentation to the User

[0477] The generated procedure is again transmitted to the user terminal and visually presented to the user.

[0478] As a concrete example, consider a situation where a temperature sensor in a factory detects 85°C, which exceeds the set value of 70°C. The user enters this reading into a terminal and sends it to the server. The server analyzes this data and determines that the cooling fan needs to be turned on. Furthermore, if the emotion recognition module detects the user's anxiety, it adjusts the procedure to "Turn on the cooling system and set the internal fan to maximum output. If you have any questions during the procedure, please refer to the manual."

[0479] The main software and hardware used are cloud computing platforms (e.g., AWS, Google Cloud) for the server, and smartphones (compatible with iOS or Android) for the user devices. The EmotionRecognition library is used for emotion recognition.

[0480] A specific example of a prompt for a generative AI model is, "The temperature sensor detected 85°C. This is an abnormal condition, exceeding the set value (70°C). Please generate the following steps. If the user is feeling uneasy, please explain the steps in detail." Using this prompt, the AI ​​can generate optimal steps and provide appropriate guidance to the user.

[0481] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0482] Step 1:

[0483] The user inputs the device's response result. The user inputs the device's response result (e.g., the temperature sensor reading "85°C") into a smartphone or tablet and presses the send button. The input data is sent to the device in text format.

[0484] (Input) Device response result (temperature "85°C")

[0485] (Output) Response result data sent from the user terminal

[0486] Step 2:

[0487] The device sends the input response results to the server. The device converts the sent data into packet format and sends it to the cloud server. At that time, security protocols (e.g., SSL / TLS) are used to maintain the integrity of the data.

[0488] (Input) Response result data entered by the user (temperature "85°C")

[0489] (Output) Response result data sent to the cloud server

[0490] Step 3:

[0491] The server analyzes the response results. The server analyzes the received data and checks for deviations from normal values ​​or set values ​​(e.g., "70°C"). The data analysis module does this and detects abnormalities.

[0492] (Input) Response result data sent to the cloud server (temperature "85°C")

[0493] (Output) Deviation judgment result (temperature "85°C" is abnormal)

[0494] Step 4:

[0495] The server investigates ways to resolve the discrepancy and generates the next steps to be taken. An AI module then runs to find the optimal solution based on past data and simulations (e.g., running a cooling system).

[0496] (Input) Deviation judgment result (temperature "85°C" is abnormal)

[0497] (Output) Initial action steps (e.g., turning on cooling system)

[0498] Step 5:

[0499] The server recognizes the user's emotions. Camera footage and audio data are sent from the user's device, and the emotion recognition module analyzes them to determine whether the user is feeling anxious.

[0500] (Input) User's video and audio data

[0501] (Output) Emotional state (e.g., anxiety)

[0502] Step 6:

[0503] The server adjusts the instructions based on the emotion. Based on the emotion recognition results, the server adjusts the instructions it presents (e.g., explains the instructions in detail).

[0504] (Input) Initial response steps (turn on cooling system), emotional state (anxiety)

[0505] (Output) Adjusted countermeasure steps (with detailed explanation)

[0506] Step 7:

[0507] The server sends the generated procedure to the terminal, converts the generated procedure into a packet format, and sends it to the user's terminal.

[0508] (Input) Adjusted countermeasure procedures (with detailed explanations)

[0509] (Output) Instructions sent to the user terminal

[0510] Step 8:

[0511] The terminal presents instructions to the user. The terminal visually displays the received instructions and presents them in a format that is easy for the user to understand (e.g., "Please turn on the cooling system and set the internal fan to maximum output. If you have any questions during the procedure, please refer to the manual.").

[0512] (Input) Procedure data received from the server

[0513] (Output) Instructions displayed to the user

[0514] Step 9:

[0515] The user carries out the presented procedure. The user performs the work according to the instructions displayed on the terminal.

[0516] (Input) Instructions displayed to the user

[0517] (Output) User execution of steps

[0518] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0519] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0520] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0521] [Second embodiment]

[0522] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0523] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0524] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0525] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0526] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0527] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0528] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0529] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0530] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0531] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0532] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0533] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0534] This invention is a system that inputs the response results of a device and generates and presents the next required commands and procedures based on them. Below, the program processing of this system is explained in natural language and in detail with concrete examples.

[0535] Program Processing Overview

[0536] 1. The user inputs the device response results into the server.

[0537] The user inputs the response results obtained from the device into the terminal while working. For example, when the user inputs the reading of a temperature sensor device, the user inputs that the temperature is "85°C" into the terminal.

[0538] The terminal transmits this input data to the server.

[0539] 2. The server analyzes the response

[0540] The server analyzes the response and checks whether it deviates from the set value or the expected normal value. For example, the server detects that the temperature "85°C" exceeds the set value "70°C."

[0541] The server records logs based on the analysis results and stores them for future analysis.

[0542] 3. The server checks for discrepancies

[0543] Based on the analysis results, the server determines whether the response deviates from the target procedure. The server determines that the temperature of 85°C is abnormal because it exceeds the normal range of 70°C.

[0544] Based on the result of this determination, the next step is to determine the necessary action.

[0545] 4. Investigate how the server resolves the discrepancy

[0546] The server's AI module investigates the best way to resolve the discrepancy. If the temperature exceeds the set point, the server's AI determines that activating the cooling system would be effective.

[0547] Validate the findings and generate specific implementation steps.

[0548] 5. The server generates the next command and steps to be executed

[0549] Based on the results of the investigation and verification, the server generates the next command and procedure to be performed, such as "switch on the cooling device" or "restart the device."

[0550] The generated procedure is formatted so that it is easy for the user to understand, and is sent to the terminal.

[0551] 6. Present the procedure for creating the device to the user

[0552] The device will then display the new instructions received from the server to the user, for example, "Please turn on the cooling device."

[0553] After the user has performed the procedure, he is prompted to enter the device's response again.

[0554] Specific examples

[0555] 1. The user inputs the device response results into the server.

[0556] Example: A user enters a temperature sensor reading of "85°C" into a terminal and presses the send button.

[0557] 2. The server analyzes the response

[0558] Example: The server analyzes the input data "85°C" and verifies that the expected normal value is "up to 70°C." The server records this information in a log and determines that there is a discrepancy.

[0559] 3. The server checks for discrepancies

[0560] Example: The server determines that the temperature "85°C" is abnormal because it exceeds the set value "70°C."

[0561] 4. Investigate how the server resolves the discrepancy

[0562] Example: A server's AI investigates whether operating cooling equipment is an effective countermeasure when temperatures exceed the limit, and verifies the procedure for operating the cooling system.

[0563] 5. The server generates the next command and steps to be executed

[0564] Example: The server generates the procedure "Turn on the cooling system and set the internal fan to maximum output."

[0565] 6. Present the procedure for creating the device to the user

[0566] Example: The terminal will prompt the user to "Please turn on the cooling system and set the internal fan to maximum power." After the user performs the steps, they will be prompted to enter the device's response again.

[0567] This system allows users to receive appropriate response procedures in real time based on the device's response results, enabling them to respond quickly, preventing work interruptions and realizing efficient and flexible work.

[0568] The processing flow will be explained below.

[0569] Step 1:

[0570] The user inputs the device response results into the terminal. The user inputs the device response results obtained during work into the terminal's input field and presses the send button. For example, the user inputs the reading from the temperature sensor, "85°C."

[0571] Step 2:

[0572] The terminal transmits the input response result to the server. The terminal converts the data input by the user into a packet format and transmits it to the server.

[0573] Step 3:

[0574] The server analyzes the response it receives. The server passes the received data to the analysis module, which analyzes the contents of the response. For example, the server detects that the temperature "85°C" deviates from the set value "70°C."

[0575] Step 4:

[0576] The server checks for any deviations. Based on the analysis results, the server determines whether there is any deviation from the target procedure. Because the temperature "85°C" exceeds the normal range of "70°C," the server determines that there is an "abnormality."

[0577] Step 5:

[0578] The server investigates ways to resolve the discrepancy. The server's AI module runs and investigates the best way to resolve the discrepancy. For example, it determines that turning on the cooling system is the best option.

[0579] Step 6:

[0580] The server verifies the investigation results. Based on the investigation results, the server verifies whether the generated response method is effective using simulations and past data.

[0581] Step 7:

[0582] The server generates the next command and procedure to be performed. Based on the verified method, the server generates the specific command and procedure to be performed next. For example, it creates a specific procedure such as "turn on the cooling system and set the fan to maximum output."

[0583] Step 8:

[0584] The server transmits the generated procedure to the terminal, converts the generated procedure into a packet format, and transmits it to the terminal.

[0585] Step 9:

[0586] The terminal presents the instructions to the user. The terminal displays the received instructions in an easy-to-read format for the user. For example, it displays "Please turn on the cooling system and set the internal fan to maximum output."

[0587] Step 10:

[0588] The user follows the instructions displayed on the device, powering on the cooling system and setting the fans to maximum power as instructed.

[0589] Step 11:

[0590] The user again inputs the device's response result into the terminal and sends it. The user checks the device status after the procedure is executed, inputs a new response result into the terminal, and sends it again.

[0591] Step 12:

[0592] The device sends a new response to the server. The device converts the received data back into packets and sends them to the server. The cycle begins again, generating and submitting new instructions as needed.

[0593] Example 1

[0594] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0595] Conventionally, the generation and presentation of procedures and commands based on device response results has often been done manually, resulting in problems such as reduced work efficiency and accuracy. Furthermore, manual analysis and response takes time, making it difficult to respond in real time. The objective of this invention is to provide a system that automatically analyzes device response results and quickly and accurately presents the necessary procedures and commands to the user.

[0596] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0597] In this invention, the server includes a means for inputting the response results of the device, a means for analyzing the response results and checking for deviations from the target procedure, and a means for investigating a method for eliminating the deviation using a generative AI model and generating the procedure to be performed next, thereby enabling the user to receive the appropriate response procedure in real time and respond quickly.

[0598] The "means for inputting the response results of the device" is a function that allows the user to input data or information obtained from the device into a terminal or the like and transmit that data to the server.

[0599] "Means of analysis" refers to the process by which the server compares the data it receives with set values ​​and expected normal values ​​to determine any deviations or abnormalities in the data.

[0600] The "means for checking deviation from the target procedure" is a function that determines, based on the analyzed data, how much the actual results deviate from the set target procedure.

[0601] "Means of investigating using a generative AI model and generating the next steps to be taken" refers to the process of using an AI module in the server to investigate the optimal method for resolving any discrepancies or anomalies that have occurred and generate specific response steps.

[0602] The "means for displaying procedures" is a function that visualizes and presents the generated response procedures and commands to the user.

[0603] "Means for inputting to a data management device" refers to an interface or operating procedure for inputting and sending the generated procedures and commands to a data management device such as a server.

[0604] The "means for generating specific operational instructions" is a function that generates specific commands and operational procedures to be executed for a specific device or system based on the analysis results.

[0605] The "means for verifying whether the operation instruction is valid" is a process for checking and confirming whether the generated specific operation instruction actually has an effect.

[0606] The present invention is a system that inputs a device response result and generates and presents the next required command or procedure based on that result. The system includes a means for inputting the device response result, a means for analyzing it, a means for checking for discrepancies, a means for generating the next procedure using a generative AI model, and a means for displaying the procedure.

[0607] Hardware and software used

[0608] Device: The computing device (smartphone, tablet, PC) operated by the user.

[0609] Server: A server that analyzes data, records logs, generates next steps, etc. It is built using a database or cloud infrastructure (e.g., Amazon Web Services or Microsoft Azure).

[0610] Generative AI model: An AI module (e.g., TensorFlow, PyTorch, etc.) used for data analysis and procedure generation.

[0611] Natural language explanation of program processing

[0612] 1. The user enters the device's response into the terminal and sends it to the server.

[0613] The user inputs the response obtained from the device into the terminal. For example, the user inputs "85°C" as the temperature sensor reading.

[0614] The terminal transmits the input data to the server.

[0615] 2. The server receives the response and analyzes the data.

[0616] The server receives the data sent from the device and analyzes it by comparing it with the set value and expected normal value. It checks whether the temperature "85°C" exceeds the set value "70°C".

[0617] The server records the analysis results as a log and detects any deviations.

[0618] 3. The server determines whether there is a discrepancy

[0619] Based on the analysis results, the server determines how much the input data deviates from the set value. Since the temperature "85°C" exceeds the set value "70°C," it is judged to be "abnormal."

[0620] Based on this decision, the server determines the next action to be taken.

[0621] 4. The server uses a generative AI model to investigate and select a method to resolve the discrepancy.

[0622] The server uses a generative AI model to investigate the best way to resolve the anomaly. If the temperature is high, the investigation confirms that turning on a cooling device would be beneficial.

[0623] Generate specific steps based on the survey results.

[0624] 5. The server generates the next steps to be taken and sends them to the device.

[0625] The server then formats the generated instructions into a format that is easy for the user to understand, such as "Please turn on the cooling device."

[0626] This procedure is sent to the terminal.

[0627] 6. Present the procedure for creating the device to the user

[0628] The device displays the instructions received from the server to the user, for example, "Please turn on the cooling device."

[0629] The user acts according to the presented procedure and inputs the response results of the device again.

[0630] Examples of specific examples and prompts

[0631] Specific examples

[0632] 1. The user inputs the device's response results.

[0633] Example: A user enters the temperature sensor reading "85°C" into the terminal input form and presses the "Submit" button.

[0634] 2. The server analyzes the response

[0635] Example: The server receives the data "85°C" and compares it with the set value "70°C" to detect an abnormality.

[0636] 3. The server checks for discrepancies

[0637] Example: The server temperature "85°C" exceeds the set value "70°C", so it is logged as an abnormality.

[0638] 4. Investigate how the server resolves the discrepancy

[0639] Example: Using a generative AI model, determine that running a cooling unit is the optimal solution.

[0640] 5. The server generates the next steps to be taken

[0641] Example: The server generates the procedure "Switch on the cooling device and set the internal fan to maximum power."

[0642] 6. Present the procedure for creating the device to the user

[0643] Example: The device displays to the user, "Please turn on the cooling device."

[0644] Prompt Sentence Examples

[0645] Generate a procedure for what to do if the temperature is too high.

[0646] Generate instructions to set the cooling system to maximum power.

[0647] Please indicate the procedure to be followed in response to an abnormal value of 85°C.

[0648] As described above, the present invention aims to provide the user with an appropriate response procedure in real time based on the response results of the device, thereby realizing a quick response.

[0649] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0650] Step 1:

[0651] The user inputs the device's response into the terminal and sends it to the server.

[0652] Specific operation: The user enters the response obtained from the device, for example, the temperature sensor reading "85°C," into the terminal and presses the send button.

[0653] Input: Temperature sensor reading "85°C".

[0654] Output: Sending data from the device to the server.

[0655] Step 2:

[0656] The server receives the response result

[0657] Specific operation: The server receives data sent from the device via HTTP request or WebSocket.

[0658] Input: Temperature data sent from the device: "85°C".

[0659] Output: The temperature data received by the server.

[0660] Step 3:

[0661] The server analyzes the response

[0662] Specific operation: The server compares the received data with the set value or expected normal value to check for deviations. For example, it checks whether the temperature "85°C" exceeds the set value "70°C".

[0663] Input: Temperature data received by the server is "85°C", set value is "70°C".

[0664] Output: Anomaly detection results.

[0665] Step 4:

[0666] The server determines whether there is a discrepancy

[0667] Specific operation: The server determines how much the temperature deviates from the set value based on the analysis results. Since the temperature "85°C" exceeds the set value "70°C", it is determined to be abnormal.

[0668] Input: Anomaly detection results.

[0669] Output: Log of abnormality detection.

[0670] Step 5:

[0671] The server uses generative AI models to explore solutions

[0672] Specific behavior: The generative AI model in the server investigates the best way to resolve the anomaly, for example, investigating whether running a cooling system is effective.

[0673] Input: Abnormality determination log, abnormal value "85°C".

[0674] Output: The optimal solution and its validation results.

[0675] Step 6:

[0676] The server generates the next steps and sends them to the device.

[0677] Specific operation: The server formats the generated instructions into a format that is easy for the user to understand and sends it to the terminal. For example, it creates and sends an instruction such as "Please turn on the cooling device."

[0678] Input: Validation results of the optimal solution.

[0679] Output: Generates specific operating instructions and sends them to the terminal.

[0680] Step 7:

[0681] Present the procedure for the device to be generated to the user

[0682] Specific behavior: The device displays the instructions sent from the server to the user, such as "Please turn on the cooling device."

[0683] Input: Specific operating instructions received from the server.

[0684] Output: The instructions presented to the user.

[0685] Step 8:

[0686] The user performs the procedure and enters the response again.

[0687] Specific operation: The user follows the presented procedure, for example, turning on the cooling device, and then inputs the device's response again.

[0688] Input: Procedures presented by the server, and the results of the user's execution.

[0689] Output: The response result of the input device again.

[0690] (Application example 1)

[0691] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0692] Picking work at logistics centers requires a lot of manpower and time, so there is a need to improve work efficiency. It is also important to quickly correct errors that occur during work and achieve accurate picking. In response to these challenges, the present invention provides a system that enables picking robots to receive optimal work procedures in real time and perform work efficiently and accurately.

[0693] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0694] In this invention, the server includes means for recalculating the picking procedure for the logistics robot based on the response result of the device and generating a corrected command, means for displaying the generated procedure to the user and inputting the response result to the server, means for presenting an appropriate procedure in real time in the logistics process, means for generating specific picking commands, and means for verifying whether the generated commands are accurate and optimal, thereby enabling improved efficiency and accuracy of picking work in logistics centers.

[0695] "Device response result" refers to the result information that a device returns in response to some input or operation.

[0696] A "procedure" refers to the sequence of tasks or operations that are used to achieve a particular purpose.

[0697] A "server" is a computer system that stores, manages, and processes data over a network.

[0698] A "logistics robot" is a robot that automates the movement and picking of goods at logistics centers and other locations.

[0699] "Real-time" refers to a method in which data input, processing, and response occur within an extremely short period of time.

[0700] "Picking procedure" refers to the order and method for removing specified products from shelves at a logistics center.

[0701] A "command" refers to an instruction that instructs a computer or robot to perform a specific action.

[0702] "Analysis" refers to the scientific process of breaking down data to understand its structure and relationships.

[0703] Deviation refers to a situation in which actual procedures or results deviate from goals or expectations.

[0704] "User" refers to a person who operates a system or device and uses it to achieve a purpose.

[0705] A "process" is a sequence of tasks or steps to achieve a specific goal.

[0706] "Generation" refers to the act of creating new data or procedures.

[0707] "Verification" refers to the process of verifying the accuracy of the results and commands generated.

[0708] This invention is a system that inputs the response results of a device and generates and presents the next necessary commands and procedures based on them. This system is particularly useful for operating picking robots in logistics centers.

[0709] System Program

[0710] A system for a picking robot used in a logistics center consists of the following elements:

[0711] Hardware: Picking robot, smartphone / tablet

[0712] Software: Robot control software, cloud-based AI analysis tools (e.g., AWS SageMaker), databases (e.g., AWS RDS)

[0713] Program processing description

[0714] Input of device response results

[0715] Users input the barcode information read by the robot and their current location information using their smartphone or tablet, and this data is sent to a cloud server.

[0716] Analysis of response results

[0717] The server analyzes the response and compares it with the product information and picking procedures in the database. For example, if an item is picked from the wrong shelf or the picking order is incorrect, the information is checked against the database.

[0718] Check for deviations

[0719] Based on the analysis results, the server determines whether the input response results deviate from the set target procedure. If a deviation is found, it is deemed to be a deviation.

[0720] Investigating ways to resolve discrepancies

[0721] The AI ​​analysis tool investigates and recalculates the optimal method to eliminate the discrepancy. Specifically, it recalculates and proposes the correct picking order and procedures.

[0722] Generate and view steps

[0723] The server generates specific steps and commands to resolve the discrepancy and displays them on the tablet or smartphone device, instructing the user or robot on the next action to be taken.

[0724] Specific examples

[0725] Data entry examples

[0726] The user enters the barcode information of the product "Product 12345" that the picking robot reads with its sensor into the terminal along with the user's current location "Section A" and sends it to the cloud server.

[0727] Specific examples of data analysis

[0728] The server checks the database to see that the received "item 12345" should be in section B, and determines that it was picked from section A by mistake.

[0729] Specific examples of checking discrepancies

[0730] The server determines that "picking from section A" is a deviation and records this.

[0731] Specific examples of methods for eliminating discrepancies

[0732] The AI ​​analysis tool recalculates the correct picking order and suggests the correct procedure: "Pick item 12345 from section B."

[0733] Example of procedure generation and display

[0734] The server generates instructions such as "Please pick product 12345 from section B" and displays them on the smartphone / tablet.

[0735] Prompt Sentence Examples

[0736] Prompt statement:

[0737] "Which section is product ID "12345" currently in and what is the correct picking procedure?"

[0738] This enables picking operations at logistics centers to be carried out efficiently and accurately.

[0739] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0740] Step 1:

[0741] The user inputs the barcode information read by the picking robot and their current location information on their smartphone or tablet. For example, they input the product ID "12345" and their current location "Section A," and send them to the cloud server. This input is done using a dedicated input form provided by the device.

[0742] Input: Product ID "12345", current location "Section A"

[0743] Output: Sending input data to the cloud server

[0744] Step 2:

[0745] The server analyzes the response received on the cloud (product ID "12345" and current location "Section A") and compares this information with the product information and picking procedures in the database.

[0746] Input: Product ID "12345", current location "Section A"

[0747] Output: Matching result (correct position or deviation information)

[0748] Step 3:

[0749] The server determines whether the input response result deviates from the set target procedure based on the collation result. If so, it determines that there is a deviation and records that information.

[0750] Input: Matching result

[0751] Output: Check result for discrepancy

[0752] Step 4:

[0753] The server's AI analysis tool investigates the optimal method for resolving the discrepancy. Specifically, it recalculates the correct picking order and procedures. For example, it recalculates the procedure for picking "product 12345" from the correct location (section B).

[0754] Input: Deviation check result

[0755] Output: Recalculation result of correct procedure

[0756] Step 5:

[0757] The server generates specific instructions and commands, such as "Please pick product 12345 from section B."

[0758] Input: Recalculated steps

[0759] Output: The specific steps or commands generated

[0760] Step 6:

[0761] The terminal displays the new instructions received from the server to the user. For example, the smartphone or tablet screen displays, "Please pick product 12345 from section B."

[0762] Input: The specific steps or commands generated

[0763] Output: On-screen instructions

[0764] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0765] This invention combines a system that inputs the response results of a device and generates and presents the next necessary commands and procedures based on that, with an emotion engine that recognizes the user's emotions. Below, the program processing of this system is explained in natural language and in detail with concrete examples.

[0766] Program Processing Overview

[0767] 1. The user inputs the device's response results.

[0768] The user inputs the response results obtained from the device into the terminal while working. For example, when the user inputs the reading of a temperature sensor device, the user inputs that the temperature is "85°C" into the terminal.

[0769] 2. The device sends the input response to the server.

[0770] The terminal sends this input data to the server, which converts the input data into a packet format and sends it to the server.

[0771] 3. The server analyzes the response

[0772] The server analyzes the response and checks whether it deviates from the set value or the expected normal value. For example, the server detects that the temperature "85°C" deviates from the set value "70°C".

[0773] 4. The server checks for discrepancies

[0774] The server uses the analysis results to determine whether there is a deviation from the target procedure. Because the temperature "85°C" exceeds the normal range of "70°C," the server determines that there is an "abnormality."

[0775] 5. Investigate how the server resolves the discrepancy

[0776] The server's AI module then goes to work and investigates the best way to resolve the discrepancy. For example, it may determine that turning on the cooling system is the best solution.

[0777] 6. The server verifies the findings

[0778] Based on the investigation results, the server verifies whether the generated response method is effective using simulations and past data.

[0779] 7. The server generates the next command and steps to be executed

[0780] Based on the verified method, the server generates specific commands and steps to be taken next, such as "turn on the cooling system and set the fans to maximum power."

[0781] 8. Recognize user emotions

[0782] The emotion engine recognizes the user's emotions, for example by using a camera and microphone to analyze the user's facial expressions and tone of voice to determine whether the user is feeling anxious or stressed.

[0783] 9. The server adjusts procedures based on emotions

[0784] The server adjusts the presented instructions based on the user's emotional data obtained from the emotion engine. For example, if the user feels anxious, the instructions may be made more detailed or more concise.

[0785] 10. The server sends the generated instructions to the device

[0786] The server converts the generated procedure into a packet format and transmits it to the terminal.

[0787] 11. The device presents instructions to the user

[0788] The device then displays the received instructions in a user-friendly format, for example, "Please turn on the cooling system and set the internal fan to maximum output."

[0789] 12. The user follows the steps provided

[0790] The user follows the instructions displayed on the terminal, turning on the cooling system and setting the fan to maximum output as instructed.

[0791] 13. The user enters the device's response again and sends it.

[0792] The user checks the device status after the procedure is executed, enters a new response result into the terminal, and transmits it again, starting the cycle again, generating and presenting new procedures as necessary.

[0793] Specific examples

[0794] 1. The user inputs the device's response results.

[0795] Example: A user enters a temperature sensor reading of "85°C" into a terminal and presses the send button.

[0796] 2. The device sends the input response to the server.

[0797] Example: The terminal sends the data "85°C" in packet format to the server.

[0798] 3. The server analyzes the response

[0799] Example: The server parses the input data "85°C" and verifies that the expected normal value is "up to 70°C."

[0800] 4. The server checks for discrepancies

[0801] Example: The server determines that the temperature "85°C" is abnormal because it exceeds the set value "70°C."

[0802] 5. Investigate how the server resolves the discrepancy

[0803] Example: A server's AI investigates whether operating cooling equipment is an effective countermeasure when temperatures exceed the limit, and verifies the procedure for operating the cooling system.

[0804] 6. The server generates the next command and procedure to be executed

[0805] Example: The server generates the procedure "Turn on the cooling system and set the internal fan to maximum output."

[0806] 7. Recognize user emotions

[0807] Example: An emotion engine analyzes a user's facial expression and tone of voice and determines that the user is feeling anxious.

[0808] 8. The server adjusts procedures based on emotions

[0809] Example: The server notices user anxiety and adjusts the procedure by adding more detailed instructions.

[0810] 9. The server sends the generated instructions to the device

[0811] Example: The server sends the adjusted instructions to the terminal.

[0812] 10. The device presents instructions to the user

[0813] Example: The terminal displays "Please turn on the cooling system and set the internal fan to maximum power" along with detailed instructions.

[0814] This system allows users to receive appropriate response procedures in real time based on the device's response results, enabling them to respond quickly. Furthermore, by using an emotion engine, it is possible to present appropriate procedures according to the user's emotional state, improving work efficiency and safety.

[0815] The processing flow will be explained below.

[0816] Step 1:

[0817] The user inputs the response from the device. The user inputs the response obtained from the device into the input field of the terminal and presses the send button. For example, the user inputs that the temperature sensor reading is 85°C.

[0818] Step 2:

[0819] The terminal sends the input response result to the server. The terminal converts the data input by the user into a packet format and transmits it to the server.

[0820] Step 3:

[0821] The server analyzes the response it receives. The server passes the received data to the analysis module, which compares the "Temperature 85°C" with the set value "70°C." Based on this analysis result, the server determines that the temperature is too high.

[0822] Step 4:

[0823] The server checks for any discrepancies. Based on the analysis results, the server determines that the temperature "85°C" is "abnormal" because it exceeds the set value "70°C." The server records this information in a log and proceeds to the next step.

[0824] Step 5:

[0825] The server investigates ways to resolve the discrepancy. The server's AI module searches for the optimal way to resolve the discrepancy. For example, it determines that running the cooling system would be effective.

[0826] Step 6:

[0827] The server verifies the investigation results. The server verifies the results of the AI ​​module using simulations and past data to confirm their reliability. This confirms that the generated response measures are effective.

[0828] Step 7:

[0829] The server generates the next command and procedure to be performed. Based on the verification results, the server generates specific commands and procedures, such as "turn on the cooling system and set the fan to maximum output."

[0830] Step 8:

[0831] The emotion engine recognizes the user's emotions. The emotion engine uses the device's built-in camera and microphone to analyze the user's facial expressions and tone of voice to determine whether the user is feeling anxious or stressed.

[0832] Step 9:

[0833] The server adjusts the steps based on the user's emotions. The server adjusts the content of the steps and the level of detail of the explanation based on the user's emotional data obtained from the emotion engine. For example, if the server determines that the user is feeling anxious, it will provide more detailed explanations of the steps.

[0834] Step 10:

[0835] The server sends the generated procedure to the terminal, converts the adjusted procedure into a packet format, and transmits the data to the terminal.

[0836] Step 11:

[0837] The terminal presents the instructions to the user. The terminal displays the received instructions in an easy-to-read format for the user. For example, it displays specific instructions such as "Turn on the cooling system and set the internal fan to maximum output" along with detailed instructions.

[0838] Step 12:

[0839] The user follows the instructions displayed on the device, powering on the cooling system and setting the fans to maximum power.

[0840] Step 13:

[0841] The user again inputs the response result of the device and sends it. The user checks the device status after the procedure is executed, inputs the new response result into the terminal, and sends it again. This process regenerates the necessary procedure.

[0842] This series of processes allows users to receive appropriate response procedures in real time, enabling them to take action quickly.In addition, the emotion engine presents procedures that take the user's emotional state into consideration, allowing them to work efficiently while reducing stress and anxiety.

[0843] Example 2

[0844] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0845] In conventional systems, when generating the next steps to be taken based on the device's response results, a uniform procedure was often presented without considering the user's emotions. As a result, even when the user was in a situation where they felt anxious or stressed, appropriate follow-up was not provided, which could lead to a decrease in work efficiency and safety. Another issue was that the generated procedures were sometimes insufficiently verified to be effective, resulting in the inability to take effective countermeasures.

[0846] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0847] In this invention, the server includes means for inputting the response results of the devices, means for analyzing the response results and checking for deviations from a target procedure, means for investigating ways to resolve the deviations and generating a procedure to be performed next, means for displaying the generated procedure, means for recognizing the user's emotion, and means for adjusting the procedure based on the recognized emotion. This makes it possible to provide an appropriate procedure according to the user's emotional state, thereby improving work efficiency and safety.

[0848] "Device" is a general term for hardware or software that has a specific function and outputs the results of its operation.

[0849] "Response results" refer to data such as measurement values ​​and operation results obtained from a device.

[0850] "Input means" refers to an interface or device that allows a user to input the response results of an apparatus into the system.

[0851] "Means for analyzing" refers to software modules or algorithms for processing the input response results and analyzing their contents.

[0852] "Target procedure" refers to the ideal work procedure or set values ​​that are predetermined by the system.

[0853] "Deviation" refers to the difference or discrepancy between the actual response result and the target procedure or set value.

[0854] "Investigative measures" refers to methods and algorithms for finding optimal solutions to resolve discrepancies.

[0855] "Means for generating" refers to a software module that has the function of generating specific steps or commands to be performed next based on the results of the investigation.

[0856] "Means for displaying" refers to a display or user interface for informing the user of the generated procedures or commands.

[0857] "User" refers to a person who operates this system and whose role is to input the response results of the device and carry out work according to the generated procedure.

[0858] "Means for recognizing emotions" refers to software or hardware that analyzes a user's facial expressions, tone of voice, etc. to determine their emotional state.

[0859] "Adjustment means" refers to a software module that has the functionality to change or adjust the content of the procedures or commands presented based on the recognized emotions.

[0860] This invention is a system in which the user inputs the device's response results into a terminal, and the server generates and presents the next required command or procedure based on the results. By combining this process with an emotion engine that recognizes the user's emotions, we have created a system that supports appropriate responses according to the user.

[0861] System configuration

[0862] Hardware and software used

[0863] 1. Terminal

[0864] Hardware: PCs, tablets, smartphones, etc.

[0865] Software: Web browser, input interface, communication module

[0866] 2. Server

[0867] Hardware: High-performance server

[0868] Software: AI module, data analysis algorithm, emotion recognition engine, communication module

[0869] 3. Emotion Recognition Engine

[0870] Hardware: Camera, microphone

[0871] Software: Image analysis software, audio analysis software

[0872] System Operation

[0873] This system operates in the following specific steps:

[0874] 1. The user inputs the device's response results.

[0875] The user inputs the response obtained from the device into the terminal, for example, entering data such as the temperature read from the temperature sensor (85°C), and presses the "Send" button.

[0876] 2. The device sends the input response to the server.

[0877] The terminal sends the input data to the server in packet format, using protocols such as HTTP.

[0878] 3. The server analyzes the response

[0879] The server analyzes the received data and compares it with expected normal values ​​or set values, for example, a set value of "70°C" stored in a database, to determine whether there is any deviation.

[0880] 4. The server investigates how to resolve the discrepancy and generates a procedure

[0881] The server's AI module investigates the best way to resolve the discrepancy and generates the next steps to take, such as turning on the cooling system.

[0882] 5. The server verifies the findings

[0883] The server verifies whether the investigation results are valid using simulation tools and past data logs.

[0884] 6. The server generates and sends the next command and procedure to be performed.

[0885] For example, it generates specific instructions such as "turn on the cooling system and set the fan to maximum output" and sends them to the terminal.

[0886] 7. Recognize user emotions

[0887] The emotion recognition engine analyzes the user's facial expressions and tone of voice to recognize emotions such as anxiety and stress.

[0888] 8. The server adjusts procedures based on emotions

[0889] The generated instructions are adjusted based on the recognized emotion data, for example, providing more detailed instructions if the user is feeling anxious.

[0890] 9. The device presents instructions to the user

[0891] The terminal displays the procedure received from the server to the user in a format that is easy for the user to understand.

[0892] 10. The user follows the steps provided

[0893] The user follows the displayed instructions, for example, turning on the cooling system and setting the fans to maximum power.

[0894] Examples of concrete examples and prompts

[0895] For example, consider a situation where a user inputs a temperature sensor reading of "85°C" into a terminal, and the server generates and presents a procedure to "turn on the cooling system and set the fan to maximum output" based on that. In this case, the following prompt sentence is used:

[0896] Sample prompt: "Generate the next steps based on the following device response result: '85°C'. The user is feeling anxious."

[0897] This system allows users to receive appropriate response procedures in real time, enabling them to respond quickly.In addition, by using an emotion recognition engine, it provides appropriate procedures according to the user's emotional state, improving work efficiency and safety.

[0898] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0899] Step 1:

[0900] The user inputs the response result of the device.

[0901] The user inputs the response obtained from the device into the terminal. For example, the reading from the temperature sensor is "85°C." The input data is the raw data from the temperature sensor (85°C), and the output is a data packet containing this data.

[0902] Step 2:

[0903] The terminal sends the input response results to the server.

[0904] The terminal converts the input data into a packet format and sends it to the server. Specifically, it sends the data using the HTTP protocol. The input is a data packet from the user, and the output is a data packet sent to the server.

[0905] Step 3:

[0906] The server analyzes the response

[0907] The server analyzes the received data (85°C). Specifically, it compares it with the set value "70°C" stored in the database. The input is the raw data (85°C) sent from the terminal, and the output is the analysis result: "85°C exceeds the set value of 70°C."

[0908] Step 4:

[0909] The server checks for discrepancies

[0910] Based on the analysis results, the server determines that the actual temperature exceeds the set value, indicating an abnormality. The input is the analysis result, and the output is the judgment on whether or not there is an abnormality (abnormality).

[0911] Step 5:

[0912] Investigate how the server resolves the discrepancy

[0913] The server's AI module runs and investigates the optimal solution. For example, it suggests turning on the cooling system. The input is the anomaly detection result, and the output is a candidate for the optimal solution (turning on the cooling system).

[0914] Step 6:

[0915] The server verifies the findings

[0916] The server verifies the effectiveness of the investigation results using simulations and past data logs. Specifically, it uses a simulation tool. The input is the candidate optimal countermeasures, and the output is the verification result (validity).

[0917] Step 7:

[0918] The server generates the next command and steps to be executed.

[0919] Based on the verification results, the server generates specific steps to be taken next, such as "turn on the cooling system and set the fan to maximum output." The input is the verification results, and the output is the specific steps.

[0920] Step 8:

[0921] Recognize user emotions

[0922] The emotion recognition engine analyzes the user's facial expressions and tone of voice to determine whether the user is feeling anxious or stressed. Specifically, it uses a camera and microphone. The input is the user's real-time voice and video data, and the output is the user's emotional state (anxiety).

[0923] Step 9:

[0924] The server adjusts the procedure based on emotions.

[0925] The server adjusts the detail and explanation of the procedure based on the emotional data. For example, if the user feels anxious, the server will provide more detailed instructions. The input is the emotional state data, and the output is the adjusted procedure.

[0926] Step 10:

[0927] The server sends the generated instructions to the device.

[0928] The server sends the generated instructions to the terminal in the form of packets, where the input is the adjusted instructions and the output is the data packets sent to the terminal.

[0929] Step 11:

[0930] The terminal presents instructions to the user

[0931] The terminal displays the instructions received from the server to the user. Specifically, it displays on the screen, "Please turn on the cooling system and set the fan to maximum output." The input is the instructions sent from the server, and the output is the display to the user.

[0932] Step 12:

[0933] The user follows the steps provided

[0934] The user follows the instructions displayed on the terminal to perform physical and software tasks, such as turning on the power to a cooling system and setting the internal fan to maximum output. The input is the displayed instructions, and the output is the result of executing the instructions.

[0935] Step 13:

[0936] The user re-enters the device's response and sends it.

[0937] The user checks the device status after the procedure is executed and inputs a new response result (e.g., temperature "75°C") into the terminal. Then, the user presses the "Send" button again to send the data to the server. The input is the new response result, and the output is the resent data packet.

[0938] (Application example 2)

[0939] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0940] While conventional device monitoring systems can generate procedures based on device response results, they are unable to adjust procedures to take into account the user's emotional state. This has led to issues such as users becoming stressed, especially in emergency situations or complex situations, and performance degradation due to operational errors and tension. This can compromise efficiency and safety in scenarios that require rapid and accurate responses.

[0941] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes a means for inputting the device's response result, a means for analyzing the response result and checking for deviation from a target procedure, a means for investigating a method for eliminating the deviation and generating a procedure to be performed next, a means for recognizing the user's emotion, and a means for adjusting the procedure based on the recognized user emotion. This makes it possible to generate an appropriate procedure based on the device's response result and further provide detailed guidance according to the user's emotion. This reduces user stress and enables a quick and accurate response.

[0942] "Device response results" are data or output information generated by a device during operation.

[0943] A "means" is a method, process, or device used to accomplish a particular purpose.

[0944] "Target procedures" are procedures or standard values ​​that are set in advance to achieve normal operation or a goal.

[0945] "Deviation" means a deviation or abnormal variation from an expected normal range or set point.

[0946] "Research" is the process of collecting and analyzing information to find appropriate methods and procedures for solving a problem.

[0947] "Generation" is the act of creating new procedures or commands based on specific conditions or data.

[0948] "Display" refers to the act of visually presenting information to a user.

[0949] "Emotion recognition" is a technology that analyzes a user's facial expressions, tone of voice, etc. to determine their emotional state.

[0950] "Adjustment" is the act of changing or modifying procedures or instructions to suit specific conditions or circumstances.

[0951] A "system" is a collection of integrated devices or processes in which multiple elements or means work together to perform a single function.

[0952] "Device" refers to an electronic device or terminal used for a specific purpose.

[0953] A "server" is a device that stores and processes data on a network and provides services to other devices and users.

[0954] "Input of response results" refers to the act of a user manually or automatically inputting data or results output by a device into the system.

[0955] "Data analysis" is the process of analyzing collected data to find meaningful information and patterns.

[0956] "Emotion-based procedure adjustment" refers to the act of appropriately changing the procedures and guidance presented to the user in consideration of the user's emotional state.

[0957] This invention relates to a system that generates and presents the next steps required based on the response results of the device, and also recognizes and adjusts the user's emotions. This system is intended to monitor and manage devices and robots used primarily in factories.

[0958] The system includes the following major components:

[0959] 1. User Device

[0960] The user device inputs the response results from the device and sends the data to the server. The device is typically a smartphone or tablet, and is equipped with a camera and microphone. This allows it to recognize the user's emotions.

[0961] 2. Server

[0962] The server is the central processing unit that analyzes the data, generates procedures as needed, and presents them to the user. The server includes the following functions:

[0963] Data Analysis Module

[0964] The server receives and analyzes the device response results sent from the user terminal, checks for deviations from the normal range, and investigates and determines the optimal method to resolve the deviations.

[0965] Procedure Generation Module

[0966] Based on the analysis results, the next steps to be taken are generated and verified to see if they are effective.

[0967] Emotion Recognition Module

[0968] It analyzes the video and audio data sent from the user's device to determine the user's emotional state (anxiety, stress, etc.).

[0969] Procedure Coordination Module

[0970] Based on the results of the emotion recognition module, the generated instructions are adjusted, specifically determining whether to provide detailed or concise instructions depending on the emotional state.

[0971] 3. Presentation to the User

[0972] The generated procedure is again transmitted to the user terminal and visually presented to the user.

[0973] As a concrete example, consider a situation where a temperature sensor in a factory detects 85°C, which exceeds the set value of 70°C. The user enters this reading into a terminal and sends it to the server. The server analyzes this data and determines that the cooling fan needs to be turned on. Furthermore, if the emotion recognition module detects the user's anxiety, it adjusts the procedure to "Turn on the cooling system and set the internal fan to maximum output. If you have any questions during the procedure, please refer to the manual."

[0974] The main software and hardware used are cloud computing platforms (e.g., AWS, Google Cloud) for the server, and smartphones (compatible with iOS or Android) for the user devices. The EmotionRecognition library is used for emotion recognition.

[0975] A specific example of a prompt for a generative AI model is, "The temperature sensor detected 85°C. This is an abnormal condition, exceeding the set value (70°C). Please generate the following steps. If the user is feeling uneasy, please explain the steps in detail." Using this prompt, the AI ​​can generate optimal steps and provide appropriate guidance to the user.

[0976] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0977] Step 1:

[0978] The user inputs the device's response result. The user inputs the device's response result (e.g., the temperature sensor reading "85°C") into a smartphone or tablet and presses the send button. The input data is sent to the device in text format.

[0979] (Input) Device response result (temperature "85°C")

[0980] (Output) Response result data sent from the user terminal

[0981] Step 2:

[0982] The device sends the input response results to the server. The device converts the sent data into packet format and sends it to the cloud server. At that time, security protocols (e.g., SSL / TLS) are used to maintain the integrity of the data.

[0983] (Input) Response result data entered by the user (temperature "85°C")

[0984] (Output) Response result data sent to the cloud server

[0985] Step 3:

[0986] The server analyzes the response results. The server analyzes the received data and checks for deviations from normal values ​​or set values ​​(e.g., "70°C"). The data analysis module does this and detects abnormalities.

[0987] (Input) Response result data sent to the cloud server (temperature "85°C")

[0988] (Output) Deviation judgment result (temperature "85°C" is abnormal)

[0989] Step 4:

[0990] The server investigates ways to resolve the discrepancy and generates the next steps to be taken. An AI module then runs to find the optimal solution based on past data and simulations (e.g., running a cooling system).

[0991] (Input) Deviation judgment result (temperature "85°C" is abnormal)

[0992] (Output) Initial action steps (e.g., turning on cooling system)

[0993] Step 5:

[0994] The server recognizes the user's emotions. Camera footage and audio data are sent from the user's device, and the emotion recognition module analyzes them to determine whether the user is feeling anxious.

[0995] (Input) User's video and audio data

[0996] (Output) Emotional state (e.g., anxiety)

[0997] Step 6:

[0998] The server adjusts the instructions based on the emotion. Based on the emotion recognition results, the server adjusts the instructions it presents (e.g., explains the instructions in detail).

[0999] (Input) Initial response steps (turn on cooling system), emotional state (anxiety)

[1000] (Output) Adjusted countermeasure steps (with detailed explanation)

[1001] Step 7:

[1002] The server sends the generated procedure to the terminal, converts the generated procedure into a packet format, and sends it to the user's terminal.

[1003] (Input) Adjusted countermeasure procedures (with detailed explanations)

[1004] (Output) Instructions sent to the user terminal

[1005] Step 8:

[1006] The terminal presents instructions to the user. The terminal visually displays the received instructions and presents them in a format that is easy for the user to understand (e.g., "Please turn on the cooling system and set the internal fan to maximum output. If you have any questions during the procedure, please refer to the manual.").

[1007] (Input) Procedure data received from the server

[1008] (Output) Instructions displayed to the user

[1009] Step 9:

[1010] The user carries out the presented procedure. The user performs the work according to the instructions displayed on the terminal.

[1011] (Input) Instructions displayed to the user

[1012] (Output) User execution of steps

[1013] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1014] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1015] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1016] [Third embodiment]

[1017] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1018] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[1019] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1020] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1021] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1022] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1023] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1024] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1025] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1026] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1027] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1028] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1029] This invention is a system that inputs the response results of a device and generates and presents the next required commands and procedures based on them. Below, the program processing of this system is explained in natural language and in detail with concrete examples.

[1030] Program Processing Overview

[1031] 1. The user inputs the device response results into the server.

[1032] The user inputs the response results obtained from the device into the terminal while working. For example, when the user inputs the reading of a temperature sensor device, the user inputs that the temperature is "85°C" into the terminal.

[1033] The terminal transmits this input data to the server.

[1034] 2. The server analyzes the response

[1035] The server analyzes the response and checks whether it deviates from the set value or the expected normal value. For example, the server detects that the temperature "85°C" exceeds the set value "70°C."

[1036] The server records logs based on the analysis results and stores them for future analysis.

[1037] 3. The server checks for discrepancies

[1038] Based on the analysis results, the server determines whether the response deviates from the target procedure. The server determines that the temperature of 85°C is abnormal because it exceeds the normal range of 70°C.

[1039] Based on the result of this determination, the next step is to determine the necessary action.

[1040] 4. Investigate how the server resolves the discrepancy

[1041] The server's AI module investigates the best way to resolve the discrepancy. If the temperature exceeds the set point, the server's AI determines that activating the cooling system would be effective.

[1042] Validate the findings and generate specific implementation steps.

[1043] 5. The server generates the next command and steps to be executed

[1044] Based on the results of the investigation and verification, the server generates the next command and procedure to be performed, such as "switch on the cooling device" or "restart the device."

[1045] The generated procedure is formatted so that it is easy for the user to understand, and is sent to the terminal.

[1046] 6. Present the procedure for creating the device to the user

[1047] The device will then display the new instructions received from the server to the user, for example, "Please turn on the cooling device."

[1048] After the user has performed the procedure, he is prompted to enter the device's response again.

[1049] Specific examples

[1050] 1. The user inputs the device response results into the server.

[1051] Example: A user enters a temperature sensor reading of "85°C" into a terminal and presses the send button.

[1052] 2. The server analyzes the response

[1053] Example: The server analyzes the input data "85°C" and verifies that the expected normal value is "up to 70°C." The server records this information in a log and determines that there is a discrepancy.

[1054] 3. The server checks for discrepancies

[1055] Example: The server determines that the temperature "85°C" is abnormal because it exceeds the set value "70°C."

[1056] 4. Investigate how the server resolves the discrepancy

[1057] Example: A server's AI investigates whether operating cooling equipment is an effective countermeasure when temperatures exceed the limit, and verifies the procedure for operating the cooling system.

[1058] 5. The server generates the next command and steps to be executed

[1059] Example: The server generates the procedure "Turn on the cooling system and set the internal fan to maximum output."

[1060] 6. Present the procedure for creating the device to the user

[1061] Example: The terminal will prompt the user to "Please turn on the cooling system and set the internal fan to maximum power." After the user performs the steps, they will be prompted to enter the device's response again.

[1062] This system allows users to receive appropriate response procedures in real time based on the device's response results, enabling them to respond quickly, preventing work interruptions and realizing efficient and flexible work.

[1063] The processing flow will be explained below.

[1064] Step 1:

[1065] The user inputs the device response results into the terminal. The user inputs the device response results obtained during work into the terminal's input field and presses the send button. For example, the user inputs the reading from the temperature sensor, "85°C."

[1066] Step 2:

[1067] The terminal transmits the input response result to the server. The terminal converts the data input by the user into a packet format and transmits it to the server.

[1068] Step 3:

[1069] The server analyzes the response it receives. The server passes the received data to the analysis module, which analyzes the contents of the response. For example, the server detects that the temperature "85°C" deviates from the set value "70°C."

[1070] Step 4:

[1071] The server checks for any deviations. Based on the analysis results, the server determines whether there is any deviation from the target procedure. Because the temperature "85°C" exceeds the normal range of "70°C," the server determines that there is an "abnormality."

[1072] Step 5:

[1073] The server investigates ways to resolve the discrepancy. The server's AI module runs and investigates the best way to resolve the discrepancy. For example, it determines that turning on the cooling system is the best option.

[1074] Step 6:

[1075] The server verifies the investigation results. Based on the investigation results, the server verifies whether the generated response method is effective using simulations and past data.

[1076] Step 7:

[1077] The server generates the next command and procedure to be performed. Based on the verified method, the server generates the specific command and procedure to be performed next. For example, it creates a specific procedure such as "turn on the cooling system and set the fan to maximum output."

[1078] Step 8:

[1079] The server transmits the generated procedure to the terminal, converts the generated procedure into a packet format, and transmits it to the terminal.

[1080] Step 9:

[1081] The terminal presents the instructions to the user. The terminal displays the received instructions in an easy-to-read format for the user. For example, it displays "Please turn on the cooling system and set the internal fan to maximum output."

[1082] Step 10:

[1083] The user follows the instructions displayed on the device, powering on the cooling system and setting the fans to maximum power as instructed.

[1084] Step 11:

[1085] The user again inputs the device's response result into the terminal and sends it. The user checks the device status after the procedure is executed, inputs a new response result into the terminal, and sends it again.

[1086] Step 12:

[1087] The device sends a new response to the server. The device converts the received data back into packets and sends them to the server. The cycle begins again, generating and submitting new instructions as needed.

[1088] Example 1

[1089] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1090] Conventionally, the generation and presentation of procedures and commands based on device response results has often been done manually, resulting in problems such as reduced work efficiency and accuracy. Furthermore, manual analysis and response takes time, making it difficult to respond in real time. The objective of this invention is to provide a system that automatically analyzes device response results and quickly and accurately presents the necessary procedures and commands to the user.

[1091] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1092] In this invention, the server includes a means for inputting the response results of the device, a means for analyzing the response results and checking for deviations from the target procedure, and a means for investigating a method for eliminating the deviation using a generative AI model and generating the procedure to be performed next, thereby enabling the user to receive the appropriate response procedure in real time and respond quickly.

[1093] The "means for inputting the response results of the device" is a function that allows the user to input data or information obtained from the device into a terminal or the like and transmit that data to the server.

[1094] "Means of analysis" refers to the process by which the server compares the data it receives with set values ​​and expected normal values ​​to determine any deviations or abnormalities in the data.

[1095] The "means for checking deviation from the target procedure" is a function that determines, based on the analyzed data, how much the actual results deviate from the set target procedure.

[1096] "Means of investigating using a generative AI model and generating the next steps to be taken" refers to the process of using an AI module in the server to investigate the optimal method for resolving any discrepancies or anomalies that have occurred and generate specific response steps.

[1097] The "means for displaying procedures" is a function that visualizes and presents the generated response procedures and commands to the user.

[1098] "Means for inputting to a data management device" refers to an interface or operating procedure for inputting and sending the generated procedures and commands to a data management device such as a server.

[1099] The "means for generating specific operational instructions" is a function that generates specific commands and operational procedures to be executed for a specific device or system based on the analysis results.

[1100] The "means for verifying whether the operation instruction is valid" is a process for checking and confirming whether the generated specific operation instruction actually has an effect.

[1101] The present invention is a system that inputs a device response result and generates and presents the next required command or procedure based on that result. The system includes a means for inputting the device response result, a means for analyzing it, a means for checking for discrepancies, a means for generating the next procedure using a generative AI model, and a means for displaying the procedure.

[1102] Hardware and software used

[1103] Device: The computing device (smartphone, tablet, PC) operated by the user.

[1104] Server: A server that analyzes data, records logs, generates next steps, etc. It is built using a database or cloud infrastructure (e.g., Amazon Web Services or Microsoft Azure).

[1105] Generative AI model: An AI module (e.g., TensorFlow, PyTorch, etc.) used for data analysis and procedure generation.

[1106] Natural language explanation of program processing

[1107] 1. The user enters the device's response into the terminal and sends it to the server.

[1108] The user inputs the response obtained from the device into the terminal. For example, the user inputs "85°C" as the temperature sensor reading.

[1109] The terminal transmits the input data to the server.

[1110] 2. The server receives the response and analyzes the data.

[1111] The server receives the data sent from the device and analyzes it by comparing it with the set value and expected normal value. It checks whether the temperature "85°C" exceeds the set value "70°C".

[1112] The server records the analysis results as a log and detects any deviations.

[1113] 3. The server determines whether there is a discrepancy

[1114] Based on the analysis results, the server determines how much the input data deviates from the set value. Since the temperature "85°C" exceeds the set value "70°C," it is judged to be "abnormal."

[1115] Based on this decision, the server determines the next action to be taken.

[1116] 4. The server uses a generative AI model to investigate and select a method to resolve the discrepancy.

[1117] The server uses a generative AI model to investigate the best way to resolve the anomaly. If the temperature is high, the investigation confirms that turning on a cooling device would be beneficial.

[1118] Generate specific steps based on the survey results.

[1119] 5. The server generates the next steps to be taken and sends them to the device.

[1120] The server then formats the generated instructions into a format that is easy for the user to understand, such as "Please turn on the cooling device."

[1121] This procedure is sent to the terminal.

[1122] 6. Present the procedure for creating the device to the user

[1123] The device displays the instructions received from the server to the user, for example, "Please turn on the cooling device."

[1124] The user acts according to the presented procedure and inputs the response results of the device again.

[1125] Examples of specific examples and prompts

[1126] Specific examples

[1127] 1. The user inputs the device's response results.

[1128] Example: A user enters the temperature sensor reading "85°C" into the terminal input form and presses the "Submit" button.

[1129] 2. The server analyzes the response

[1130] Example: The server receives the data "85°C" and compares it with the set value "70°C" to detect an abnormality.

[1131] 3. The server checks for discrepancies

[1132] Example: The server temperature "85°C" exceeds the set value "70°C", so it is logged as an abnormality.

[1133] 4. Investigate how the server resolves the discrepancy

[1134] Example: Using a generative AI model, determine that running a cooling unit is the optimal solution.

[1135] 5. The server generates the next steps to be taken

[1136] Example: The server generates the procedure "Switch on the cooling device and set the internal fan to maximum power."

[1137] 6. Present the procedure for creating the device to the user

[1138] Example: The device displays to the user, "Please turn on the cooling device."

[1139] Prompt Sentence Examples

[1140] Generate a procedure for what to do if the temperature is too high.

[1141] Generate instructions to set the cooling system to maximum power.

[1142] Please indicate the procedure to be followed in response to an abnormal value of 85°C.

[1143] As described above, the present invention aims to provide the user with an appropriate response procedure in real time based on the response results of the device, thereby realizing a quick response.

[1144] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1145] Step 1:

[1146] The user inputs the device's response into the terminal and sends it to the server.

[1147] Specific operation: The user enters the response obtained from the device, for example, the temperature sensor reading "85°C," into the terminal and presses the send button.

[1148] Input: Temperature sensor reading "85°C".

[1149] Output: Sending data from the device to the server.

[1150] Step 2:

[1151] The server receives the response result

[1152] Specific operation: The server receives data sent from the device via HTTP request or WebSocket.

[1153] Input: Temperature data sent from the device: "85°C".

[1154] Output: The temperature data received by the server.

[1155] Step 3:

[1156] The server analyzes the response

[1157] Specific operation: The server compares the received data with the set value or expected normal value to check for deviations. For example, it checks whether the temperature "85°C" exceeds the set value "70°C".

[1158] Input: Temperature data received by the server is "85°C", set value is "70°C".

[1159] Output: Anomaly detection results.

[1160] Step 4:

[1161] The server determines whether there is a discrepancy

[1162] Specific operation: The server determines how much the temperature deviates from the set value based on the analysis results. Since the temperature "85°C" exceeds the set value "70°C", it is determined to be abnormal.

[1163] Input: Anomaly detection results.

[1164] Output: Log of abnormality detection.

[1165] Step 5:

[1166] The server uses generative AI models to explore solutions

[1167] Specific behavior: The generative AI model in the server investigates the best way to resolve the anomaly, for example, investigating whether running a cooling system is effective.

[1168] Input: Abnormality determination log, abnormal value "85°C".

[1169] Output: The optimal solution and its validation results.

[1170] Step 6:

[1171] The server generates the next steps and sends them to the device.

[1172] Specific operation: The server formats the generated instructions into a format that is easy for the user to understand and sends it to the terminal. For example, it creates and sends an instruction such as "Please turn on the cooling device."

[1173] Input: Validation results of the optimal solution.

[1174] Output: Generates specific operating instructions and sends them to the terminal.

[1175] Step 7:

[1176] Present the procedure for the device to be generated to the user

[1177] Specific behavior: The device displays the instructions sent from the server to the user, such as "Please turn on the cooling device."

[1178] Input: Specific operating instructions received from the server.

[1179] Output: The instructions presented to the user.

[1180] Step 8:

[1181] The user performs the procedure and enters the response again.

[1182] Specific operation: The user follows the presented procedure, for example, turning on the cooling device, and then inputs the device's response again.

[1183] Input: Procedures presented by the server, and the results of the user's execution.

[1184] Output: The response result of the input device again.

[1185] (Application example 1)

[1186] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1187] Picking work at logistics centers requires a lot of manpower and time, so there is a need to improve work efficiency. It is also important to quickly correct errors that occur during work and achieve accurate picking. In response to these challenges, the present invention provides a system that enables picking robots to receive optimal work procedures in real time and perform work efficiently and accurately.

[1188] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1189] In this invention, the server includes means for recalculating the picking procedure for the logistics robot based on the response result of the device and generating a corrected command, means for displaying the generated procedure to the user and inputting the response result to the server, means for presenting an appropriate procedure in real time in the logistics process, means for generating specific picking commands, and means for verifying whether the generated commands are accurate and optimal, thereby enabling improved efficiency and accuracy of picking work in logistics centers.

[1190] "Device response result" refers to the result information that a device returns in response to some input or operation.

[1191] A "procedure" refers to the sequence of tasks or operations that are used to achieve a particular purpose.

[1192] A "server" is a computer system that stores, manages, and processes data over a network.

[1193] A "logistics robot" is a robot that automates the movement and picking of goods at logistics centers and other locations.

[1194] "Real-time" refers to a method in which data input, processing, and response occur within an extremely short period of time.

[1195] "Picking procedure" refers to the order and method for removing specified products from shelves at a logistics center.

[1196] A "command" refers to an instruction that instructs a computer or robot to perform a specific action.

[1197] "Analysis" refers to the scientific process of breaking down data to understand its structure and relationships.

[1198] Deviation refers to a situation in which actual procedures or results deviate from goals or expectations.

[1199] "User" refers to a person who operates a system or device and uses it to achieve a purpose.

[1200] A "process" is a sequence of tasks or steps to achieve a specific goal.

[1201] "Generation" refers to the act of creating new data or procedures.

[1202] "Verification" refers to the process of verifying the accuracy of the results and commands generated.

[1203] This invention is a system that inputs the response results of a device and generates and presents the next necessary commands and procedures based on them. This system is particularly useful for operating picking robots in logistics centers.

[1204] System Program

[1205] A system for a picking robot used in a logistics center consists of the following elements:

[1206] Hardware: Picking robot, smartphone / tablet

[1207] Software: Robot control software, cloud-based AI analysis tools (e.g., AWS SageMaker), databases (e.g., AWS RDS)

[1208] Program processing description

[1209] Input of device response results

[1210] Users input the barcode information read by the robot and their current location information using their smartphone or tablet, and this data is sent to a cloud server.

[1211] Analysis of response results

[1212] The server analyzes the response and compares it with the product information and picking procedures in the database. For example, if an item is picked from the wrong shelf or the picking order is incorrect, the information is checked against the database.

[1213] Check for deviations

[1214] Based on the analysis results, the server determines whether the input response results deviate from the set target procedure. If a deviation is found, it is deemed to be a deviation.

[1215] Investigating ways to resolve discrepancies

[1216] The AI ​​analysis tool investigates and recalculates the optimal method to eliminate the discrepancy. Specifically, it recalculates and proposes the correct picking order and procedures.

[1217] Generate and view steps

[1218] The server generates specific steps and commands to resolve the discrepancy and displays them on the tablet or smartphone device, instructing the user or robot on the next action to be taken.

[1219] Specific examples

[1220] Data entry examples

[1221] The user enters the barcode information of the product "Product 12345" that the picking robot reads with its sensor into the terminal along with the user's current location "Section A" and sends it to the cloud server.

[1222] Specific examples of data analysis

[1223] The server checks the database to see that the received "item 12345" should be in section B, and determines that it was picked from section A by mistake.

[1224] Specific examples of checking discrepancies

[1225] The server determines that "picking from section A" is a deviation and records this.

[1226] Specific examples of methods for eliminating discrepancies

[1227] The AI ​​analysis tool recalculates the correct picking order and suggests the correct procedure: "Pick item 12345 from section B."

[1228] Example of procedure generation and display

[1229] The server generates instructions such as "Please pick product 12345 from section B" and displays them on the smartphone / tablet.

[1230] Prompt Sentence Examples

[1231] Prompt statement:

[1232] "Which section is product ID "12345" currently in and what is the correct picking procedure?"

[1233] This enables picking operations at logistics centers to be carried out efficiently and accurately.

[1234] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1235] Step 1:

[1236] The user inputs the barcode information read by the picking robot and their current location information on their smartphone or tablet. For example, they input the product ID "12345" and their current location "Section A," and send them to the cloud server. This input is done using a dedicated input form provided by the device.

[1237] Input: Product ID "12345", current location "Section A"

[1238] Output: Sending input data to the cloud server

[1239] Step 2:

[1240] The server analyzes the response received on the cloud (product ID "12345" and current location "Section A") and compares this information with the product information and picking procedures in the database.

[1241] Input: Product ID "12345", current location "Section A"

[1242] Output: Matching result (correct position or deviation information)

[1243] Step 3:

[1244] The server determines whether the input response result deviates from the set target procedure based on the collation result. If so, it determines that there is a deviation and records that information.

[1245] Input: Matching result

[1246] Output: Check result for discrepancy

[1247] Step 4:

[1248] The server's AI analysis tool investigates the optimal method for resolving the discrepancy. Specifically, it recalculates the correct picking order and procedures. For example, it recalculates the procedure for picking "product 12345" from the correct location (section B).

[1249] Input: Deviation check result

[1250] Output: Recalculation result of correct procedure

[1251] Step 5:

[1252] The server generates specific instructions and commands, such as "Please pick product 12345 from section B."

[1253] Input: Recalculated steps

[1254] Output: The specific steps or commands generated

[1255] Step 6:

[1256] The terminal displays the new instructions received from the server to the user. For example, the smartphone or tablet screen displays, "Please pick product 12345 from section B."

[1257] Input: The specific steps or commands generated

[1258] Output: On-screen instructions

[1259] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1260] This invention combines a system that inputs the response results of a device and generates and presents the next necessary commands and procedures based on that, with an emotion engine that recognizes the user's emotions. Below, the program processing of this system is explained in natural language and in detail with concrete examples.

[1261] Program Processing Overview

[1262] 1. The user inputs the device's response results.

[1263] The user inputs the response results obtained from the device into the terminal while working. For example, when the user inputs the reading of a temperature sensor device, the user inputs that the temperature is "85°C" into the terminal.

[1264] 2. The device sends the input response to the server.

[1265] The terminal sends this input data to the server, which converts the input data into a packet format and sends it to the server.

[1266] 3. The server analyzes the response

[1267] The server analyzes the response and checks whether it deviates from the set value or the expected normal value. For example, the server detects that the temperature "85°C" deviates from the set value "70°C".

[1268] 4. The server checks for discrepancies

[1269] The server uses the analysis results to determine whether there is a deviation from the target procedure. Because the temperature "85°C" exceeds the normal range of "70°C," the server determines that there is an "abnormality."

[1270] 5. Investigate how the server resolves the discrepancy

[1271] The server's AI module then goes to work and investigates the best way to resolve the discrepancy. For example, it may determine that turning on the cooling system is the best solution.

[1272] 6. The server verifies the findings

[1273] Based on the investigation results, the server verifies whether the generated response method is effective using simulations and past data.

[1274] 7. The server generates the next command and steps to be executed

[1275] Based on the verified method, the server generates specific commands and steps to be taken next, such as "turn on the cooling system and set the fans to maximum power."

[1276] 8. Recognize user emotions

[1277] The emotion engine recognizes the user's emotions, for example by using a camera and microphone to analyze the user's facial expressions and tone of voice to determine whether the user is feeling anxious or stressed.

[1278] 9. The server adjusts procedures based on emotions

[1279] The server adjusts the presented instructions based on the user's emotional data obtained from the emotion engine. For example, if the user feels anxious, the instructions may be made more detailed or more concise.

[1280] 10. The server sends the generated instructions to the device

[1281] The server converts the generated procedure into a packet format and transmits it to the terminal.

[1282] 11. The device presents instructions to the user

[1283] The device then displays the received instructions in a user-friendly format, for example, "Please turn on the cooling system and set the internal fan to maximum output."

[1284] 12. The user follows the steps provided

[1285] The user follows the instructions displayed on the terminal, turning on the cooling system and setting the fan to maximum output as instructed.

[1286] 13. The user enters the device's response again and sends it.

[1287] The user checks the device status after the procedure is executed, enters a new response result into the terminal, and transmits it again, starting the cycle again, generating and presenting new procedures as necessary.

[1288] Specific examples

[1289] 1. The user inputs the device's response results.

[1290] Example: A user enters a temperature sensor reading of "85°C" into a terminal and presses the send button.

[1291] 2. The device sends the input response to the server.

[1292] Example: The terminal sends the data "85°C" in packet format to the server.

[1293] 3. The server analyzes the response

[1294] Example: The server parses the input data "85°C" and verifies that the expected normal value is "up to 70°C."

[1295] 4. The server checks for discrepancies

[1296] Example: The server determines that the temperature "85°C" is abnormal because it exceeds the set value "70°C."

[1297] 5. Investigate how the server resolves the discrepancy

[1298] Example: A server's AI investigates whether operating cooling equipment is an effective countermeasure when temperatures exceed the limit, and verifies the procedure for operating the cooling system.

[1299] 6. The server generates the next command and procedure to be executed

[1300] Example: The server generates the procedure "Turn on the cooling system and set the internal fan to maximum output."

[1301] 7. Recognize user emotions

[1302] Example: An emotion engine analyzes a user's facial expression and tone of voice and determines that the user is feeling anxious.

[1303] 8. The server adjusts procedures based on emotions

[1304] Example: The server notices user anxiety and adjusts the procedure by adding more detailed instructions.

[1305] 9. The server sends the generated instructions to the device

[1306] Example: The server sends the adjusted instructions to the terminal.

[1307] 10. The device presents instructions to the user

[1308] Example: The terminal displays "Please turn on the cooling system and set the internal fan to maximum power" along with detailed instructions.

[1309] This system allows users to receive appropriate response procedures in real time based on the device's response results, enabling them to respond quickly. Furthermore, by using an emotion engine, it is possible to present appropriate procedures according to the user's emotional state, improving work efficiency and safety.

[1310] The processing flow will be explained below.

[1311] Step 1:

[1312] The user inputs the response from the device. The user inputs the response obtained from the device into the input field of the terminal and presses the send button. For example, the user inputs that the temperature sensor reading is 85°C.

[1313] Step 2:

[1314] The terminal sends the input response result to the server. The terminal converts the data input by the user into a packet format and transmits it to the server.

[1315] Step 3:

[1316] The server analyzes the response it receives. The server passes the received data to the analysis module, which compares the "Temperature 85°C" with the set value "70°C." Based on this analysis result, the server determines that the temperature is too high.

[1317] Step 4:

[1318] The server checks for any discrepancies. Based on the analysis results, the server determines that the temperature "85°C" is "abnormal" because it exceeds the set value "70°C." The server records this information in a log and proceeds to the next step.

[1319] Step 5:

[1320] The server investigates ways to resolve the discrepancy. The server's AI module searches for the optimal way to resolve the discrepancy. For example, it determines that running the cooling system would be effective.

[1321] Step 6:

[1322] The server verifies the investigation results. The server verifies the results of the AI ​​module using simulations and past data to confirm their reliability. This confirms that the generated response measures are effective.

[1323] Step 7:

[1324] The server generates the next command and procedure to be performed. Based on the verification results, the server generates specific commands and procedures, such as "turn on the cooling system and set the fan to maximum output."

[1325] Step 8:

[1326] The emotion engine recognizes the user's emotions. The emotion engine uses the device's built-in camera and microphone to analyze the user's facial expressions and tone of voice to determine whether the user is feeling anxious or stressed.

[1327] Step 9:

[1328] The server adjusts the steps based on the user's emotions. The server adjusts the content of the steps and the level of detail of the explanation based on the user's emotional data obtained from the emotion engine. For example, if the server determines that the user is feeling anxious, it will provide more detailed explanations of the steps.

[1329] Step 10:

[1330] The server sends the generated procedure to the terminal, converts the adjusted procedure into a packet format, and transmits the data to the terminal.

[1331] Step 11:

[1332] The terminal presents the instructions to the user. The terminal displays the received instructions in an easy-to-read format for the user. For example, it displays specific instructions such as "Turn on the cooling system and set the internal fan to maximum output" along with detailed instructions.

[1333] Step 12:

[1334] The user follows the instructions displayed on the device, powering on the cooling system and setting the fans to maximum power.

[1335] Step 13:

[1336] The user again inputs the response result of the device and sends it. The user checks the device status after the procedure is executed, inputs the new response result into the terminal, and sends it again. This process regenerates the necessary procedure.

[1337] This series of processes allows users to receive appropriate response procedures in real time, enabling them to take action quickly.In addition, the emotion engine presents procedures that take the user's emotional state into consideration, allowing them to work efficiently while reducing stress and anxiety.

[1338] Example 2

[1339] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1340] In conventional systems, when generating the next steps to be taken based on the device's response results, a uniform procedure was often presented without considering the user's emotions. As a result, even when the user was in a situation where they felt anxious or stressed, appropriate follow-up was not provided, which could lead to a decrease in work efficiency and safety. Another issue was that the generated procedures were sometimes insufficiently verified to be effective, resulting in the inability to take effective countermeasures.

[1341] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1342] In this invention, the server includes means for inputting the response results of the devices, means for analyzing the response results and checking for deviations from a target procedure, means for investigating ways to resolve the deviations and generating a procedure to be performed next, means for displaying the generated procedure, means for recognizing the user's emotion, and means for adjusting the procedure based on the recognized emotion. This makes it possible to provide an appropriate procedure according to the user's emotional state, thereby improving work efficiency and safety.

[1343] "Device" is a general term for hardware or software that has a specific function and outputs the results of its operation.

[1344] "Response results" refer to data such as measurement values ​​and operation results obtained from a device.

[1345] "Input means" refers to an interface or device that allows a user to input the response results of an apparatus into the system.

[1346] "Means for analyzing" refers to software modules or algorithms for processing the input response results and analyzing their contents.

[1347] "Target procedure" refers to the ideal work procedure or set values ​​that are predetermined by the system.

[1348] "Deviation" refers to the difference or discrepancy between the actual response result and the target procedure or set value.

[1349] "Investigative measures" refers to methods and algorithms for finding optimal solutions to resolve discrepancies.

[1350] "Means for generating" refers to a software module that has the function of generating specific steps or commands to be performed next based on the results of the investigation.

[1351] "Means for displaying" refers to a display or user interface for informing the user of the generated procedures or commands.

[1352] "User" refers to a person who operates this system and whose role is to input the response results of the device and carry out work according to the generated procedure.

[1353] "Means for recognizing emotions" refers to software or hardware that analyzes a user's facial expressions, tone of voice, etc. to determine their emotional state.

[1354] "Adjustment means" refers to a software module that has the functionality to change or adjust the content of the procedures or commands presented based on the recognized emotions.

[1355] This invention is a system in which the user inputs the device's response results into a terminal, and the server generates and presents the next required command or procedure based on the results. By combining this process with an emotion engine that recognizes the user's emotions, we have created a system that supports appropriate responses according to the user.

[1356] System configuration

[1357] Hardware and software used

[1358] 1. Terminal

[1359] Hardware: PCs, tablets, smartphones, etc.

[1360] Software: Web browser, input interface, communication module

[1361] 2. Server

[1362] Hardware: High-performance server

[1363] Software: AI module, data analysis algorithm, emotion recognition engine, communication module

[1364] 3. Emotion Recognition Engine

[1365] Hardware: Camera, microphone

[1366] Software: Image analysis software, audio analysis software

[1367] System Operation

[1368] This system operates in the following specific steps:

[1369] 1. The user inputs the device's response results.

[1370] The user inputs the response obtained from the device into the terminal, for example, entering data such as the temperature read from the temperature sensor (85°C), and presses the "Send" button.

[1371] 2. The device sends the input response to the server.

[1372] The terminal sends the input data to the server in packet format, using protocols such as HTTP.

[1373] 3. The server analyzes the response

[1374] The server analyzes the received data and compares it with expected normal values ​​or set values, for example, a set value of "70°C" stored in a database, to determine whether there is any deviation.

[1375] 4. The server investigates how to resolve the discrepancy and generates a procedure

[1376] The server's AI module investigates the best way to resolve the discrepancy and generates the next steps to take, such as turning on the cooling system.

[1377] 5. The server verifies the findings

[1378] The server verifies whether the investigation results are valid using simulation tools and past data logs.

[1379] 6. The server generates and sends the next command and procedure to be performed.

[1380] For example, it generates specific instructions such as "turn on the cooling system and set the fan to maximum output" and sends them to the terminal.

[1381] 7. Recognize user emotions

[1382] The emotion recognition engine analyzes the user's facial expressions and tone of voice to recognize emotions such as anxiety and stress.

[1383] 8. The server adjusts procedures based on emotions

[1384] The generated instructions are adjusted based on the recognized emotion data, for example, providing more detailed instructions if the user is feeling anxious.

[1385] 9. The device presents instructions to the user

[1386] The terminal displays the procedure received from the server to the user in a format that is easy for the user to understand.

[1387] 10. The user follows the steps provided

[1388] The user follows the displayed instructions, for example, turning on the cooling system and setting the fans to maximum power.

[1389] Examples of concrete examples and prompts

[1390] For example, consider a situation where a user inputs a temperature sensor reading of "85°C" into a terminal, and the server generates and presents a procedure to "turn on the cooling system and set the fan to maximum output" based on that. In this case, the following prompt sentence is used:

[1391] Sample prompt: "Generate the next steps based on the following device response result: '85°C'. The user is feeling anxious."

[1392] This system allows users to receive appropriate response procedures in real time, enabling them to respond quickly.In addition, by using an emotion recognition engine, it provides appropriate procedures according to the user's emotional state, improving work efficiency and safety.

[1393] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1394] Step 1:

[1395] The user inputs the response result of the device.

[1396] The user inputs the response obtained from the device into the terminal. For example, the reading from the temperature sensor is "85°C." The input data is the raw data from the temperature sensor (85°C), and the output is a data packet containing this data.

[1397] Step 2:

[1398] The terminal sends the input response results to the server.

[1399] The terminal converts the input data into a packet format and sends it to the server. Specifically, it sends the data using the HTTP protocol. The input is a data packet from the user, and the output is a data packet sent to the server.

[1400] Step 3:

[1401] The server analyzes the response

[1402] The server analyzes the received data (85°C). Specifically, it compares it with the set value "70°C" stored in the database. The input is the raw data (85°C) sent from the terminal, and the output is the analysis result: "85°C exceeds the set value of 70°C."

[1403] Step 4:

[1404] The server checks for discrepancies

[1405] Based on the analysis results, the server determines that the actual temperature exceeds the set value, indicating an abnormality. The input is the analysis result, and the output is the judgment on whether or not there is an abnormality (abnormality).

[1406] Step 5:

[1407] Investigate how the server resolves the discrepancy

[1408] The server's AI module runs and investigates the optimal solution. For example, it suggests turning on the cooling system. The input is the anomaly detection result, and the output is a candidate for the optimal solution (turning on the cooling system).

[1409] Step 6:

[1410] The server verifies the findings

[1411] The server verifies the effectiveness of the investigation results using simulations and past data logs. Specifically, it uses a simulation tool. The input is the candidate optimal countermeasures, and the output is the verification result (validity).

[1412] Step 7:

[1413] The server generates the next command and steps to be executed.

[1414] Based on the verification results, the server generates specific steps to be taken next, such as "turn on the cooling system and set the fan to maximum output." The input is the verification results, and the output is the specific steps.

[1415] Step 8:

[1416] Recognize user emotions

[1417] The emotion recognition engine analyzes the user's facial expressions and tone of voice to determine whether the user is feeling anxious or stressed. Specifically, it uses a camera and microphone. The input is the user's real-time voice and video data, and the output is the user's emotional state (anxiety).

[1418] Step 9:

[1419] The server adjusts the procedure based on emotions.

[1420] The server adjusts the detail and explanation of the procedure based on the emotional data. For example, if the user feels anxious, the server will provide more detailed instructions. The input is the emotional state data, and the output is the adjusted procedure.

[1421] Step 10:

[1422] The server sends the generated instructions to the device.

[1423] The server sends the generated instructions to the terminal in the form of packets, where the input is the adjusted instructions and the output is the data packets sent to the terminal.

[1424] Step 11:

[1425] The terminal presents instructions to the user

[1426] The terminal displays the instructions received from the server to the user. Specifically, it displays on the screen, "Please turn on the cooling system and set the fan to maximum output." The input is the instructions sent from the server, and the output is the display to the user.

[1427] Step 12:

[1428] The user follows the steps provided

[1429] The user follows the instructions displayed on the terminal to perform physical and software tasks, such as turning on the power to a cooling system and setting the internal fan to maximum output. The input is the displayed instructions, and the output is the result of executing the instructions.

[1430] Step 13:

[1431] The user re-enters the device's response and sends it.

[1432] The user checks the device status after the procedure is executed and inputs a new response result (e.g., temperature "75°C") into the terminal. Then, the user presses the "Send" button again to send the data to the server. The input is the new response result, and the output is the resent data packet.

[1433] (Application example 2)

[1434] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1435] While conventional device monitoring systems can generate procedures based on device response results, they are unable to adjust procedures to take into account the user's emotional state. This has led to issues such as users becoming stressed, especially in emergency situations or complex situations, and performance degradation due to operational errors and tension. This can compromise efficiency and safety in scenarios that require rapid and accurate responses.

[1436] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes a means for inputting the device's response result, a means for analyzing the response result and checking for deviation from a target procedure, a means for investigating a method for eliminating the deviation and generating a procedure to be performed next, a means for recognizing the user's emotion, and a means for adjusting the procedure based on the recognized user emotion. This makes it possible to generate an appropriate procedure based on the device's response result and further provide detailed guidance according to the user's emotion. This reduces user stress and enables a quick and accurate response.

[1437] "Device response results" are data or output information generated by a device during operation.

[1438] A "means" is a method, process, or device used to accomplish a particular purpose.

[1439] "Target procedures" are procedures or standard values ​​that are set in advance to achieve normal operation or a goal.

[1440] "Deviation" means a deviation or abnormal variation from an expected normal range or set point.

[1441] "Research" is the process of collecting and analyzing information to find appropriate methods and procedures for solving a problem.

[1442] "Generation" is the act of creating new procedures or commands based on specific conditions or data.

[1443] "Display" refers to the act of visually presenting information to a user.

[1444] "Emotion recognition" is a technology that analyzes a user's facial expressions, tone of voice, etc. to determine their emotional state.

[1445] "Adjustment" is the act of changing or modifying procedures or instructions to suit specific conditions or circumstances.

[1446] A "system" is a collection of integrated devices or processes in which multiple elements or means work together to perform a single function.

[1447] "Device" refers to an electronic device or terminal used for a specific purpose.

[1448] A "server" is a device that stores and processes data on a network and provides services to other devices and users.

[1449] "Input of response results" refers to the act of a user manually or automatically inputting data or results output by a device into the system.

[1450] "Data analysis" is the process of analyzing collected data to find meaningful information and patterns.

[1451] "Emotion-based procedure adjustment" refers to the act of appropriately changing the procedures and guidance presented to the user in consideration of the user's emotional state.

[1452] This invention relates to a system that generates and presents the next steps required based on the response results of the device, and also recognizes and adjusts the user's emotions. This system is intended to monitor and manage devices and robots used primarily in factories.

[1453] The system includes the following major components:

[1454] 1. User Device

[1455] The user device inputs the response results from the device and sends the data to the server. The device is typically a smartphone or tablet, and is equipped with a camera and microphone. This allows it to recognize the user's emotions.

[1456] 2. Server

[1457] The server is the central processing unit that analyzes the data, generates procedures as needed, and presents them to the user. The server includes the following functions:

[1458] Data Analysis Module

[1459] The server receives and analyzes the device response results sent from the user terminal, checks for deviations from the normal range, and investigates and determines the optimal method to resolve the deviations.

[1460] Procedure Generation Module

[1461] Based on the analysis results, the next steps to be taken are generated and verified to see if they are effective.

[1462] Emotion Recognition Module

[1463] It analyzes the video and audio data sent from the user's device to determine the user's emotional state (anxiety, stress, etc.).

[1464] Procedure Coordination Module

[1465] Based on the results of the emotion recognition module, the generated instructions are adjusted, specifically determining whether to provide detailed or concise instructions depending on the emotional state.

[1466] 3. Presentation to the User

[1467] The generated procedure is again transmitted to the user terminal and visually presented to the user.

[1468] As a concrete example, consider a situation where a temperature sensor in a factory detects 85°C, which exceeds the set value of 70°C. The user enters this reading into a terminal and sends it to the server. The server analyzes this data and determines that the cooling fan needs to be turned on. Furthermore, if the emotion recognition module detects the user's anxiety, it adjusts the procedure to "Turn on the cooling system and set the internal fan to maximum output. If you have any questions during the procedure, please refer to the manual."

[1469] The main software and hardware used are cloud computing platforms (e.g., AWS, Google Cloud) for the server, and smartphones (compatible with iOS or Android) for the user devices. The EmotionRecognition library is used for emotion recognition.

[1470] A specific example of a prompt for a generative AI model is, "The temperature sensor detected 85°C. This is an abnormal condition, exceeding the set value (70°C). Please generate the following steps. If the user is feeling uneasy, please explain the steps in detail." Using this prompt, the AI ​​can generate optimal steps and provide appropriate guidance to the user.

[1471] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1472] Step 1:

[1473] The user inputs the device's response result. The user inputs the device's response result (e.g., the temperature sensor reading "85°C") into a smartphone or tablet and presses the send button. The input data is sent to the device in text format.

[1474] (Input) Device response result (temperature "85°C")

[1475] (Output) Response result data sent from the user terminal

[1476] Step 2:

[1477] The device sends the input response results to the server. The device converts the sent data into packet format and sends it to the cloud server. At that time, security protocols (e.g., SSL / TLS) are used to maintain the integrity of the data.

[1478] (Input) Response result data entered by the user (temperature "85°C")

[1479] (Output) Response result data sent to the cloud server

[1480] Step 3:

[1481] The server analyzes the response results. The server analyzes the received data and checks for deviations from normal values ​​or set values ​​(e.g., "70°C"). The data analysis module does this and detects abnormalities.

[1482] (Input) Response result data sent to the cloud server (temperature "85°C")

[1483] (Output) Deviation judgment result (temperature "85°C" is abnormal)

[1484] Step 4:

[1485] The server investigates ways to resolve the discrepancy and generates the next steps to be taken. An AI module then runs to find the optimal solution based on past data and simulations (e.g., running a cooling system).

[1486] (Input) Deviation judgment result (temperature "85°C" is abnormal)

[1487] (Output) Initial action steps (e.g., turning on cooling system)

[1488] Step 5:

[1489] The server recognizes the user's emotions. Camera footage and audio data are sent from the user's device, and the emotion recognition module analyzes them to determine whether the user is feeling anxious.

[1490] (Input) User's video and audio data

[1491] (Output) Emotional state (e.g., anxiety)

[1492] Step 6:

[1493] The server adjusts the instructions based on the emotion. Based on the emotion recognition results, the server adjusts the instructions it presents (e.g., explains the instructions in detail).

[1494] (Input) Initial response steps (turn on cooling system), emotional state (anxiety)

[1495] (Output) Adjusted countermeasure steps (with detailed explanation)

[1496] Step 7:

[1497] The server sends the generated procedure to the terminal, converts the generated procedure into a packet format, and sends it to the user's terminal.

[1498] (Input) Adjusted countermeasure procedures (with detailed explanations)

[1499] (Output) Instructions sent to the user terminal

[1500] Step 8:

[1501] The terminal presents instructions to the user. The terminal visually displays the received instructions and presents them in a format that is easy for the user to understand (e.g., "Please turn on the cooling system and set the internal fan to maximum output. If you have any questions during the procedure, please refer to the manual.").

[1502] (Input) Procedure data received from the server

[1503] (Output) Instructions displayed to the user

[1504] Step 9:

[1505] The user carries out the presented procedure. The user performs the work according to the instructions displayed on the terminal.

[1506] (Input) Instructions displayed to the user

[1507] (Output) User execution of steps

[1508] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1509] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1510] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1511] [Fourth embodiment]

[1512] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1513] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1514] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1515] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1516] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1517] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1518] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1519] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1520] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1521] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1522] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1523] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1524] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1525] This invention is a system that inputs the response results of a device and generates and presents the next required commands and procedures based on them. Below, the program processing of this system is explained in natural language and in detail with concrete examples.

[1526] Program Processing Overview

[1527] 1. The user inputs the device response results into the server.

[1528] The user inputs the response results obtained from the device into the terminal while working. For example, when the user inputs the reading of a temperature sensor device, the user inputs that the temperature is "85°C" into the terminal.

[1529] The terminal transmits this input data to the server.

[1530] 2. The server analyzes the response

[1531] The server analyzes the response and checks whether it deviates from the set value or the expected normal value. For example, the server detects that the temperature "85°C" exceeds the set value "70°C."

[1532] The server records logs based on the analysis results and stores them for future analysis.

[1533] 3. The server checks for discrepancies

[1534] Based on the analysis results, the server determines whether the response deviates from the target procedure. The server determines that the temperature of 85°C is abnormal because it exceeds the normal range of 70°C.

[1535] Based on the result of this determination, the next step is to determine the necessary action.

[1536] 4. Investigate how the server resolves the discrepancy

[1537] The server's AI module investigates the best way to resolve the discrepancy. If the temperature exceeds the set point, the server's AI determines that activating the cooling system would be effective.

[1538] Validate the findings and generate specific implementation steps.

[1539] 5. The server generates the next command and steps to be executed

[1540] Based on the results of the investigation and verification, the server generates the next command and procedure to be performed, such as "switch on the cooling device" or "restart the device."

[1541] The generated procedure is formatted so that it is easy for the user to understand, and is sent to the terminal.

[1542] 6. Present the procedure for creating the device to the user

[1543] The device will then display the new instructions received from the server to the user, for example, "Please turn on the cooling device."

[1544] After the user has performed the procedure, he is prompted to enter the device's response again.

[1545] Specific examples

[1546] 1. The user inputs the device response results into the server.

[1547] Example: A user enters a temperature sensor reading of "85°C" into a terminal and presses the send button.

[1548] 2. The server analyzes the response

[1549] Example: The server analyzes the input data "85°C" and verifies that the expected normal value is "up to 70°C." The server records this information in a log and determines that there is a discrepancy.

[1550] 3. The server checks for discrepancies

[1551] Example: The server determines that the temperature "85°C" is abnormal because it exceeds the set value "70°C."

[1552] 4. Investigate how the server resolves the discrepancy

[1553] Example: A server's AI investigates whether operating cooling equipment is an effective countermeasure when temperatures exceed the limit, and verifies the procedure for operating the cooling system.

[1554] 5. The server generates the next command and steps to be executed

[1555] Example: The server generates the procedure "Turn on the cooling system and set the internal fan to maximum output."

[1556] 6. Present the procedure for creating the device to the user

[1557] Example: The terminal will prompt the user to "Please turn on the cooling system and set the internal fan to maximum power." After the user performs the steps, they will be prompted to enter the device's response again.

[1558] This system allows users to receive appropriate response procedures in real time based on the device's response results, enabling them to respond quickly, preventing work interruptions and realizing efficient and flexible work.

[1559] The processing flow will be explained below.

[1560] Step 1:

[1561] The user inputs the device response results into the terminal. The user inputs the device response results obtained during work into the terminal's input field and presses the send button. For example, the user inputs the reading from the temperature sensor, "85°C."

[1562] Step 2:

[1563] The terminal transmits the input response result to the server. The terminal converts the data input by the user into a packet format and transmits it to the server.

[1564] Step 3:

[1565] The server analyzes the response it receives. The server passes the received data to the analysis module, which analyzes the contents of the response. For example, the server detects that the temperature "85°C" deviates from the set value "70°C."

[1566] Step 4:

[1567] The server checks for any deviations. Based on the analysis results, the server determines whether there is any deviation from the target procedure. Because the temperature "85°C" exceeds the normal range of "70°C," the server determines that there is an "abnormality."

[1568] Step 5:

[1569] The server investigates ways to resolve the discrepancy. The server's AI module runs and investigates the best way to resolve the discrepancy. For example, it determines that turning on the cooling system is the best option.

[1570] Step 6:

[1571] The server verifies the investigation results. Based on the investigation results, the server verifies whether the generated response method is effective using simulations and past data.

[1572] Step 7:

[1573] The server generates the next command and procedure to be performed. Based on the verified method, the server generates the specific command and procedure to be performed next. For example, it creates a specific procedure such as "turn on the cooling system and set the fan to maximum output."

[1574] Step 8:

[1575] The server transmits the generated procedure to the terminal, converts the generated procedure into a packet format, and transmits it to the terminal.

[1576] Step 9:

[1577] The terminal presents the instructions to the user. The terminal displays the received instructions in an easy-to-read format for the user. For example, it displays "Please turn on the cooling system and set the internal fan to maximum output."

[1578] Step 10:

[1579] The user follows the instructions displayed on the device, powering on the cooling system and setting the fans to maximum power as instructed.

[1580] Step 11:

[1581] The user again inputs the device's response result into the terminal and sends it. The user checks the device status after the procedure is executed, inputs a new response result into the terminal, and sends it again.

[1582] Step 12:

[1583] The device sends a new response to the server. The device converts the received data back into packets and sends them to the server. The cycle begins again, generating and submitting new instructions as needed.

[1584] Example 1

[1585] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1586] Conventionally, the generation and presentation of procedures and commands based on device response results has often been done manually, resulting in problems such as reduced work efficiency and accuracy. Furthermore, manual analysis and response takes time, making it difficult to respond in real time. The objective of this invention is to provide a system that automatically analyzes device response results and quickly and accurately presents the necessary procedures and commands to the user.

[1587] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1588] In this invention, the server includes a means for inputting the response results of the device, a means for analyzing the response results and checking for deviations from the target procedure, and a means for investigating a method for eliminating the deviation using a generative AI model and generating the procedure to be performed next, thereby enabling the user to receive the appropriate response procedure in real time and respond quickly.

[1589] The "means for inputting the response results of the device" is a function that allows the user to input data or information obtained from the device into a terminal or the like and transmit that data to the server.

[1590] "Means of analysis" refers to the process by which the server compares the data it receives with set values ​​and expected normal values ​​to determine any deviations or abnormalities in the data.

[1591] The "means for checking deviation from the target procedure" is a function that determines, based on the analyzed data, how much the actual results deviate from the set target procedure.

[1592] "Means of investigating using a generative AI model and generating the next steps to be taken" refers to the process of using an AI module in the server to investigate the optimal method for resolving any discrepancies or anomalies that have occurred and generate specific response steps.

[1593] The "means for displaying procedures" is a function that visualizes and presents the generated response procedures and commands to the user.

[1594] "Means for inputting to a data management device" refers to an interface or operating procedure for inputting and sending the generated procedures and commands to a data management device such as a server.

[1595] The "means for generating specific operational instructions" is a function that generates specific commands and operational procedures to be executed for a specific device or system based on the analysis results.

[1596] The "means for verifying whether the operation instruction is valid" is a process for checking and confirming whether the generated specific operation instruction actually has an effect.

[1597] The present invention is a system that inputs a device response result and generates and presents the next required command or procedure based on that result. The system includes a means for inputting the device response result, a means for analyzing it, a means for checking for discrepancies, a means for generating the next procedure using a generative AI model, and a means for displaying the procedure.

[1598] Hardware and software used

[1599] Device: The computing device (smartphone, tablet, PC) operated by the user.

[1600] Server: A server that analyzes data, records logs, generates next steps, etc. It is built using a database or cloud infrastructure (e.g., Amazon Web Services or Microsoft Azure).

[1601] Generative AI model: An AI module (e.g., TensorFlow, PyTorch, etc.) used for data analysis and procedure generation.

[1602] Natural language explanation of program processing

[1603] 1. The user enters the device's response into the terminal and sends it to the server.

[1604] The user inputs the response obtained from the device into the terminal. For example, the user inputs "85°C" as the temperature sensor reading.

[1605] The terminal transmits the input data to the server.

[1606] 2. The server receives the response and analyzes the data.

[1607] The server receives the data sent from the device and analyzes it by comparing it with the set value and expected normal value. It checks whether the temperature "85°C" exceeds the set value "70°C".

[1608] The server records the analysis results as a log and detects any deviations.

[1609] 3. The server determines whether there is a discrepancy

[1610] Based on the analysis results, the server determines how much the input data deviates from the set value. Since the temperature "85°C" exceeds the set value "70°C," it is judged to be "abnormal."

[1611] Based on this decision, the server determines the next action to be taken.

[1612] 4. The server uses a generative AI model to investigate and select a method to resolve the discrepancy.

[1613] The server uses a generative AI model to investigate the best way to resolve the anomaly. If the temperature is high, the investigation confirms that turning on a cooling device would be beneficial.

[1614] Generate specific steps based on the survey results.

[1615] 5. The server generates the next steps to be taken and sends them to the device.

[1616] The server then formats the generated instructions into a format that is easy for the user to understand, such as "Please turn on the cooling device."

[1617] This procedure is sent to the terminal.

[1618] 6. Present the procedure for creating the device to the user

[1619] The device displays the instructions received from the server to the user, for example, "Please turn on the cooling device."

[1620] The user acts according to the presented procedure and inputs the response results of the device again.

[1621] Examples of specific examples and prompts

[1622] Specific examples

[1623] 1. The user inputs the device's response results.

[1624] Example: A user enters the temperature sensor reading "85°C" into the terminal input form and presses the "Submit" button.

[1625] 2. The server analyzes the response

[1626] Example: The server receives the data "85°C" and compares it with the set value "70°C" to detect an abnormality.

[1627] 3. The server checks for discrepancies

[1628] Example: The server temperature "85°C" exceeds the set value "70°C", so it is logged as an abnormality.

[1629] 4. Investigate how the server resolves the discrepancy

[1630] Example: Using a generative AI model, determine that running a cooling unit is the optimal solution.

[1631] 5. The server generates the next steps to be taken

[1632] Example: The server generates the procedure "Switch on the cooling device and set the internal fan to maximum power."

[1633] 6. Present the procedure for creating the device to the user

[1634] Example: The device displays to the user, "Please turn on the cooling device."

[1635] Prompt Sentence Examples

[1636] Generate a procedure for what to do if the temperature is too high.

[1637] Generate instructions to set the cooling system to maximum power.

[1638] Please indicate the procedure to be followed in response to an abnormal value of 85°C.

[1639] As described above, the present invention aims to provide the user with an appropriate response procedure in real time based on the response results of the device, thereby realizing a quick response.

[1640] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1641] Step 1:

[1642] The user inputs the device's response into the terminal and sends it to the server.

[1643] Specific operation: The user enters the response obtained from the device, for example, the temperature sensor reading "85°C," into the terminal and presses the send button.

[1644] Input: Temperature sensor reading "85°C".

[1645] Output: Sending data from the device to the server.

[1646] Step 2:

[1647] The server receives the response result

[1648] Specific operation: The server receives data sent from the device via HTTP request or WebSocket.

[1649] Input: Temperature data sent from the device: "85°C".

[1650] Output: The temperature data received by the server.

[1651] Step 3:

[1652] The server analyzes the response

[1653] Specific operation: The server compares the received data with the set value or expected normal value to check for deviations. For example, it checks whether the temperature "85°C" exceeds the set value "70°C".

[1654] Input: Temperature data received by the server is "85°C", set value is "70°C".

[1655] Output: Anomaly detection results.

[1656] Step 4:

[1657] The server determines whether there is a discrepancy

[1658] Specific operation: The server determines how much the temperature deviates from the set value based on the analysis results. Since the temperature "85°C" exceeds the set value "70°C", it is determined to be abnormal.

[1659] Input: Anomaly detection results.

[1660] Output: Log of abnormality detection.

[1661] Step 5:

[1662] The server uses generative AI models to explore solutions

[1663] Specific behavior: The generative AI model in the server investigates the best way to resolve the anomaly, for example, investigating whether running a cooling system is effective.

[1664] Input: Abnormality determination log, abnormal value "85°C".

[1665] Output: The optimal solution and its validation results.

[1666] Step 6:

[1667] The server generates the next steps and sends them to the device.

[1668] Specific operation: The server formats the generated instructions into a format that is easy for the user to understand and sends it to the terminal. For example, it creates and sends an instruction such as "Please turn on the cooling device."

[1669] Input: Validation results of the optimal solution.

[1670] Output: Generates specific operating instructions and sends them to the terminal.

[1671] Step 7:

[1672] Present the procedure for the device to be generated to the user

[1673] Specific behavior: The device displays the instructions sent from the server to the user, such as "Please turn on the cooling device."

[1674] Input: Specific operating instructions received from the server.

[1675] Output: The instructions presented to the user.

[1676] Step 8:

[1677] The user performs the procedure and enters the response again.

[1678] Specific operation: The user follows the presented procedure, for example, turning on the cooling device, and then inputs the device's response again.

[1679] Input: Procedures presented by the server, and the results of the user's execution.

[1680] Output: The response result of the input device again.

[1681] (Application example 1)

[1682] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1683] Picking work at logistics centers requires a lot of manpower and time, so there is a need to improve work efficiency. It is also important to quickly correct errors that occur during work and achieve accurate picking. In response to these challenges, the present invention provides a system that enables picking robots to receive optimal work procedures in real time and perform work efficiently and accurately.

[1684] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1685] In this invention, the server includes means for recalculating the picking procedure for the logistics robot based on the response result of the device and generating a corrected command, means for displaying the generated procedure to the user and inputting the response result to the server, means for presenting an appropriate procedure in real time in the logistics process, means for generating specific picking commands, and means for verifying whether the generated commands are accurate and optimal, thereby enabling improved efficiency and accuracy of picking work in logistics centers.

[1686] "Device response result" refers to the result information that a device returns in response to some input or operation.

[1687] A "procedure" refers to the sequence of tasks or operations that are used to achieve a particular purpose.

[1688] A "server" is a computer system that stores, manages, and processes data over a network.

[1689] A "logistics robot" is a robot that automates the movement and picking of goods at logistics centers and other locations.

[1690] "Real-time" refers to a method in which data input, processing, and response occur within an extremely short period of time.

[1691] "Picking procedure" refers to the order and method for removing specified products from shelves at a logistics center.

[1692] A "command" refers to an instruction that instructs a computer or robot to perform a specific action.

[1693] "Analysis" refers to the scientific process of breaking down data to understand its structure and relationships.

[1694] Deviation refers to a situation in which actual procedures or results deviate from goals or expectations.

[1695] "User" refers to a person who operates a system or device and uses it to achieve a purpose.

[1696] A "process" is a sequence of tasks or steps to achieve a specific goal.

[1697] "Generation" refers to the act of creating new data or procedures.

[1698] "Verification" refers to the process of verifying the accuracy of the results and commands generated.

[1699] This invention is a system that inputs the response results of a device and generates and presents the next necessary commands and procedures based on them. This system is particularly useful for operating picking robots in logistics centers.

[1700] System Program

[1701] A system for a picking robot used in a logistics center consists of the following elements:

[1702] Hardware: Picking robot, smartphone / tablet

[1703] Software: Robot control software, cloud-based AI analysis tools (e.g., AWS SageMaker), databases (e.g., AWS RDS)

[1704] Program processing description

[1705] Input of device response results

[1706] Users input the barcode information read by the robot and their current location information using their smartphone or tablet, and this data is sent to a cloud server.

[1707] Analysis of response results

[1708] The server analyzes the response and compares it with the product information and picking procedures in the database. For example, if an item is picked from the wrong shelf or the picking order is incorrect, the information is checked against the database.

[1709] Check for deviations

[1710] Based on the analysis results, the server determines whether the input response results deviate from the set target procedure. If a deviation is found, it is deemed to be a deviation.

[1711] Investigating ways to resolve discrepancies

[1712] The AI ​​analysis tool investigates and recalculates the optimal method to eliminate the discrepancy. Specifically, it recalculates and proposes the correct picking order and procedures.

[1713] Generate and view steps

[1714] The server generates specific steps and commands to resolve the discrepancy and displays them on the tablet or smartphone device, instructing the user or robot on the next action to be taken.

[1715] Specific examples

[1716] Data entry examples

[1717] The user enters the barcode information of the product "Product 12345" that the picking robot reads with its sensor into the terminal along with the user's current location "Section A" and sends it to the cloud server.

[1718] Specific examples of data analysis

[1719] The server checks the database to see that the received "item 12345" should be in section B, and determines that it was picked from section A by mistake.

[1720] Specific examples of checking discrepancies

[1721] The server determines that "picking from section A" is a deviation and records this.

[1722] Specific examples of methods for eliminating discrepancies

[1723] The AI ​​analysis tool recalculates the correct picking order and suggests the correct procedure: "Pick item 12345 from section B."

[1724] Example of procedure generation and display

[1725] The server generates instructions such as "Please pick product 12345 from section B" and displays them on the smartphone / tablet.

[1726] Prompt Sentence Examples

[1727] Prompt statement:

[1728] "Which section is product ID "12345" currently in and what is the correct picking procedure?"

[1729] This enables picking operations at logistics centers to be carried out efficiently and accurately.

[1730] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1731] Step 1:

[1732] The user inputs the barcode information read by the picking robot and their current location information on their smartphone or tablet. For example, they input the product ID "12345" and their current location "Section A," and send them to the cloud server. This input is done using a dedicated input form provided by the device.

[1733] Input: Product ID "12345", current location "Section A"

[1734] Output: Sending input data to the cloud server

[1735] Step 2:

[1736] The server analyzes the response received on the cloud (product ID "12345" and current location "Section A") and compares this information with the product information and picking procedures in the database.

[1737] Input: Product ID "12345", current location "Section A"

[1738] Output: Matching result (correct position or deviation information)

[1739] Step 3:

[1740] The server determines whether the input response result deviates from the set target procedure based on the collation result. If so, it determines that there is a deviation and records that information.

[1741] Input: Matching result

[1742] Output: Check result for discrepancy

[1743] Step 4:

[1744] The server's AI analysis tool investigates the optimal method for resolving the discrepancy. Specifically, it recalculates the correct picking order and procedures. For example, it recalculates the procedure for picking "product 12345" from the correct location (section B).

[1745] Input: Deviation check result

[1746] Output: Recalculation result of correct procedure

[1747] Step 5:

[1748] The server generates specific instructions and commands, such as "Please pick product 12345 from section B."

[1749] Input: Recalculated steps

[1750] Output: The specific steps or commands generated

[1751] Step 6:

[1752] The terminal displays the new instructions received from the server to the user. For example, the smartphone or tablet screen displays, "Please pick product 12345 from section B."

[1753] Input: The specific steps or commands generated

[1754] Output: On-screen instructions

[1755] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1756] This invention combines a system that inputs the response results of a device and generates and presents the next necessary commands and procedures based on that, with an emotion engine that recognizes the user's emotions. Below, the program processing of this system is explained in natural language and in detail with concrete examples.

[1757] Program Processing Overview

[1758] 1. The user inputs the device's response results.

[1759] The user inputs the response results obtained from the device into the terminal while working. For example, when the user inputs the reading of a temperature sensor device, the user inputs that the temperature is "85°C" into the terminal.

[1760] 2. The device sends the input response to the server.

[1761] The terminal sends this input data to the server, which converts the input data into a packet format and sends it to the server.

[1762] 3. The server analyzes the response

[1763] The server analyzes the response and checks whether it deviates from the set value or the expected normal value. For example, the server detects that the temperature "85°C" deviates from the set value "70°C".

[1764] 4. The server checks for discrepancies

[1765] The server uses the analysis results to determine whether there is a deviation from the target procedure. Because the temperature "85°C" exceeds the normal range of "70°C," the server determines that there is an "abnormality."

[1766] 5. Investigate how the server resolves the discrepancy

[1767] The server's AI module then goes to work and investigates the best way to resolve the discrepancy. For example, it may determine that turning on the cooling system is the best solution.

[1768] 6. The server verifies the findings

[1769] Based on the investigation results, the server verifies whether the generated response method is effective using simulations and past data.

[1770] 7. The server generates the next command and steps to be executed

[1771] Based on the verified method, the server generates specific commands and steps to be taken next, such as "turn on the cooling system and set the fans to maximum power."

[1772] 8. Recognize user emotions

[1773] The emotion engine recognizes the user's emotions, for example by using a camera and microphone to analyze the user's facial expressions and tone of voice to determine whether the user is feeling anxious or stressed.

[1774] 9. The server adjusts procedures based on emotions

[1775] The server adjusts the presented instructions based on the user's emotional data obtained from the emotion engine. For example, if the user feels anxious, the instructions may be made more detailed or more concise.

[1776] 10. The server sends the generated instructions to the device

[1777] The server converts the generated procedure into a packet format and transmits it to the terminal.

[1778] 11. The device presents instructions to the user

[1779] The device then displays the received instructions in a user-friendly format, for example, "Please turn on the cooling system and set the internal fan to maximum output."

[1780] 12. The user follows the steps provided

[1781] The user follows the instructions displayed on the terminal, turning on the cooling system and setting the fan to maximum output as instructed.

[1782] 13. The user enters the device's response again and sends it.

[1783] The user checks the device status after the procedure is executed, enters a new response result into the terminal, and transmits it again, starting the cycle again, generating and presenting new procedures as necessary.

[1784] Specific examples

[1785] 1. The user inputs the device's response results.

[1786] Example: A user enters a temperature sensor reading of "85°C" into a terminal and presses the send button.

[1787] 2. The device sends the input response to the server.

[1788] Example: The terminal sends the data "85°C" in packet format to the server.

[1789] 3. The server analyzes the response

[1790] Example: The server parses the input data "85°C" and verifies that the expected normal value is "up to 70°C."

[1791] 4. The server checks for discrepancies

[1792] Example: The server determines that the temperature "85°C" is abnormal because it exceeds the set value "70°C."

[1793] 5. Investigate how the server resolves the discrepancy

[1794] Example: A server's AI investigates whether operating cooling equipment is an effective countermeasure when temperatures exceed the limit, and verifies the procedure for operating the cooling system.

[1795] 6. The server generates the next command and procedure to be executed

[1796] Example: The server generates the procedure "Turn on the cooling system and set the internal fan to maximum output."

[1797] 7. Recognize user emotions

[1798] Example: An emotion engine analyzes a user's facial expression and tone of voice and determines that the user is feeling anxious.

[1799] 8. The server adjusts procedures based on emotions

[1800] Example: The server notices user anxiety and adjusts the procedure by adding more detailed instructions.

[1801] 9. The server sends the generated instructions to the device

[1802] Example: The server sends the adjusted instructions to the terminal.

[1803] 10. The device presents instructions to the user

[1804] Example: The terminal displays "Please turn on the cooling system and set the internal fan to maximum power" along with detailed instructions.

[1805] This system allows users to receive appropriate response procedures in real time based on the device's response results, enabling them to respond quickly. Furthermore, by using an emotion engine, it is possible to present appropriate procedures according to the user's emotional state, improving work efficiency and safety.

[1806] The processing flow will be explained below.

[1807] Step 1:

[1808] The user inputs the response from the device. The user inputs the response obtained from the device into the input field of the terminal and presses the send button. For example, the user inputs that the temperature sensor reading is 85°C.

[1809] Step 2:

[1810] The terminal sends the input response result to the server. The terminal converts the data input by the user into a packet format and transmits it to the server.

[1811] Step 3:

[1812] The server analyzes the response it receives. The server passes the received data to the analysis module, which compares the "Temperature 85°C" with the set value "70°C." Based on this analysis result, the server determines that the temperature is too high.

[1813] Step 4:

[1814] The server checks for any discrepancies. Based on the analysis results, the server determines that the temperature "85°C" is "abnormal" because it exceeds the set value "70°C." The server records this information in a log and proceeds to the next step.

[1815] Step 5:

[1816] The server investigates ways to resolve the discrepancy. The server's AI module searches for the optimal way to resolve the discrepancy. For example, it determines that running the cooling system would be effective.

[1817] Step 6:

[1818] The server verifies the investigation results. The server verifies the results of the AI ​​module using simulations and past data to confirm their reliability. This confirms that the generated response measures are effective.

[1819] Step 7:

[1820] The server generates the next command and procedure to be performed. Based on the verification results, the server generates specific commands and procedures, such as "turn on the cooling system and set the fan to maximum output."

[1821] Step 8:

[1822] The emotion engine recognizes the user's emotions. The emotion engine uses the device's built-in camera and microphone to analyze the user's facial expressions and tone of voice to determine whether the user is feeling anxious or stressed.

[1823] Step 9:

[1824] The server adjusts the steps based on the user's emotions. The server adjusts the content of the steps and the level of detail of the explanation based on the user's emotional data obtained from the emotion engine. For example, if the server determines that the user is feeling anxious, it will provide more detailed explanations of the steps.

[1825] Step 10:

[1826] The server sends the generated procedure to the terminal, converts the adjusted procedure into a packet format, and transmits the data to the terminal.

[1827] Step 11:

[1828] The terminal presents the instructions to the user. The terminal displays the received instructions in an easy-to-read format for the user. For example, it displays specific instructions such as "Turn on the cooling system and set the internal fan to maximum output" along with detailed instructions.

[1829] Step 12:

[1830] The user follows the instructions displayed on the device, powering on the cooling system and setting the fans to maximum power.

[1831] Step 13:

[1832] The user again inputs the response result of the device and sends it. The user checks the device status after the procedure is executed, inputs the new response result into the terminal, and sends it again. This process regenerates the necessary procedure.

[1833] This series of processes allows users to receive appropriate response procedures in real time, enabling them to take action quickly.In addition, the emotion engine presents procedures that take the user's emotional state into consideration, allowing them to work efficiently while reducing stress and anxiety.

[1834] Example 2

[1835] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1836] In conventional systems, when generating the next steps to be taken based on the device's response results, a uniform procedure was often presented without considering the user's emotions. As a result, even when the user was in a situation where they felt anxious or stressed, appropriate follow-up was not provided, which could lead to a decrease in work efficiency and safety. Another issue was that the generated procedures were sometimes insufficiently verified to be effective, resulting in the inability to take effective countermeasures.

[1837] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1838] In this invention, the server includes means for inputting the response results of the devices, means for analyzing the response results and checking for deviations from a target procedure, means for investigating ways to resolve the deviations and generating a procedure to be performed next, means for displaying the generated procedure, means for recognizing the user's emotion, and means for adjusting the procedure based on the recognized emotion. This makes it possible to provide an appropriate procedure according to the user's emotional state, thereby improving work efficiency and safety.

[1839] "Device" is a general term for hardware or software that has a specific function and outputs the results of its operation.

[1840] "Response results" refer to data such as measurement values ​​and operation results obtained from a device.

[1841] "Input means" refers to an interface or device that allows a user to input the response results of an apparatus into the system.

[1842] "Means for analyzing" refers to software modules or algorithms for processing the input response results and analyzing their contents.

[1843] "Target procedure" refers to the ideal work procedure or set values ​​that are predetermined by the system.

[1844] "Deviation" refers to the difference or discrepancy between the actual response result and the target procedure or set value.

[1845] "Investigative measures" refers to methods and algorithms for finding optimal solutions to resolve discrepancies.

[1846] "Means for generating" refers to a software module that has the function of generating specific steps or commands to be performed next based on the results of the investigation.

[1847] "Means for displaying" refers to a display or user interface for informing the user of the generated procedures or commands.

[1848] "User" refers to a person who operates this system and whose role is to input the response results of the device and carry out work according to the generated procedure.

[1849] "Means for recognizing emotions" refers to software or hardware that analyzes a user's facial expressions, tone of voice, etc. to determine their emotional state.

[1850] "Adjustment means" refers to a software module that has the functionality to change or adjust the content of the procedures or commands presented based on the recognized emotions.

[1851] This invention is a system in which the user inputs the device's response results into a terminal, and the server generates and presents the next required command or procedure based on the results. By combining this process with an emotion engine that recognizes the user's emotions, we have created a system that supports appropriate responses according to the user.

[1852] System configuration

[1853] Hardware and software used

[1854] 1. Terminal

[1855] Hardware: PCs, tablets, smartphones, etc.

[1856] Software: Web browser, input interface, communication module

[1857] 2. Server

[1858] Hardware: High-performance server

[1859] Software: AI module, data analysis algorithm, emotion recognition engine, communication module

[1860] 3. Emotion Recognition Engine

[1861] Hardware: Camera, microphone

[1862] Software: Image analysis software, audio analysis software

[1863] System Operation

[1864] This system operates in the following specific steps:

[1865] 1. The user inputs the device's response results.

[1866] The user inputs the response obtained from the device into the terminal, for example, entering data such as the temperature read from the temperature sensor (85°C), and presses the "Send" button.

[1867] 2. The device sends the input response to the server.

[1868] The terminal sends the input data to the server in packet format, using protocols such as HTTP.

[1869] 3. The server analyzes the response

[1870] The server analyzes the received data and compares it with expected normal values ​​or set values, for example, a set value of "70°C" stored in a database, to determine whether there is any deviation.

[1871] 4. The server investigates how to resolve the discrepancy and generates a procedure

[1872] The server's AI module investigates the best way to resolve the discrepancy and generates the next steps to take, such as turning on the cooling system.

[1873] 5. The server verifies the findings

[1874] The server verifies whether the investigation results are valid using simulation tools and past data logs.

[1875] 6. The server generates and sends the next command and procedure to be performed.

[1876] For example, it generates specific instructions such as "turn on the cooling system and set the fan to maximum output" and sends them to the terminal.

[1877] 7. Recognize user emotions

[1878] The emotion recognition engine analyzes the user's facial expressions and tone of voice to recognize emotions such as anxiety and stress.

[1879] 8. The server adjusts procedures based on emotions

[1880] The generated instructions are adjusted based on the recognized emotion data, for example, providing more detailed instructions if the user is feeling anxious.

[1881] 9. The device presents instructions to the user

[1882] The terminal displays the procedure received from the server to the user in a format that is easy for the user to understand.

[1883] 10. The user follows the steps provided

[1884] The user follows the displayed instructions, for example, turning on the cooling system and setting the fans to maximum power.

[1885] Examples of concrete examples and prompts

[1886] For example, consider a situation where a user inputs a temperature sensor reading of "85°C" into a terminal, and the server generates and presents a procedure to "turn on the cooling system and set the fan to maximum output" based on that. In this case, the following prompt sentence is used:

[1887] Sample prompt: "Generate the next steps based on the following device response result: '85°C'. The user is feeling anxious."

[1888] This system allows users to receive appropriate response procedures in real time, enabling them to respond quickly.In addition, by using an emotion recognition engine, it provides appropriate procedures according to the user's emotional state, improving work efficiency and safety.

[1889] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1890] Step 1:

[1891] The user inputs the response result of the device.

[1892] The user inputs the response obtained from the device into the terminal. For example, the reading from the temperature sensor is "85°C." The input data is the raw data from the temperature sensor (85°C), and the output is a data packet containing this data.

[1893] Step 2:

[1894] The terminal sends the input response results to the server.

[1895] The terminal converts the input data into a packet format and sends it to the server. Specifically, it sends the data using the HTTP protocol. The input is a data packet from the user, and the output is a data packet sent to the server.

[1896] Step 3:

[1897] The server analyzes the response

[1898] The server analyzes the received data (85°C). Specifically, it compares it with the set value "70°C" stored in the database. The input is the raw data (85°C) sent from the terminal, and the output is the analysis result: "85°C exceeds the set value of 70°C."

[1899] Step 4:

[1900] The server checks for discrepancies

[1901] Based on the analysis results, the server determines that the actual temperature exceeds the set value, indicating an abnormality. The input is the analysis result, and the output is the judgment on whether or not there is an abnormality (abnormality).

[1902] Step 5:

[1903] Investigate how the server resolves the discrepancy

[1904] The server's AI module runs and investigates the optimal solution. For example, it suggests turning on the cooling system. The input is the anomaly detection result, and the output is a candidate for the optimal solution (turning on the cooling system).

[1905] Step 6:

[1906] The server verifies the findings

[1907] The server verifies the effectiveness of the investigation results using simulations and past data logs. Specifically, it uses a simulation tool. The input is the candidate optimal countermeasures, and the output is the verification result (validity).

[1908] Step 7:

[1909] The server generates the next command and steps to be executed.

[1910] Based on the verification results, the server generates specific steps to be taken next, such as "turn on the cooling system and set the fan to maximum output." The input is the verification results, and the output is the specific steps.

[1911] Step 8:

[1912] Recognize user emotions

[1913] The emotion recognition engine analyzes the user's facial expressions and tone of voice to determine whether the user is feeling anxious or stressed. Specifically, it uses a camera and microphone. The input is the user's real-time voice and video data, and the output is the user's emotional state (anxiety).

[1914] Step 9:

[1915] The server adjusts the procedure based on emotions.

[1916] The server adjusts the detail and explanation of the procedure based on the emotional data. For example, if the user feels anxious, the server will provide more detailed instructions. The input is the emotional state data, and the output is the adjusted procedure.

[1917] Step 10:

[1918] The server sends the generated instructions to the device.

[1919] The server sends the generated instructions to the terminal in the form of packets, where the input is the adjusted instructions and the output is the data packets sent to the terminal.

[1920] Step 11:

[1921] The terminal presents instructions to the user

[1922] The terminal displays the instructions received from the server to the user. Specifically, it displays on the screen, "Please turn on the cooling system and set the fan to maximum output." The input is the instructions sent from the server, and the output is the display to the user.

[1923] Step 12:

[1924] The user follows the steps provided

[1925] The user follows the instructions displayed on the terminal to perform physical and software tasks, such as turning on the power to a cooling system and setting the internal fan to maximum output. The input is the displayed instructions, and the output is the result of executing the instructions.

[1926] Step 13:

[1927] The user re-enters the device's response and sends it.

[1928] The user checks the device status after the procedure is executed and inputs a new response result (e.g., temperature "75°C") into the terminal. Then, the user presses the "Send" button again to send the data to the server. The input is the new response result, and the output is the resent data packet.

[1929] (Application example 2)

[1930] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1931] While conventional device monitoring systems can generate procedures based on device response results, they are unable to adjust procedures to take into account the user's emotional state. This has led to issues such as users becoming stressed, especially in emergency situations or complex situations, and performance degradation due to operational errors and tension. This can compromise efficiency and safety in scenarios that require rapid and accurate responses.

[1932] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes a means for inputting the device's response result, a means for analyzing the response result and checking for deviation from a target procedure, a means for investigating a method for eliminating the deviation and generating a procedure to be performed next, a means for recognizing the user's emotion, and a means for adjusting the procedure based on the recognized user emotion. This makes it possible to generate an appropriate procedure based on the device's response result and further provide detailed guidance according to the user's emotion. This reduces user stress and enables a quick and accurate response.

[1933] "Device response results" are data or output information generated by a device during operation.

[1934] A "means" is a method, process, or device used to accomplish a particular purpose.

[1935] "Target procedures" are procedures or standard values ​​that are set in advance to achieve normal operation or a goal.

[1936] "Deviation" means a deviation or abnormal variation from an expected normal range or set point.

[1937] "Research" is the process of collecting and analyzing information to find appropriate methods and procedures for solving a problem.

[1938] "Generation" is the act of creating new procedures or commands based on specific conditions or data.

[1939] "Display" refers to the act of visually presenting information to a user.

[1940] "Emotion recognition" is a technology that analyzes a user's facial expressions, tone of voice, etc. to determine their emotional state.

[1941] "Adjustment" is the act of changing or modifying procedures or instructions to suit specific conditions or circumstances.

[1942] A "system" is a collection of integrated devices or processes in which multiple elements or means work together to perform a single function.

[1943] "Device" refers to an electronic device or terminal used for a specific purpose.

[1944] A "server" is a device that stores and processes data on a network and provides services to other devices and users.

[1945] "Input of response results" refers to the act of a user manually or automatically inputting data or results output by a device into the system.

[1946] "Data analysis" is the process of analyzing collected data to find meaningful information and patterns.

[1947] "Emotion-based procedure adjustment" refers to the act of appropriately changing the procedures and guidance presented to the user in consideration of the user's emotional state.

[1948] This invention relates to a system that generates and presents the next steps required based on the response results of the device, and also recognizes and adjusts the user's emotions. This system is intended to monitor and manage devices and robots used primarily in factories.

[1949] The system includes the following major components:

[1950] 1. User Device

[1951] The user device inputs the response results from the device and sends the data to the server. The device is typically a smartphone or tablet, and is equipped with a camera and microphone. This allows it to recognize the user's emotions.

[1952] 2. Server

[1953] The server is the central processing unit that analyzes the data, generates procedures as needed, and presents them to the user. The server includes the following functions:

[1954] Data Analysis Module

[1955] The server receives and analyzes the device response results sent from the user terminal, checks for deviations from the normal range, and investigates and determines the optimal method to resolve the deviations.

[1956] Procedure Generation Module

[1957] Based on the analysis results, the next steps to be taken are generated and verified to see if they are effective.

[1958] Emotion Recognition Module

[1959] It analyzes the video and audio data sent from the user's device to determine the user's emotional state (anxiety, stress, etc.).

[1960] Procedure Coordination Module

[1961] Based on the results of the emotion recognition module, the generated instructions are adjusted, specifically determining whether to provide detailed or concise instructions depending on the emotional state.

[1962] 3. Presentation to the User

[1963] The generated procedure is again transmitted to the user terminal and visually presented to the user.

[1964] As a concrete example, consider a situation where a temperature sensor in a factory detects 85°C, which exceeds the set value of 70°C. The user enters this reading into a terminal and sends it to the server. The server analyzes this data and determines that the cooling fan needs to be turned on. Furthermore, if the emotion recognition module detects the user's anxiety, it adjusts the procedure to "Turn on the cooling system and set the internal fan to maximum output. If you have any questions during the procedure, please refer to the manual."

[1965] The main software and hardware used are cloud computing platforms (e.g., AWS, Google Cloud) for the server, and smartphones (compatible with iOS or Android) for the user devices. The EmotionRecognition library is used for emotion recognition.

[1966] A specific example of a prompt for a generative AI model is, "The temperature sensor detected 85°C. This is an abnormal condition, exceeding the set value (70°C). Please generate the following steps. If the user is feeling uneasy, please explain the steps in detail." Using this prompt, the AI ​​can generate optimal steps and provide appropriate guidance to the user.

[1967] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1968] Step 1:

[1969] The user inputs the device's response result. The user inputs the device's response result (e.g., the temperature sensor reading "85°C") into a smartphone or tablet and presses the send button. The input data is sent to the device in text format.

[1970] (Input) Device response result (temperature "85°C")

[1971] (Output) Response result data sent from the user terminal

[1972] Step 2:

[1973] The device sends the input response results to the server. The device converts the sent data into packet format and sends it to the cloud server. At that time, security protocols (e.g., SSL / TLS) are used to maintain the integrity of the data.

[1974] (Input) Response result data entered by the user (temperature "85°C")

[1975] (Output) Response result data sent to the cloud server

[1976] Step 3:

[1977] The server analyzes the response results. The server analyzes the received data and checks for deviations from normal values ​​or set values ​​(e.g., "70°C"). The data analysis module does this and detects abnormalities.

[1978] (Input) Response result data sent to the cloud server (temperature "85°C")

[1979] (Output) Deviation judgment result (temperature "85°C" is abnormal)

[1980] Step 4:

[1981] The server investigates ways to resolve the discrepancy and generates the next steps to be taken. An AI module then runs to find the optimal solution based on past data and simulations (e.g., running a cooling system).

[1982] (Input) Deviation judgment result (temperature "85°C" is abnormal)

[1983] (Output) Initial action steps (e.g., turning on cooling system)

[1984] Step 5:

[1985] The server recognizes the user's emotions. Camera footage and audio data are sent from the user's device, and the emotion recognition module analyzes them to determine whether the user is feeling anxious.

[1986] (Input) User's video and audio data

[1987] (Output) Emotional state (e.g., anxiety)

[1988] Step 6:

[1989] The server adjusts the instructions based on the emotion. Based on the emotion recognition results, the server adjusts the instructions it presents (e.g., explains the instructions in detail).

[1990] (Input) Initial response steps (turn on cooling system), emotional state (anxiety)

[1991] (Output) Adjusted countermeasure steps (with detailed explanation)

[1992] Step 7:

[1993] The server sends the generated procedure to the terminal, converts the generated procedure into a packet format, and sends it to the user's terminal.

[1994] (Input) Adjusted countermeasure procedures (with detailed explanations)

[1995] (Output) Instructions sent to the user terminal

[1996] Step 8:

[1997] The terminal presents instructions to the user. The terminal visually displays the received instructions and presents them in a format that is easy for the user to understand (e.g., "Please turn on the cooling system and set the internal fan to maximum output. If you have any questions during the procedure, please refer to the manual.").

[1998] (Input) Procedure data received from the server

[1999] (Output) Instructions displayed to the user

[2000] Step 9:

[2001] The user carries out the presented procedure. The user performs the work according to the instructions displayed on the terminal.

[2002] (Input) Instructions displayed to the user

[2003] (Output) User execution of steps

[2004] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[2005] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[2006] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[2007] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[2008] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[2009] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[2010] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[2011] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, motorcycles, and other devices, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[2012] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[2013] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[2014] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[2015] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[2016] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[2017] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[2018] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[2019] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[2020] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[2021] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[2022] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[2023] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[2024] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[2025] The following is further disclosed regarding the above embodiment.

[2026] (Claim 1)

[2027] means for inputting the response results of the device;

[2028] means for analyzing the response result and checking for deviation from a target procedure;

[2029] means for investigating a method for eliminating the discrepancy and generating a procedure to be performed next;

[2030] means for displaying the generated procedure;

[2031] A system including:

[2032] (Claim 2)

[2033] 2. The system according to claim 1, further comprising: means for displaying the generated procedure to a user and inputting the response result to a server.

[2034] (Claim 3)

[2035] means for generating a specific command for eliminating the discrepancy based on the response result;

[2036] means for verifying that the specific command is valid;

[2037] The system of claim 1 further comprising:

[2038] "Example 1"

[2039] (Claim 1)

[2040] means for inputting the response results of the device;

[2041] means for analyzing the response result and checking for deviation from a target procedure;

[2042] A means for investigating a method for eliminating the discrepancy using a generative AI model and generating a procedure to be performed next;

[2043] means for displaying the generated procedure;

[2044] A system including:

[2045] (Claim 2)

[2046] 2. The system of claim 1, further comprising means for displaying the generated procedure to a user and inputting the response results into a data management device.

[2047] (Claim 3)

[2048] means for generating specific operation instructions for eliminating the discrepancy based on the response result;

[2049] means for verifying whether the specific operation instructions are valid;

[2050] The system of claim 1 further comprising:

[2051] "Application Example 1"

[2052] (Claim 1)

[2053] means for inputting the response results of the device;

[2054] means for analyzing the response result and checking for deviation from a target procedure;

[2055] means for investigating a method for eliminating the discrepancy and generating a procedure to be performed next;

[2056] means for displaying the generated procedure;

[2057] a means for recalculating a picking procedure of the logistics robot based on the response result and generating a corrected command;

[2058] A system including:

[2059] (Claim 2)

[2060] means for displaying the generated procedure to a user and inputting the response result to a server;

[2061] A means of presenting appropriate steps in the logistics process in real time;

[2062] The system of claim 1 further comprising:

[2063] (Claim 3)

[2064] a means for generating a specific picking command for eliminating the discrepancy;

[2065] means for verifying that the generated commands are accurate and optimal;

[2066] The system of claim 1 further comprising:

[2067] "Example 2: Combining Emotion Engines"

[2068] (Claim 1)

[2069] means for inputting the response results of the device;

[2070] means for analyzing the response result and checking for deviation from a target procedure;

[2071] means for investigating a method for eliminating the discrepancy and generating a procedure to be performed next;

[2072] means for displaying the generated procedure;

[2073] means for recognizing a user's emotion;

[2074] means for adjusting a procedure based on the recognized emotion;

[2075] A system including:

[2076] (Claim 2)

[2077] 2. The system according to claim 1, further comprising: means for displaying the generated procedure to a user and inputting the response result to a server.

[2078] (Claim 3)

[2079] means for generating a specific command for eliminating the discrepancy based on the response result;

[2080] means for verifying that the specific command is valid;

[2081] The system of claim 1 further comprising:

[2082] "Application example 2 when combining emotion engines"

[2083] (Claim 1)

[2084] means for inputting the response results of the device;

[2085] means for analyzing the response result and checking for deviation from a target procedure;

[2086] means for investigating a method for eliminating the discrepancy and generating a procedure to be performed next;

[2087] means for displaying the generated procedure;

[2088] means for recognizing a user's emotion;

[2089] means for adjusting a procedure based on the recognized user emotion;

[2090] A system including:

[2091] (Claim 2)

[2092] means for displaying the generated procedure to a user and inputting the response result to a server;

[2093] A means for recognizing a user's emotion using a smart device;

[2094] The system of claim 1 further comprising:

[2095] (Claim 3)

[2096] means for generating a specific command for eliminating the discrepancy based on the response result;

[2097] means for verifying that the specific command is valid;

[2098] means for adjusting the details of the command in response to the user's emotions;

[2099] The system of claim 1 further comprising: [Explanation of symbols]

[2100] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. means for inputting the response results of the device; means for analyzing the response result and checking for deviation from a target procedure; means for investigating a method for eliminating the discrepancy and generating a procedure to be performed next; means for displaying the generated procedure; A system including:

2. 2. The system according to claim 1, further comprising: means for displaying the generated procedure to a user and inputting the response result to a server.

3. means for generating a specific command for eliminating the discrepancy based on the response result; means for verifying that the specific command is valid; The system of claim 1 further comprising:

Images