system

Abstract

We provide the system. [Solution] A means for receiving operation commands, A planning generation means that generates an operation plan using a generative model, Operating means for performing remote control of a machine device, A monitoring system that monitors the progress of the work and provides feedback, An adjustment mechanism to adjust the operation in response to unforeseen circumstances, A system that includes this.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a method for controlling a persona chatbot, which is performed by at least one processor, including steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] The problems to be solved by the present invention are to reduce the burden on the elderly and workers and ensure safety in snow removal work in snowy areas and work at construction sites. In particular, due to the serious shortage of operators for heavy machinery work, it is necessary to overcome this and realize efficient work. Also, a mechanism that enables work in a 24-hour system and efficiently performs operations is required.

Means for Solving the Problems

[0005] The present invention provides a system comprising: a receiving means for receiving operation commands; a planning means for generating operation plans using a generation model; an operation means for performing remote operations on machinery; a monitoring means for monitoring the progress of work and providing feedback; and an adjustment means for adjusting operations in response to unforeseen circumstances. This allows users to easily request work using a terminal, and the generation model creates an optimal operation plan, enabling remote operation of heavy machinery. Furthermore, by monitoring work in real time and making adjustments as needed, operations can be performed safely and efficiently.

[0006] "Operation commands" refer to specific instructions for actions or tasks that a user gives to a machine via a terminal device.

[0007] A "reception mechanism" is a component that receives operation commands and interprets that data for use within the processing system.

[0008] A "generative model" refers to machine learning algorithms and techniques used to formulate optimal work plans based on historical data and environmental information.

[0009] A "plan generation means" is a component that has the function of automatically formulating an operation plan for a machine or device by utilizing a generation model.

[0010] "Mechanical equipment" refers to heavy machinery and other equipment used to actually carry out tasks such as snow removal and construction.

[0011] "Remote control" refers to a method of controlling the operation of a machine or device from a physically distant location using communication means.

[0012] "Operating means" refers to a component that has the function of sending commands to a machine or device from a distance and controlling its operation.

[0013] A "monitoring device" is a component that has the function of observing and recording the progress of work and the status of machinery in real time.

[0014] "Feedback" refers to the process of presenting information for adjusting various parameters and operations within a system based on the observed data.

[0015] "Adjustment means" is a component having a function for readjusting the operation schedule and procedure of a mechanical device in response to unforeseen situations and environmental changes.

Brief Description of Drawings

[0016] [Figure 1] It is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] It is a conceptual diagram showing an example of the main functions of a data processing device and a smart device according to the first embodiment. [Figure 3] It is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] It is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] It is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] It is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] It is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] It is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] It shows an emotion map to which a plurality of emotions are mapped. [Figure 10] It shows an emotion map to which a plurality of emotions are mapped. [Figure 11] It is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] It is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13]It is a sequence diagram showing the processing flow of the data processing system in Example 2 when the emotion engine is combined. [Figure 14] It is a sequence diagram showing the processing flow of the data processing system in Application Example 2 when the emotion engine is combined.

Embodiment for Carrying Out the Invention

[0017] Hereinafter, an example of an embodiment of the system according to the technology of the present disclosure will be described according to the accompanying drawings.

[0018] First, the terms used in the following description will be explained.

[0019] In the following embodiments, the numbered processor (hereinafter simply referred to as "processor") may be one arithmetic unit or a combination of a plurality of arithmetic units. Also, the processor may be one type of arithmetic unit or a combination of a plurality of types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0020] In the following embodiments, the numbered RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0021] In the following embodiments, the numbered storage is one or more non-volatile storage devices that store various programs and various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes, etc.

[0022] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0023] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0024] [First Embodiment]

[0025] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0026] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0027] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0028] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0029] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0030] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0031] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0032] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0033] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0034] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0035] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0036] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0037] Embodiments of the present invention will now be described. This system mainly consists of a server, a terminal, and heavy machinery, and is a mechanism that enables remote control using a generative model.

[0038] First, users issue work instructions using their devices. Specifically, they use smartphones or PCs to input work details, work location information, and desired completion time, thereby sending operational instructions to the system. This information is received by the server.

[0039] The server analyzes the work instructions received from the user and generates an optimal work plan using a generative model. This plan is automatically created by AI based on past performance data and environmental information. The server then determines which heavy machinery is best suited for use according to this plan and sends work instructions to that machinery.

[0040] The heavy machinery autonomously begins work based on instructions from the server. Utilizing its onboard sensors, the machinery accurately performs the instructed tasks while monitoring its surroundings in real time. If an unexpected problem occurs during operation, the server immediately analyzes the information and provides new instructions through a generative model, enabling a rapid response.

[0041] As a concrete example, consider a case where a user requests snow removal for a parking lot in a certain area. The user sets the location of the parking lot and the desired snow removal time on a terminal and issues a command. The server calculates the optimal route for snow removal and issues instructions to heavy machinery. The heavy machinery follows the instructions, autonomously begins snow removal work, and sends a completion report to the server. Upon receiving this completion report, the server notifies the user that the work has been completed. In this way, this system solves the problems of an elderly workforce and labor shortages, and enables efficient 24-hour operation.

[0042] The following describes the processing flow.

[0043] Step 1:

[0044] The user uses a terminal to input work instructions. Specifically, they set the location information of the area where snow removal work is to be performed, the work details, and the desired completion time, and send this information to the system. This information is then transmitted to the server.

[0045] Step 2:

[0046] The server analyzes the work instructions received from the user and creates a work plan using a generative model. The plan includes the optimal work route and motion sequence, as well as the selection of heavy machinery. At this stage, the AI ​​analyzes historical data and current environmental information to generate an efficient plan.

[0047] Step 3:

[0048] Based on the plan, the server generates specific work instructions for the appropriate heavy machinery and prepares them for remote operation. These instructions include detailed operational instructions such as route, speed, and operating procedures. These are then transmitted to the heavy machinery's control system.

[0049] Step 4:

[0050] The heavy machinery autonomously begins work based on instructions received from the server. Onboard sensors monitor the surrounding environment in real time, and the machinery performs its work while proceeding along the designated route. If any obstacles or unexpected situations occur during operation, the heavy machinery reports them to the server via data from the sensors.

[0051] Step 5:

[0052] The server analyzes data and feedback from the heavy machinery and adjusts instructions according to the new situation. It makes optimal route changes and speed adjustments in response to unforeseen circumstances and obstacles, and sends revised instructions to the heavy machinery.

[0053] Step 6:

[0054] Once the work is complete, the heavy machinery sends a completion report to the server. The server receives this report, confirms that the work was completed successfully, and then sends a completion notification to the user. This allows the user to confirm in real time that the snow removal was completed without any problems.

[0055] (Example 1)

[0056] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0057] Effective autonomous operation of machinery and equipment in remote locations, along with real-time progress feedback, requires sophisticated work planning and rapid fault response capabilities. However, in conventional systems, these processes are often performed manually, resulting in low work efficiency and delays in responding to unforeseen circumstances.

[0058] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0059] In this invention, the server includes a reception means using an information terminal to receive operation commands, a plan generation means that generates an operation plan based on past performance data and environmental information using a generation artificial intelligence model, and an operation means that transmits work instructions to selected heavy machinery using wireless communication technology. This enables autonomous operation of machinery and real-time monitoring of work status, resulting in efficient and rapid response.

[0060] An "operation command" is instructional information that specifies the exact tasks and conditions that a user should perform on a machine or device.

[0061] An "information terminal" is a technological device used by a user to input operating commands, and includes, for example, smartphones and personal computers.

[0062] A "reception method" is a technique that has the function of receiving operation commands transmitted from an information terminal and storing them in a database necessary for processing.

[0063] A "generative artificial intelligence model" is an AI model trained using algorithms based on a vast amount of historical data, and is a means of creating the optimal operation plan under specific conditions.

[0064] An "operation plan" is a plan of specific work processes and sequences that a machine should perform, generated by a generative artificial intelligence model.

[0065] "Plan generation means" refers to a function or system for formulating an operation plan using an artificial intelligence model based on operation commands.

[0066] "Heavy machinery" refers to large mechanical devices used to perform work in designated locations, and includes excavators, snowplows, and other similar equipment.

[0067] "Wireless communication technology" refers to methods for sending and receiving data between electronic devices without using cables, and examples include Wi-Fi and Bluetooth.

[0068] "Operation means" refers to a method of remotely controlling heavy machinery by issuing instructions based on a generated operation plan.

[0069] "Feedback" is an information transmission process that reports information about the progress of a machine's operation to a server in real time.

[0070] "Monitoring" is the process of continuously observing and recording the surrounding conditions and work progress of machinery and equipment using sensors.

[0071] "Operation adjustment means" refers to a function for replanning operations in order to remove factors that hinder the progress of work when unforeseen circumstances occur.

[0072] This invention provides a method for efficiently and autonomously executing user work commands using a system comprising an information terminal, a computer server, and a remotely controllable mechanical device. The embodiments for carrying out this invention are described in detail below.

[0073] Users input work instructions using information terminals such as smartphones or personal computers. These work instructions include information such as the content of the specified work, geographical area, and desired completion time. A dedicated application runs on the terminal and securely transmits the information entered by the user to the server. This process uses the SSL / TLS protocol to ensure data security.

[0074] The server generates an optimal work plan based on the received work instructions, utilizing a generative artificial intelligence model. This AI model is trained using machine learning libraries such as TENSORFLOW® and analyzes historical and current environmental data. This enables rapid and highly accurate plan generation.

[0075] The generated work plan is transmitted to the selected heavy machinery via wireless communication technology. The heavy machinery is equipped with LiDAR sensors and cameras, which are used to monitor the surrounding environment during operation. This ensures that the work is executed accurately, and the progress is constantly fed back to the server.

[0076] As a concrete example, if a user requests snow removal work at a parking lot in a certain area, the user uses a terminal to input the location of the parking lot and the desired work time, and then issues a command. Based on this information, the server calculates the optimal snow removal route and sends instructions to the heavy machinery. The heavy machinery starts working according to the instructions and reports to the server upon completion. Upon receiving this report, the server notifies the user that the work has been completed.

[0077] An example of an input prompt for the generating AI model would be: "Optimize snow removal operations for the specified parking lot. The location information is [specific location], and the desired time is [specific time]."

[0078] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0079] Step 1:

[0080] The user enters work instructions using a terminal. Specifically, they launch a dedicated application and enter the work details, location information of the target area, and desired completion time into a form. Once this information is entered, the application encrypts the data and sends it to the server using a secure protocol (SSL / TLS). The input data generates output as a work instruction.

[0081] Step 2:

[0082] The server receives work instructions from the terminal and saves their contents to the database. The received data is first validated. This process checks for any errors in the data format or input content. Once validation is complete, the content is output as valid data for processing.

[0083] Step 3:

[0084] The server invokes a generating AI model based on the received work instructions to generate an optimal work plan. At this stage, the AI ​​model receives historical work data and current environmental data as input and performs data analysis and machine learning. This results in an optimized plan of necessary work steps and resources to be used.

[0085] Step 4:

[0086] The server selects the heavy machinery to be used based on the created work plan. It evaluates the operating status and location of the heavy machinery in real time and selects the optimal machine. Once the selection is complete, it sends work instructions to the heavy machinery using wireless communication technology (e.g., MQTT protocol). The output contains information on the selected heavy machinery and the work instructions.

[0087] Step 5:

[0088] The heavy machinery begins work according to instructions sent from the server. Using LiDAR sensors and cameras mounted on the machinery, it performs the instructed tasks while monitoring the surrounding environment in real time. This monitoring data is returned to the server as progress information during the work. The output includes work progress and environmental information.

[0089] Step 6:

[0090] Once the work is completed, the heavy machinery reports this information to the server. The server receives the report, records the work results in the database, and notifies the user based on the prompt message. The output consists of confirmation of work completion and a notification to the user.

[0091] (Application Example 1)

[0092] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0093] Conventional autonomous assembly systems suffered from insufficient optimization of work plans, making it difficult to improve the efficiency and flexibility of assembly operations. Furthermore, they lacked the means to quickly respond to obstacles and changing environments, thus creating a need for improved productivity and safety.

[0094] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0095] In this invention, the server includes a receiving means for receiving operation commands, a planning means for generating an operation plan using a generative model, and an optimization means for using a generative model to optimize assembly work. This enables efficient and safe assembly work by an autonomous device.

[0096] A "reception mechanism for receiving operation commands" is a function that receives operation instructions entered by the user and converts them into a format that can be processed within the system.

[0097] "Planning generation means for generating operation plans using generative models" refers to a function that automatically generates optimal work schedules and procedures using AI models.

[0098] "Optimization methods using generative models to optimize assembly work" refers to a function that analyzes work processes using AI models and identifies areas for improvement in order to achieve efficient assembly work.

[0099] "Operating means for performing remote control on a machine" refers to a function that remotely controls and operates a machine based on instructions from a server.

[0100] A "monitoring method that monitors the progress of work and provides feedback" is a function that constantly monitors the status of work and immediately provides necessary information back to the user or system.

[0101] "Adjustment mechanisms for adjusting operations in response to unforeseen circumstances" refer to functions that quickly correct the system's response to sudden failures or changes, thereby ensuring the continuation of normal operations.

[0102] An "autonomous device" is a device that can perform tasks automatically without external intervention.

[0103] The "real-time optimization function" is a feature that immediately derives and applies the optimal work procedure based on information obtained during the work process.

[0104] The system for realizing this invention consists of a server, a terminal, and an autonomous device. It begins with the user inputting operation commands using the terminal and sending them to the server. These operation commands include details such as the task content and deadline. The server receives this information and uses a generative AI model to create an optimal work plan. This plan is designed to maximize work efficiency, and the generative model analyzes historical data and real-time environmental information.

[0105] Based on the plan, the server sends specific instructions to the autonomous device. The autonomous device receives the instructions from the server and automatically begins the assembly work. The device is equipped with sensors that allow it to continuously monitor its surroundings. Data from the sensors is used to monitor the progress of the work in real time and is sent back to the server as feedback. If an unforeseen problem occurs during the work, the server uses a generative AI model to immediately generate new instructions and adjust the device's operation.

[0106] As a concrete example, consider the efficient assembly of bolts and nuts on an automobile parts assembly line. In this case, by sending a prompt message from a terminal saying, "Generate an efficient combination pattern for automobile engine assembly," the server can formulate a work plan and optimize the work in real time. With such a configuration, the server can manage the work efficiently and safely, improving the overall productivity of the system.

[0107] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0108] Step 1:

[0109] The user inputs operational commands using a terminal. The information entered includes the work details, start time, and destination, and the terminal transmits this data to the server.

[0110] Step 2:

[0111] The server analyzes the operation commands received from the user. Based on the input data, it automatically generates an optimal work plan using a generation AI model. It formulates an efficient work schedule by considering past work history data and environmental information. The generated work plan is obtained as output.

[0112] Step 3:

[0113] Based on the generated work plan, the server creates and sends specific operating instructions to the autonomous device. These instructions include details such as which parts should be assembled and in what order. Upon receiving these instructions, the autonomous device generates an instruction list as output.

[0114] Step 4:

[0115] The autonomous device begins assembly work according to the operation instructions received from the server. Sensors mounted on the device detect the surrounding environment and transmit progress information to the server in real time. The input is the operation instruction, and the output is the progress status.

[0116] Step 5:

[0117] If an unforeseen event occurs during operation, the server analyzes the progress information and quickly generates new operation instructions using a generation AI model. Inputs are progress information and data on unforeseen events, while output is the updated operation instructions. The generated instructions are immediately transmitted to the autonomous device and adjusted to ensure smooth operation.

[0118] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0119] Embodiments of the present invention will now be described. This system includes a function to recognize the user's emotions and dynamically adjust the operation plan based on those emotions. The main components are a server, a terminal, a mechanical device, and an emotion engine.

[0120] Users input work instructions via a terminal, and during this process, an emotion sensor reads the user's emotions from their facial expressions and voice. This emotion data is sent to the server along with the work instructions. By using emotion recognition technology, it is possible to create operation plans that reflect the user's stress levels and expectations.

[0121] The server uses an emotion engine to analyze the user's current emotional state and fine-tunes the operation plan derived from the generative model based on the results. For example, if the user is anxious, the server can provide detailed reports on the progress of the work and share real-time progress to provide reassurance. At this stage, the server selects heavy machinery suitable for the designated work area and prepares the specific instructions necessary for remote operation.

[0122] Next, the server sends work instructions to the heavy machinery, which then autonomously begins work based on the instructions from the server. During the work, various sensors on the heavy machinery monitor the surrounding environment, avoiding obstacles and proceeding with the work as planned. If an unforeseen situation occurs, a situation report including emotional data is fed back to the server, and the server uses a generative model to issue new work instructions. In this process, changes in the user's emotions are also taken into consideration, and feedback is adjusted as needed.

[0123] For example, when a user feels a sense of urgency during snow removal work in a store's parking lot during heavy snowfall, the system quickly creates a plan and visualizes the work status as needed to reassure the user. Thus, a key point of this invention is to achieve safe and efficient work execution through emotionally sensitive feedback and work adjustments.

[0124] The following describes the processing flow.

[0125] Step 1:

[0126] The user uses a terminal to input work instructions and sends the work area and desired completion time to the system. Additionally, an emotion sensor built into the terminal monitors the user's facial expressions and voice, collecting emotional data at that moment. This data is then sent to a server.

[0127] Step 2:

[0128] The server receives work instructions and emotional data from the user. The received emotional data is analyzed by an emotion engine to determine the user's emotional state. For example, if the user's voice contains anxiety or tension, this is detected and recorded as the emotional state.

[0129] Step 3:

[0130] The server uses a generative model to create a normal operation plan, but it adjusts the plan appropriately by incorporating the results of the emotion engine's analysis. Specifically, it takes measures such as increasing information sharing at important decision points to alleviate user anxiety as much as possible.

[0131] Step 4:

[0132] The server sends the adjusted operating instructions to the heavy machinery. These instructions include the route, operating speed, and avoidance procedures. The plan is optimized here, taking user sentiment into consideration.

[0133] Step 5:

[0134] The heavy machinery receives instructions from the server and autonomously begins work. During operation, the machinery monitors its surroundings with sensors and reports the situation to the server in real time. Adjustments necessary for the progress of the work are continuously made in conjunction with emotional data.

[0135] Step 6:

[0136] Once the work is complete, the heavy machinery reports completion to the server. The server confirms this and provides the user with a completion notification and sentiment-based feedback. This allows the user to confidently confirm that the work has been completed.

[0137] (Example 2)

[0138] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0139] Conventional remote control systems have problems such as failing to consider the user's psychological state and not adequately ensuring work efficiency and safety. Furthermore, the lack of real-time interaction between operation instructions and actual machine operation makes users prone to anxiety and stress. In addition, unforeseen circumstances often require simple manual intervention, placing a burden on the user.

[0140] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0141] In this invention, the server includes a receiving means for receiving operation commands, a plan generation means for generating operation plans using a generative model and dynamically adjusting them based on the user's emotions, and an emotion recognition means for analyzing the user's facial expressions and voice to collect emotion data. This enables real-time adjustment of operation plans that take the user's emotions into consideration, improving work efficiency and safety, and allowing for a rapid response to unforeseen circumstances.

[0142] An "operation command" refers to a specific work instruction that a user sends to the system via their terminal.

[0143] A "generative model" is a type of artificial intelligence model used by systems to create operation plans, enabling dynamic plan generation based on data.

[0144] A "plan generation means" is a component of a system that has the function of generating an operation plan and adjusting it based on the user's emotions.

[0145] "Emotion recognition means" refers to a component of a system that collects emotional data by analyzing the user's facial expressions and voice.

[0146] "Operating means" refers to the function of a system that performs instructed operations in order to remotely control a machine or device.

[0147] "Monitoring means" refers to a system function that monitors the progress of work and provides users with real-time feedback on the progress.

[0148] A "regulation mechanism" is a component of a system that appropriately adjusts its operation in the event of unforeseen circumstances and provides feedback based on changes in the user's emotions.

[0149] The system of this invention analyzes the user's emotions and dynamically adjusts the operation plan based on them. It mainly consists of a server, terminals, mechanical devices, and software that links them together.

[0150] The terminal provides an interface for receiving operation commands entered by the user. The terminal is equipped with an emotion sensor that analyzes the user's facial expressions and voice in real time, thereby collecting emotion data. The collected emotion data is securely encrypted and transmitted to the server.

[0151] The server uses a generative AI model to generate and dynamically adjust operation plans based on the user's emotional state. It utilizes emotion recognition techniques to analyze the user's stress levels, sense of security, and other states, and modifies the generated plan accordingly. For example, if the user is feeling anxious, the operation plan can be modified to provide more detailed progress updates.

[0152] The machinery receives operation commands transmitted from the server and begins operating autonomously. Heavy machinery, such as equipment, constantly monitors its surroundings using various sensors to perform tasks safely. In the event of an unforeseen situation, the situation, including emotional data, is quickly fed back to the server, and new work instructions are generated.

[0153] A concrete example would be a situation where a user is anxious about bad weather during garden maintenance work. In this case, the server would visually display the work progress to reassure the user. An example of a prompt to the generated AI model would be, "If the user is perceived as anxious, please tell me how to display the progress in detail," which would allow for appropriate feedback.

[0154] This system makes it possible to perform tasks safely and efficiently, tailored to the user's emotional state.

[0155] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0156] Step 1:

[0157] The user operates the terminal and inputs specific work instructions. The terminal analyzes the user's facial expressions and voice using an emotion sensor and collects emotion data. The input consists of work instructions and emotion data. This data is encrypted and sent to the server.

[0158] Step 2:

[0159] The server receives emotional data and work instructions sent from the terminal. The emotion engine is activated to analyze the received data. In this process, the input emotional data is analyzed to determine the user's psychological state. The output is the analysis results, such as the user's stress level and level of reassurance.

[0160] Step 3:

[0161] The server uses a generative AI model to generate an operation plan for the work instruction based on the received analysis results. The generated plan is dynamically adjusted based on the user's emotional state. For example, if the user is anxious, a plan with more detailed steps will be generated. The adjusted operation plan is obtained as output.

[0162] Step 4:

[0163] The server transmits a coordinated operation plan to the machine. Based on the received command, the machine begins the specified task. During this process, the machine uses sensors to monitor its surroundings. The operation plan from the server is received as input, and the status of the task execution is generated as output.

[0164] Step 5:

[0165] If the machine detects environmental changes or unforeseen circumstances during operation, it feeds this information back to the server along with emotional data. The server then inputs this information back into the AI ​​model to generate new operation commands. The output is an updated work command.

[0166] Step 6:

[0167] Based on the updated work orders, the server checks the progress and provides feedback on the user's terminal. The progress is visually displayed to the user, incorporating features to provide reassurance. Feedback information is then generated for the user as output. This entire process optimizes work efficiency based on emotions.

[0168] (Application Example 2)

[0169] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0170] Conventional information delivery systems have difficulty providing information tailored to the emotional state of users, making it impossible to deliver information that is optimal for each individual user, thus hindering the improvement of the customer experience. Furthermore, the lack of flexible feedback functions that respond to users' emotions made it difficult to effectively improve user satisfaction.

[0171] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0172] In this invention, the server includes a receiving means for receiving operation commands, an emotion recognition means for collecting user emotion information using emotion recognition technology and providing information based on that information, and an information optimization means for optimizing and displaying information based on the user's emotions. This makes it possible to provide optimal information and effective feedback in real time according to the user's emotional state.

[0173] A "reception mechanism for receiving operation commands" is an interface that receives instruction information from the user and allows the system to start working based on that information.

[0174] "Planning generation means that generates an operation plan using a generative model" refers to a process that has an algorithm that determines the optimal work process based on specified conditions.

[0175] "Operating means for performing remote control of mechanical devices" refers to a system for controlling equipment located at a remote location and performing tasks according to instructions.

[0176] A "monitoring system for monitoring work progress and providing feedback" is a system for monitoring the current work status in real time and reporting the work status as needed.

[0177] "Adjustment mechanisms for adjusting operations in response to unforeseen circumstances" refer to functions that detect unexpected changes or failures and take appropriate measures to ensure the system continues to operate.

[0178] "An emotion recognition means that collects human emotion information using emotion recognition technology and provides information based on it" refers to a function that analyzes the user's emotional state from facial expressions, voice, etc., and provides information accordingly.

[0179] "Information optimization means that optimizes and displays information based on human emotions" refers to an algorithm that selects the most appropriate information according to the user's emotions and displays it on the user's device.

[0180] A system implementing this invention can achieve a superior user experience by providing and optimizing information based on the user's emotions. To achieve this, a server, terminal, and emotion recognition engine function as key elements.

[0181] The server first receives operation commands sent from the terminal through a receiving mechanism. Terminals equipped with emotion recognition technology use a camera and microphone to collect data such as the user's facial expressions and voice. This data is immediately transmitted to the server via a network protocol (e.g., Bluetooth or Wi-Fi).

[0182] On the server, an emotion recognition engine analyzes emotional data and uses a generative AI model to generate an optimal operation plan tailored to the user's emotions. By employing information optimization techniques, information and service content are selected based on the user's characteristics, and feedback is provided quickly. This makes it possible to provide appropriate information in real time to ensure user satisfaction.

[0183] As a concrete example, if the emotion recognition engine detects that a customer visiting a physical store appears slightly tired, the server sends a discount coupon for a drink usable in the store to the customer's device. This allows the customer to refresh themselves immediately, improving their satisfaction. An example of a prompt message to implement such a response would be, "Please advise on what products should be offered when a customer appears tired."

[0184] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0185] Step 1:

[0186] The device uses a camera to capture the user's facial expressions and a microphone to record their voice. The input data consists of facial image data and audio data, which are acquired as raw data for emotion recognition. This data is transmitted to a server via Bluetooth or Wi-Fi.

[0187] Step 2:

[0188] The server inputs the received facial image data and audio data into the emotion recognition engine. The emotion recognition engine uses a facial recognition algorithm and an audio emotion analysis algorithm to evaluate the user's emotional state. During the data analysis process, feature extraction and pattern matching are performed, and the output is a category of the emotional state (e.g., joy, surprise, fatigue).

[0189] Step 3:

[0190] The server generates an optimal action plan using a generative AI model based on the obtained emotional state categories. In this process, the AI ​​derives an appropriate response based on the prompt text. The generative model performs computational processing to select product and service information that matches the user's emotions, and as a result outputs customized information.

[0191] Step 4:

[0192] The server sends the generated operation plan and selected information to the terminal. The output information includes product information and promotional information displayed on the user's terminal. This allows the user to receive personalized information in real time and consider specific actions (e.g., purchasing a product, using a service).

[0193] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0194] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0195] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0196] [Second Embodiment]

[0197] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0198] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0199] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0200] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0201] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0202] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0203] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0204] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0205] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0206] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0207] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0208] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0209] Embodiments of the present invention will now be described. This system mainly consists of a server, a terminal, and heavy machinery, and is a mechanism that enables remote control using a generative model.

[0210] First, users issue work instructions using their devices. Specifically, they use smartphones or PCs to input work details, work location information, and desired completion time, thereby sending operational instructions to the system. This information is received by the server.

[0211] The server analyzes the work instructions received from the user and generates an optimal work plan using a generative model. This plan is automatically created by AI based on past performance data and environmental information. The server then determines which heavy machinery is best suited for use according to this plan and sends work instructions to that machinery.

[0212] The heavy machinery autonomously begins work based on instructions from the server. Utilizing its onboard sensors, the machinery accurately performs the instructed tasks while monitoring its surroundings in real time. If an unexpected problem occurs during operation, the server immediately analyzes the information and provides new instructions through a generative model, enabling a rapid response.

[0213] As a concrete example, consider a case where a user requests snow removal for a parking lot in a certain area. The user sets the location of the parking lot and the desired snow removal time on a terminal and issues a command. The server calculates the optimal route for snow removal and issues instructions to heavy machinery. The heavy machinery follows the instructions, autonomously begins snow removal work, and sends a completion report to the server. Upon receiving this completion report, the server notifies the user that the work has been completed. In this way, this system solves the problems of an elderly workforce and labor shortages, and enables efficient 24-hour operation.

[0214] The following describes the processing flow.

[0215] Step 1:

[0216] The user uses a terminal to input work instructions. Specifically, they set the location information of the area where snow removal work is to be performed, the work details, and the desired completion time, and send this information to the system. This information is then transmitted to the server.

[0217] Step 2:

[0218] The server analyzes the work instructions received from the user and creates a work plan using a generative model. The plan includes the optimal work route and motion sequence, as well as the selection of heavy machinery. At this stage, the AI ​​analyzes historical data and current environmental information to generate an efficient plan.

[0219] Step 3:

[0220] Based on the plan, the server generates specific work instructions for the appropriate heavy machinery and prepares them for remote operation. These instructions include detailed operational instructions such as route, speed, and operating procedures. These are then transmitted to the heavy machinery's control system.

[0221] Step 4:

[0222] The heavy machinery autonomously begins work based on instructions received from the server. Onboard sensors monitor the surrounding environment in real time, and the machinery performs its work while proceeding along the designated route. If any obstacles or unexpected situations occur during operation, the heavy machinery reports them to the server via data from the sensors.

[0223] Step 5:

[0224] The server analyzes data and feedback from the heavy machinery and adjusts instructions according to the new situation. It makes optimal route changes and speed adjustments in response to unforeseen circumstances and obstacles, and sends revised instructions to the heavy machinery.

[0225] Step 6:

[0226] Once the work is complete, the heavy machinery sends a completion report to the server. The server receives this report, confirms that the work was completed successfully, and then sends a completion notification to the user. This allows the user to confirm in real time that the snow removal was completed without any problems.

[0227] (Example 1)

[0228] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0229] Effective autonomous operation of machinery and equipment in remote locations, along with real-time progress feedback, requires sophisticated work planning and rapid fault response capabilities. However, in conventional systems, these processes are often performed manually, resulting in low work efficiency and delays in responding to unforeseen circumstances.

[0230] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0231] In this invention, the server includes a reception means using an information terminal to receive operation commands, a plan generation means that generates an operation plan based on past performance data and environmental information using a generation artificial intelligence model, and an operation means that transmits work instructions to selected heavy machinery using wireless communication technology. This enables autonomous operation of machinery and real-time monitoring of work status, resulting in efficient and rapid response.

[0232] An "operation command" is instructional information that specifies the exact tasks and conditions that a user should perform on a machine or device.

[0233] An "information terminal" is a technological device used by a user to input operating commands, and includes, for example, smartphones and personal computers.

[0234] A "reception method" is a technique that has the function of receiving operation commands transmitted from an information terminal and storing them in a database necessary for processing.

[0235] A "generative artificial intelligence model" is an AI model trained using algorithms based on a vast amount of historical data, and is a means of creating the optimal operation plan under specific conditions.

[0236] An "operation plan" is a plan of specific work processes and sequences that a machine should perform, generated by a generative artificial intelligence model.

[0237] "Plan generation means" refers to a function or system for formulating an operation plan using an artificial intelligence model based on operation commands.

[0238] "Heavy machinery" refers to large mechanical devices used to perform work in designated locations, and includes excavators, snowplows, and other similar equipment.

[0239] "Wireless communication technology" refers to methods for sending and receiving data between electronic devices without using cables, and examples include Wi-Fi and Bluetooth.

[0240] "Operation means" refers to a method of remotely controlling heavy machinery by issuing instructions based on a generated operation plan.

[0241] "Feedback" is an information transmission process that reports information about the progress of a machine's operation to a server in real time.

[0242] "Monitoring" is the process of continuously observing and recording the surrounding conditions and work progress of machinery and equipment using sensors.

[0243] "Operation adjustment means" refers to a function for replanning operations in order to remove factors that hinder the progress of work when unforeseen circumstances occur.

[0244] This invention provides a method for efficiently and autonomously executing user work commands using a system comprising an information terminal, a computer server, and a remotely controllable mechanical device. The embodiments for carrying out this invention are described in detail below.

[0245] Users input work instructions using information terminals such as smartphones or personal computers. These work instructions include information such as the content of the specified work, geographical area, and desired completion time. A dedicated application runs on the terminal and securely transmits the information entered by the user to the server. This process uses the SSL / TLS protocol to ensure data security.

[0246] The server generates an optimal work plan based on the received work instructions, utilizing a generative artificial intelligence model. This AI model is trained using machine learning libraries such as TensorFlow and analyzes historical and current environmental data. This enables rapid and highly accurate plan generation.

[0247] The generated work plan is transmitted to the selected heavy machinery via wireless communication technology. The heavy machinery is equipped with LiDAR sensors and cameras, which are used to monitor the surrounding environment during operation. This ensures that the work is executed accurately, and the progress is constantly fed back to the server.

[0248] As a concrete example, if a user requests snow removal work at a parking lot in a certain area, the user uses a terminal to input the location of the parking lot and the desired work time, and then issues a command. Based on this information, the server calculates the optimal snow removal route and sends instructions to the heavy machinery. The heavy machinery starts working according to the instructions and reports to the server upon completion. Upon receiving this report, the server notifies the user that the work has been completed.

[0249] An example of an input prompt for the generating AI model would be: "Optimize snow removal operations for the specified parking lot. The location information is [specific location], and the desired time is [specific time]."

[0250] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0251] Step 1:

[0252] The user enters work instructions using a terminal. Specifically, they launch a dedicated application and enter the work details, location information of the target area, and desired completion time into a form. Once this information is entered, the application encrypts the data and sends it to the server using a secure protocol (SSL / TLS). The input data generates output as a work instruction.

[0253] Step 2:

[0254] The server receives work instructions from the terminal and saves their contents to the database. The received data is first validated. This process checks for any errors in the data format or input content. Once validation is complete, the content is output as valid data for processing.

[0255] Step 3:

[0256] The server invokes a generating AI model based on the received work instructions to generate an optimal work plan. At this stage, the AI ​​model receives historical work data and current environmental data as input and performs data analysis and machine learning. This results in an optimized plan of necessary work steps and resources to be used.

[0257] Step 4:

[0258] The server selects the heavy machinery to be used based on the created work plan. It evaluates the operating status and location of the heavy machinery in real time and selects the optimal machine. Once the selection is complete, it sends work instructions to the heavy machinery using wireless communication technology (e.g., MQTT protocol). The output contains information on the selected heavy machinery and the work instructions.

[0259] Step 5:

[0260] The heavy machinery begins work according to instructions sent from the server. Using LiDAR sensors and cameras mounted on the machinery, it performs the instructed tasks while monitoring the surrounding environment in real time. This monitoring data is returned to the server as progress information during the work. The output includes work progress and environmental information.

[0261] Step 6:

[0262] Once the work is completed, the heavy machinery reports this information to the server. The server receives the report, records the work results in the database, and notifies the user based on the prompt message. The output consists of confirmation of work completion and a notification to the user.

[0263] (Application Example 1)

[0264] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0265] Conventional autonomous assembly systems suffered from insufficient optimization of work plans, making it difficult to improve the efficiency and flexibility of assembly operations. Furthermore, they lacked the means to quickly respond to obstacles and changing environments, creating a need for improved productivity and safety.

[0266] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0267] In this invention, the server includes a receiving means for receiving operation commands, a planning means for generating an operation plan using a generative model, and an optimization means for using a generative model to optimize assembly work. This enables efficient and safe assembly work by an autonomous device.

[0268] A "reception mechanism for receiving operation commands" is a function that receives operation instructions entered by the user and converts them into a format that can be processed within the system.

[0269] "Planning generation means for generating operation plans using generative models" refers to a function that automatically generates optimal work schedules and procedures using AI models.

[0270] "Optimization methods using generative models to optimize assembly work" refers to a function that analyzes work processes using an AI model and identifies areas for improvement in order to achieve efficient assembly work.

[0271] "Operating means for performing remote control on a machine" refers to a function that remotely controls and operates a machine based on instructions from a server.

[0272] A "monitoring method that monitors the progress of work and provides feedback" is a function that constantly monitors the status of work and immediately provides necessary information back to the user or system.

[0273] "Adjustment mechanisms for adjusting operations in response to unforeseen circumstances" refer to functions that quickly correct the system's response to sudden failures or changes, thereby ensuring the continuation of normal operations.

[0274] An "autonomous device" is a device that can perform tasks automatically without external intervention.

[0275] The "real-time optimization function" is a feature that instantly derives and applies the optimal work procedure based on information obtained during the work process.

[0276] The system for realizing this invention consists of a server, a terminal, and an autonomous device. It begins with the user inputting operation commands using the terminal and sending them to the server. These operation commands include details such as the task content and deadline. The server receives this information and uses a generative AI model to create an optimal work plan. This plan is designed to maximize work efficiency, and the generative model analyzes historical data and real-time environmental information.

[0277] The server sends specific instructions to the autonomous device based on the plan. The autonomous device receives the instructions from the server and automatically starts the assembly work. The device is equipped with sensors that can sequentially grasp the surrounding environment. The data from the sensors monitors the progress of the work in real time and is sent to the server as feedback. If an unexpected situation occurs during the work, the server uses the generated AI model to immediately generate new instructions and adjust the operation of the device.

[0278] As a specific example, consider the case of efficiently assembling bolts and nuts on an automobile parts assembly line. In this case, by sending a prompt sentence "Please generate an efficient combination pattern for assembling the automobile engine." from the terminal, the server can formulate a work plan and optimize the work in real time. With such a configuration, the server can manage the work efficiently and safely, and improve the productivity of the entire system.

[0279] The flow of the specific process in Application Example 1 will be described using FIG. 12.

[0280] Step 1:

[0281] The user inputs an operation command using the terminal. The information input is the work content, work start time, destination, etc., and the terminal sends these data to the server.

[0282] Step 2:

[0283] The server analyzes the operation command received from the user. Based on the input data, it automatically generates an optimal work plan using the generated AI model. Considering past work history data and environmental information, it formulates an efficient work schedule. The generated work plan is obtained as the output.

[0284] Step 3:

[0285] Based on the generated work plan, the server creates and sends specific operation instructions to the autonomous device. The operation instructions include details such as which parts should be assembled in which order. When this instruction reaches the autonomous device side, an instruction list is generated as output.

[0286] Step 4:

[0287] The autonomous device starts the assembly work according to the operation instructions received from the server. The sensors installed on the device detect the surrounding environment and send the progress information to the server in real time. The input is the operation instructions, and the output is the progress status.

[0288] Step 5:

[0289] If an unexpected situation occurs during the work, the server analyzes the progress information and quickly generates new operation instructions by the generated AI model. The input is the progress information and the data of the unexpected situation, and the output is the updated operation instructions. The generated instructions are immediately sent to the autonomous device to adjust the work to proceed smoothly.

[0290] Furthermore, an emotion engine for estimating the user's emotion may be combined. That is, the specific processing unit 290 may estimate the user's emotion using the emotion recognition model 59 and perform specific processing using the user's emotion.

[0291] Embodiments of the present invention will be described. This system includes a function of recognizing the user's emotion and dynamically adjusting the operation plan based on it. The main components are a server, a terminal, a mechanical device, and an emotion engine.

[0292] The user inputs a work instruction via the terminal, and at that time, the emotion sensor reads the emotion from the user's expression and voice. This emotion data is sent to the server together with the work instruction. By using emotion recognition technology, it is possible to formulate an operation plan that reflects the user's stress and expectation value.

[0293] The server uses an emotion engine to analyze the user's current emotional state and fine-tunes the operation plan derived from the generative model based on the results. For example, if the user is anxious, the server can provide detailed reports on the progress of the work and share real-time progress to provide reassurance. At this stage, the server selects heavy machinery suitable for the designated work area and prepares the specific instructions necessary for remote operation.

[0294] Next, the server sends work instructions to the heavy machinery, which then autonomously begins work based on the instructions from the server. During the work, various sensors on the heavy machinery monitor the surrounding environment, avoiding obstacles and proceeding with the work as planned. If an unforeseen situation occurs, a situation report including emotional data is fed back to the server, and the server uses a generative model to issue new work instructions. In this process, changes in the user's emotions are also taken into consideration, and feedback is adjusted as needed.

[0295] For example, when a user feels a sense of urgency during snow removal work in a store's parking lot during heavy snowfall, the system quickly creates a plan and visualizes the work status as needed to reassure the user. Thus, a key point of this invention is to achieve safe and efficient work execution through emotionally sensitive feedback and work adjustments.

[0296] The following describes the processing flow.

[0297] Step 1:

[0298] The user uses a terminal to input work instructions and sends the work area and desired completion time to the system. Additionally, an emotion sensor built into the terminal monitors the user's facial expressions and voice, collecting emotional data at that moment. This data is then sent to a server.

[0299] Step 2:

[0300] The server receives a work instruction and emotion data from the user. It analyzes the received emotion data with an emotion engine to determine the user's emotional state. For example, if the user's voice contains uneasiness or tension, this is detected and recorded as the emotional state.

[0301] Step 3:

[0302] The server creates a normal operation plan using a generation model and appropriately adjusts the plan by reflecting the analysis result of the emotion engine. Specifically, measures such as increasing information sharing at important decision points are taken to reduce the user's uneasiness as much as possible.

[0303] Step 4:

[0304] The server sends the adjusted operation instructions to the heavy machinery. The operation instructions include the route, operating speed, avoidance procedures, etc. Here, the plan is optimized in a way that takes into account the user's emotions.

[0305] Step 5:

[0306] The heavy machinery receives the instructions from the server and autonomously starts working. During the work, the heavy machinery monitors the surrounding environment with sensors and reports the situation to the server in real time. Adjustments necessary for the progress of the work are continuously made in combination with the emotion data.

[0307] Step 6:

[0308] When the work is completed, the heavy machinery reports the completion to the server. The server confirms this and provides the user with a notice of work completion and feedback based on the emotion. As a result, the user can confirm the completion of the work with confidence.

[0309] (Example 2)

[0310] Next, Example 2 will be described. In the following description, the data processing device 12 is referred to as the "server", and the smart glasses 214 are referred to as the "terminal".

[0311] Conventional remote control systems have problems such as failing to consider the user's psychological state and not adequately ensuring work efficiency and safety. Furthermore, the lack of real-time interaction between operation instructions and actual machine operation makes users prone to anxiety and stress. In addition, unforeseen circumstances often require simple manual intervention, placing a burden on the user.

[0312] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0313] In this invention, the server includes a receiving means for receiving operation commands, a plan generation means for generating operation plans using a generative model and dynamically adjusting them based on the user's emotions, and an emotion recognition means for analyzing the user's facial expressions and voice to collect emotion data. This enables real-time adjustment of operation plans that take the user's emotions into consideration, improving work efficiency and safety, and allowing for a rapid response to unforeseen circumstances.

[0314] An "operation command" refers to a specific work instruction that a user sends to the system via their terminal.

[0315] A "generative model" is a type of artificial intelligence model used by systems to create operation plans, enabling dynamic plan generation based on data.

[0316] A "plan generation means" is a component of a system that has the function of generating an operation plan and adjusting it based on the user's emotions.

[0317] "Emotion recognition means" refers to a component of a system that collects emotional data by analyzing the user's facial expressions and voice.

[0318] "Operating means" refers to the function of a system that performs instructed operations in order to remotely control a machine or device.

[0319] "Monitoring means" refers to a system function that monitors the progress of work and provides users with real-time feedback on the progress.

[0320] A "regulation mechanism" is a component of a system that appropriately adjusts its operation in the event of unforeseen circumstances and provides feedback based on changes in the user's emotions.

[0321] The system of this invention analyzes the user's emotions and dynamically adjusts the operation plan based on them. It mainly consists of a server, terminals, mechanical devices, and software that links them together.

[0322] The terminal provides an interface for receiving operation commands entered by the user. The terminal is equipped with an emotion sensor that analyzes the user's facial expressions and voice in real time, thereby collecting emotion data. The collected emotion data is securely encrypted and transmitted to the server.

[0323] The server uses a generative AI model to generate and dynamically adjust operation plans based on the user's emotional state. It utilizes emotion recognition techniques to analyze the user's stress levels, sense of security, and other states, and modifies the generated plan accordingly. For example, if the user is feeling anxious, the operation plan can be modified to provide more detailed progress updates.

[0324] The machinery receives operation commands transmitted from the server and begins operating autonomously. Heavy machinery, such as equipment, constantly monitors its surroundings using various sensors to perform tasks safely. In the event of an unforeseen situation, the situation, including emotional data, is quickly fed back to the server, and new work instructions are generated.

[0325] A concrete example would be a situation where a user is anxious about bad weather during garden maintenance work. In this case, the server would visually display the work progress to reassure the user. An example of a prompt to the generated AI model would be, "If the user is perceived as anxious, please tell me how to display the progress in detail," which would allow for appropriate feedback.

[0326] This system makes it possible to perform tasks safely and efficiently, tailored to the user's emotional state.

[0327] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0328] Step 1:

[0329] The user operates the terminal and inputs specific work instructions. The terminal analyzes the user's facial expressions and voice using an emotion sensor and collects emotion data. The input consists of work instructions and emotion data. This data is encrypted and sent to the server.

[0330] Step 2:

[0331] The server receives emotional data and work instructions sent from the terminal. The emotion engine is activated to analyze the received data. In this process, the input emotional data is analyzed to determine the user's psychological state. The output is the analysis results, such as the user's stress level and level of reassurance.

[0332] Step 3:

[0333] The server uses a generative AI model to generate an operation plan for the work instruction based on the received analysis results. The generated plan is dynamically adjusted based on the user's emotional state. For example, if the user is anxious, a plan with more detailed steps will be generated. The adjusted operation plan is obtained as output.

[0334] Step 4:

[0335] The server transmits a coordinated operation plan to the machine. Based on the received command, the machine begins the specified task. During this process, the machine uses sensors to monitor its surroundings. The operation plan from the server is received as input, and the status of the task execution is generated as output.

[0336] Step 5:

[0337] If the machine detects environmental changes or unforeseen circumstances during operation, it feeds this information back to the server along with emotional data. The server then inputs this information back into the AI ​​model to generate new operation commands. The output is an updated work command.

[0338] Step 6:

[0339] Based on the updated work orders, the server checks the progress and provides feedback on the user's terminal. The progress is visually displayed to the user, incorporating features to provide reassurance. Feedback information is then generated for the user as output. This entire process optimizes work efficiency based on emotions.

[0340] (Application Example 2)

[0341] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0342] Conventional information delivery systems have difficulty providing information tailored to the emotional state of users, making it impossible to deliver information that is optimal for each individual user, thus hindering the improvement of the customer experience. Furthermore, the lack of flexible feedback functions that respond to users' emotions made it difficult to effectively improve user satisfaction.

[0343] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0344] In this invention, the server includes a receiving means for receiving operation commands, an emotion recognition means for collecting user emotion information using emotion recognition technology and providing information based on that information, and an information optimization means for optimizing and displaying information based on the user's emotions. This makes it possible to provide optimal information and effective feedback in real time according to the user's emotional state.

[0345] A "reception mechanism for receiving operation commands" is an interface that receives instruction information from the user and allows the system to start working based on that information.

[0346] "Planning generation means that generates an operation plan using a generative model" refers to a process that has an algorithm that determines the optimal work process based on specified conditions.

[0347] "Operating means for performing remote control of mechanical devices" refers to a system for controlling equipment located at a remote location and performing tasks according to instructions.

[0348] A "monitoring system for monitoring work progress and providing feedback" is a system for monitoring the current work status in real time and reporting the work status as needed.

[0349] "Adjustment mechanisms for adjusting operations in response to unforeseen circumstances" refer to functions that detect unexpected changes or failures and take appropriate measures to ensure the system continues to operate.

[0350] "An emotion recognition means that collects human emotion information using emotion recognition technology and provides information based on it" refers to a function that analyzes the user's emotional state from facial expressions, voice, etc., and provides information accordingly.

[0351] "Information optimization means that optimizes and displays information based on human emotions" refers to an algorithm that selects the most appropriate information according to the user's emotions and displays it on the user's device.

[0352] A system implementing this invention can achieve a superior user experience by providing and optimizing information based on the user's emotions. To achieve this, a server, terminal, and emotion recognition engine function as key elements.

[0353] The server first receives operation commands sent from the terminal through a receiving mechanism. Terminals equipped with emotion recognition technology use a camera and microphone to collect data such as the user's facial expressions and voice. This data is immediately transmitted to the server via a network protocol (e.g., Bluetooth or Wi-Fi).

[0354] On the server, an emotion recognition engine analyzes emotional data and uses a generative AI model to generate an optimal operation plan tailored to the user's emotions. By employing information optimization techniques, information and service content are selected based on the user's characteristics, and feedback is provided quickly. This makes it possible to provide appropriate information in real time to ensure user satisfaction.

[0355] As a concrete example, if the emotion recognition engine detects that a customer visiting a physical store appears slightly tired, the server sends a discount coupon for a drink usable in the store to the customer's device. This allows the customer to refresh themselves immediately, improving their satisfaction. An example of a prompt message to implement such a response would be, "Please advise on what products should be offered when a customer appears tired."

[0356] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0357] Step 1:

[0358] The device uses a camera to capture the user's facial expressions and a microphone to record their voice. The input data consists of facial image data and audio data, which are acquired as raw data for emotion recognition. This data is transmitted to a server via Bluetooth or Wi-Fi.

[0359] Step 2:

[0360] The server inputs the received facial image data and audio data into the emotion recognition engine. The emotion recognition engine uses a facial recognition algorithm and an audio emotion analysis algorithm to evaluate the user's emotional state. During the data analysis process, feature extraction and pattern matching are performed, and the output is a category of the emotional state (e.g., joy, surprise, fatigue).

[0361] Step 3:

[0362] The server generates an optimal action plan using a generative AI model based on the obtained emotional state categories. In this process, the AI ​​derives an appropriate response based on the prompt text. The generative model performs computational processing to select product and service information that matches the user's emotions, and as a result outputs customized information.

[0363] Step 4:

[0364] The server sends the generated operation plan and selected information to the terminal. The output information includes product information and promotional information displayed on the user's terminal. This allows the user to receive personalized information in real time and consider specific actions (e.g., purchasing a product, using a service).

[0365] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0366] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0367] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0368] [Third Embodiment]

[0369] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0370] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0371] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0372] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0373] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0374] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0375] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0376] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0377] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0378] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0379] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0380] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0381] Embodiments of the present invention will now be described. This system mainly consists of a server, a terminal, and heavy machinery, and is a mechanism that enables remote control using a generative model.

[0382] First, users issue work instructions using their devices. Specifically, they use smartphones or PCs to input work details, work location information, and desired completion time, thereby sending operational instructions to the system. This information is received by the server.

[0383] The server analyzes the work instructions received from the user and generates an optimal work plan using a generative model. This plan is automatically created by AI based on past performance data and environmental information. The server then determines which heavy machinery is best suited for use according to this plan and sends work instructions to that machinery.

[0384] The heavy machinery autonomously begins work based on instructions from the server. Utilizing its onboard sensors, the machinery accurately performs the instructed tasks while monitoring its surroundings in real time. If an unexpected problem occurs during operation, the server immediately analyzes the information and provides new instructions through a generative model, enabling a rapid response.

[0385] As a concrete example, consider a case where a user requests snow removal for a parking lot in a certain area. The user sets the location of the parking lot and the desired snow removal time on a terminal and issues a command. The server calculates the optimal route for snow removal and issues instructions to heavy machinery. The heavy machinery follows the instructions, autonomously begins snow removal work, and sends a completion report to the server. Upon receiving this completion report, the server notifies the user that the work has been completed. In this way, this system solves the problems of an elderly workforce and labor shortages, and enables efficient 24-hour operation.

[0386] The following describes the processing flow.

[0387] Step 1:

[0388] The user uses a terminal to input work instructions. Specifically, they set the location information of the area where snow removal work is to be performed, the work details, and the desired completion time, and send this information to the system. This information is then transmitted to the server.

[0389] Step 2:

[0390] The server analyzes the work instructions received from the user and creates a work plan using a generative model. The plan includes the optimal work route and motion sequence, as well as the selection of heavy machinery. At this stage, the AI ​​analyzes historical data and current environmental information to generate an efficient plan.

[0391] Step 3:

[0392] Based on the plan, the server generates specific work instructions for the appropriate heavy machinery and prepares them for remote operation. These instructions include detailed operational instructions such as route, speed, and operating procedures. These are then transmitted to the heavy machinery's control system.

[0393] Step 4:

[0394] The heavy machinery autonomously begins work based on instructions received from the server. Onboard sensors monitor the surrounding environment in real time, and the machinery performs its work while proceeding along the designated route. If any obstacles or unexpected situations occur during operation, the heavy machinery reports them to the server via data from the sensors.

[0395] Step 5:

[0396] The server analyzes data and feedback from the heavy machinery and adjusts instructions according to the new situation. It makes optimal route changes and speed adjustments in response to unforeseen circumstances and obstacles, and sends revised instructions to the heavy machinery.

[0397] Step 6:

[0398] Once the work is complete, the heavy machinery sends a completion report to the server. The server receives this report, confirms that the work was completed successfully, and then sends a completion notification to the user. This allows the user to confirm in real time that the snow removal was completed without any problems.

[0399] (Example 1)

[0400] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0401] Effective autonomous operation of machinery and equipment in remote locations, along with real-time progress feedback, requires sophisticated work planning and rapid fault response capabilities. However, in conventional systems, these processes are often performed manually, resulting in low work efficiency and delays in responding to unforeseen circumstances.

[0402] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0403] In this invention, the server includes a reception means using an information terminal to receive operation commands, a plan generation means that generates an operation plan based on past performance data and environmental information using a generation artificial intelligence model, and an operation means that transmits work instructions to selected heavy machinery using wireless communication technology. This enables autonomous operation of machinery and real-time monitoring of work status, resulting in efficient and rapid response.

[0404] An "operation command" is instructional information that specifies the exact tasks and conditions that a user should perform on a machine or device.

[0405] An "information terminal" is a technological device used by a user to input operating commands, and includes, for example, smartphones and personal computers.

[0406] A "reception method" is a technique that has the function of receiving operation commands transmitted from an information terminal and storing them in a database necessary for processing.

[0407] A "generative artificial intelligence model" is an AI model trained using algorithms based on a vast amount of historical data, and is a means of creating the optimal operation plan under specific conditions.

[0408] An "operation plan" is a plan of specific work processes and sequences that a machine should perform, generated by a generative artificial intelligence model.

[0409] "Plan generation means" refers to a function or system for formulating an operation plan using an artificial intelligence model based on operation commands.

[0410] "Heavy machinery" refers to large mechanical devices used to perform work in designated locations, and includes excavators, snowplows, and other similar equipment.

[0411] "Wireless communication technology" refers to methods for sending and receiving data between electronic devices without using cables, and examples include Wi-Fi and Bluetooth.

[0412] "Operation means" refers to a method of remotely controlling heavy machinery by issuing instructions based on a generated operation plan.

[0413] "Feedback" is an information transmission process that reports information about the progress of a machine's operation to a server in real time.

[0414] "Monitoring" is the process of continuously observing and recording the surrounding conditions and work progress of machinery and equipment using sensors.

[0415] "Operation adjustment means" refers to a function for replanning operations in order to remove factors that hinder the progress of work when unforeseen circumstances occur.

[0416] This invention provides a method for efficiently and autonomously executing user work commands using a system comprising an information terminal, a computer server, and a remotely controllable mechanical device. The embodiments for carrying out this invention are described in detail below.

[0417] Users input work instructions using information terminals such as smartphones or personal computers. These work instructions include information such as the content of the specified work, geographical area, and desired completion time. A dedicated application runs on the terminal and securely transmits the information entered by the user to the server. This process uses the SSL / TLS protocol to ensure data security.

[0418] The server generates an optimal work plan based on the received work instructions, utilizing a generative artificial intelligence model. This AI model is trained using machine learning libraries such as TensorFlow and analyzes historical and current environmental data. This enables rapid and highly accurate plan generation.

[0419] The generated work plan is transmitted to the selected heavy machinery via wireless communication technology. The heavy machinery is equipped with LiDAR sensors and cameras, which are used to monitor the surrounding environment during operation. This ensures that the work is executed accurately, and the progress is constantly fed back to the server.

[0420] As a concrete example, if a user requests snow removal work at a parking lot in a certain area, the user uses a terminal to input the location of the parking lot and the desired work time, and then issues a command. Based on this information, the server calculates the optimal snow removal route and sends instructions to the heavy machinery. The heavy machinery starts working according to the instructions and reports to the server upon completion. Upon receiving this report, the server notifies the user that the work has been completed.

[0421] An example of an input prompt for the generating AI model would be: "Optimize snow removal operations for the specified parking lot. The location information is [specific location], and the desired time is [specific time]."

[0422] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0423] Step 1:

[0424] The user enters work instructions using a terminal. Specifically, they launch a dedicated application and enter the work details, location information of the target area, and desired completion time into a form. Once this information is entered, the application encrypts the data and sends it to the server using a secure protocol (SSL / TLS). The input data generates output as a work instruction.

[0425] Step 2:

[0426] The server receives work instructions from the terminal and saves their contents to the database. The received data is first validated. This process checks for any errors in the data format or input content. Once validation is complete, the content is output as valid data for processing.

[0427] Step 3:

[0428] The server invokes a generating AI model based on the received work instructions to generate an optimal work plan. At this stage, the AI ​​model receives historical work data and current environmental data as input and performs data analysis and machine learning. This results in an optimized plan of necessary work steps and resources to be used.

[0429] Step 4:

[0430] The server selects the heavy machinery to be used based on the created work plan. It evaluates the operating status and location of the heavy machinery in real time and selects the optimal machine. Once the selection is complete, it sends work instructions to the heavy machinery using wireless communication technology (e.g., MQTT protocol). The output contains information on the selected heavy machinery and the work instructions.

[0431] Step 5:

[0432] The heavy machinery begins work according to instructions sent from the server. Using LiDAR sensors and cameras mounted on the machinery, it performs the instructed tasks while monitoring the surrounding environment in real time. This monitoring data is returned to the server as progress information during the work. The output includes work progress and environmental information.

[0433] Step 6:

[0434] Once the work is completed, the heavy machinery reports this information to the server. The server receives the report, records the work results in the database, and notifies the user based on the prompt message. The output consists of confirmation of work completion and a notification to the user.

[0435] (Application Example 1)

[0436] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0437] Conventional autonomous assembly systems suffered from insufficient optimization of work plans, making it difficult to improve the efficiency and flexibility of assembly operations. Furthermore, they lacked the means to quickly respond to obstacles and changing environments, thus creating a need for improved productivity and safety.

[0438] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0439] In this invention, the server includes a receiving means for receiving operation commands, a planning means for generating an operation plan using a generative model, and an optimization means for using a generative model to optimize assembly work. This enables efficient and safe assembly work by an autonomous device.

[0440] A "reception mechanism for receiving operation commands" is a function that receives operation instructions entered by the user and converts them into a format that can be processed within the system.

[0441] "Planning generation means for generating operation plans using generative models" refers to a function that automatically generates optimal work schedules and procedures using AI models.

[0442] "Optimization methods using generative models to optimize assembly work" refers to a function that analyzes work processes using AI models and identifies areas for improvement in order to achieve efficient assembly work.

[0443] "Operating means for performing remote control on a machine" refers to a function that remotely controls and operates a machine based on instructions from a server.

[0444] A "monitoring method that monitors the progress of work and provides feedback" is a function that constantly monitors the status of work and immediately provides necessary information back to the user or system.

[0445] "Adjustment mechanisms for adjusting operations in response to unforeseen circumstances" refer to functions that quickly correct the system's response to sudden failures or changes, thereby ensuring the continuation of normal operations.

[0446] An "autonomous device" is a device that can perform tasks automatically without external intervention.

[0447] The "real-time optimization function" is a feature that immediately derives and applies the optimal work procedure based on information obtained during the work process.

[0448] The system for realizing this invention consists of a server, a terminal, and an autonomous device. It begins with the user inputting operation commands using the terminal and sending them to the server. These operation commands include details such as the task content and deadline. The server receives this information and uses a generative AI model to create an optimal work plan. This plan is designed to maximize work efficiency, and the generative model analyzes historical data and real-time environmental information.

[0449] Based on the plan, the server sends specific instructions to the autonomous device. The autonomous device receives the instructions from the server and automatically begins the assembly work. The device is equipped with sensors that allow it to continuously monitor its surroundings. Data from the sensors is used to monitor the progress of the work in real time and is sent back to the server as feedback. If an unforeseen problem occurs during the work, the server uses a generative AI model to immediately generate new instructions and adjust the device's operation.

[0450] As a concrete example, consider the efficient assembly of bolts and nuts on an automobile parts assembly line. In this case, by sending a prompt message from a terminal saying, "Generate an efficient combination pattern for automobile engine assembly," the server can formulate a work plan and optimize the work in real time. With such a configuration, the server can manage the work efficiently and safely, improving the overall productivity of the system.

[0451] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0452] Step 1:

[0453] The user inputs operational commands using a terminal. The information entered includes the work details, start time, and destination, and the terminal transmits this data to the server.

[0454] Step 2:

[0455] The server analyzes the operation commands received from the user. Based on the input data, it automatically generates an optimal work plan using a generation AI model. It formulates an efficient work schedule by considering past work history data and environmental information. The generated work plan is obtained as output.

[0456] Step 3:

[0457] Based on the generated work plan, the server creates and sends specific operating instructions to the autonomous device. These instructions include details such as which parts should be assembled and in what order. Upon receiving these instructions, the autonomous device generates an instruction list as output.

[0458] Step 4:

[0459] The autonomous device begins assembly work according to the operation instructions received from the server. Sensors mounted on the device detect the surrounding environment and transmit progress information to the server in real time. The input is the operation instruction, and the output is the progress status.

[0460] Step 5:

[0461] If an unforeseen event occurs during operation, the server analyzes the progress information and quickly generates new operation instructions using a generation AI model. Inputs are progress information and data on unforeseen events, while output is the updated operation instructions. The generated instructions are immediately transmitted to the autonomous device and adjusted to ensure smooth operation.

[0462] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0463] Embodiments of the present invention will now be described. This system includes a function to recognize the user's emotions and dynamically adjust the operation plan based on those emotions. The main components are a server, a terminal, a mechanical device, and an emotion engine.

[0464] Users input work instructions via a terminal, and during this process, an emotion sensor reads the user's emotions from their facial expressions and voice. This emotion data is sent to the server along with the work instructions. By using emotion recognition technology, it is possible to create operation plans that reflect the user's stress levels and expectations.

[0465] The server uses an emotion engine to analyze the user's current emotional state and fine-tunes the operation plan derived from the generative model based on the results. For example, if the user is anxious, the server can provide detailed reports on the progress of the work and share real-time progress to provide reassurance. At this stage, the server selects heavy machinery suitable for the designated work area and prepares the specific instructions necessary for remote operation.

[0466] Next, the server sends work instructions to the heavy machinery, which then autonomously begins work based on the instructions from the server. During the work, various sensors on the heavy machinery monitor the surrounding environment, avoiding obstacles and proceeding with the work as planned. If an unforeseen situation occurs, a situation report including emotional data is fed back to the server, and the server uses a generative model to issue new work instructions. In this process, changes in the user's emotions are also taken into consideration, and feedback is adjusted as needed.

[0467] For example, when a user feels a sense of urgency during snow removal work in a store's parking lot during heavy snowfall, the system quickly creates a plan and visualizes the work status as needed to reassure the user. Thus, a key point of this invention is to achieve safe and efficient work execution through emotionally sensitive feedback and work adjustments.

[0468] The following describes the processing flow.

[0469] Step 1:

[0470] The user uses a terminal to input work instructions and sends the work area and desired completion time to the system. Additionally, an emotion sensor built into the terminal monitors the user's facial expressions and voice, collecting emotional data at that moment. This data is then sent to a server.

[0471] Step 2:

[0472] The server receives work instructions and emotional data from the user. The received emotional data is analyzed by an emotion engine to determine the user's emotional state. For example, if the user's voice contains anxiety or tension, this is detected and recorded as the emotional state.

[0473] Step 3:

[0474] The server uses a generative model to create a normal operation plan, but it adjusts the plan appropriately by incorporating the results of the emotion engine's analysis. Specifically, it takes measures such as increasing information sharing at important decision points to alleviate user anxiety as much as possible.

[0475] Step 4:

[0476] The server sends the adjusted operating instructions to the heavy machinery. These instructions include the route, operating speed, and avoidance procedures. The plan is optimized here, taking user sentiment into consideration.

[0477] Step 5:

[0478] The heavy machinery receives instructions from the server and autonomously begins work. During operation, the machinery monitors its surroundings with sensors and reports the situation to the server in real time. Adjustments necessary for the progress of the work are continuously made in conjunction with emotional data.

[0479] Step 6:

[0480] Once the work is complete, the heavy machinery reports completion to the server. The server confirms this and provides the user with a completion notification and sentiment-based feedback. This allows the user to confidently confirm that the work has been completed.

[0481] (Example 2)

[0482] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0483] Conventional remote control systems have problems such as failing to consider the user's psychological state and not adequately ensuring work efficiency and safety. Furthermore, the lack of real-time interaction between operation instructions and actual machine operation makes users prone to anxiety and stress. In addition, unforeseen circumstances often require simple manual intervention, placing a burden on the user.

[0484] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0485] In this invention, the server includes a receiving means for receiving operation commands, a plan generation means for generating operation plans using a generative model and dynamically adjusting them based on the user's emotions, and an emotion recognition means for analyzing the user's facial expressions and voice to collect emotion data. This enables real-time adjustment of operation plans that take the user's emotions into consideration, improving work efficiency and safety, and allowing for a rapid response to unforeseen circumstances.

[0486] An "operation command" refers to a specific work instruction that a user sends to the system via their terminal.

[0487] A "generative model" is a type of artificial intelligence model used by systems to create operation plans, enabling dynamic plan generation based on data.

[0488] A "plan generation means" is a component of a system that has the function of generating an operation plan and adjusting it based on the user's emotions.

[0489] "Emotion recognition means" refers to a component of a system that collects emotional data by analyzing the user's facial expressions and voice.

[0490] "Operating means" refers to the function of a system that performs instructed operations in order to remotely control a machine or device.

[0491] "Monitoring means" refers to a system function that monitors the progress of work and provides users with real-time feedback on the progress.

[0492] A "regulation mechanism" is a component of a system that appropriately adjusts its operation in the event of unforeseen circumstances and provides feedback based on changes in the user's emotions.

[0493] The system of this invention analyzes the user's emotions and dynamically adjusts the operation plan based on them. It mainly consists of a server, terminals, mechanical devices, and software that links them together.

[0494] The terminal provides an interface for receiving operation commands entered by the user. The terminal is equipped with an emotion sensor that analyzes the user's facial expressions and voice in real time, thereby collecting emotion data. The collected emotion data is securely encrypted and transmitted to the server.

[0495] The server uses a generative AI model to generate and dynamically adjust operation plans based on the user's emotional state. It utilizes emotion recognition techniques to analyze the user's stress levels, sense of security, and other states, and modifies the generated plan accordingly. For example, if the user is feeling anxious, the operation plan can be modified to provide more detailed progress updates.

[0496] The machinery receives operation commands transmitted from the server and begins operating autonomously. Heavy machinery, such as equipment, constantly monitors its surroundings using various sensors to perform tasks safely. In the event of an unforeseen situation, the situation, including emotional data, is quickly fed back to the server, and new work instructions are generated.

[0497] A concrete example would be a situation where a user is anxious about bad weather during garden maintenance work. In this case, the server would visually display the work progress to reassure the user. An example of a prompt to the generated AI model would be, "If the user is perceived as anxious, please tell me how to display the progress in detail," which would allow for appropriate feedback.

[0498] This system makes it possible to perform tasks safely and efficiently, tailored to the user's emotional state.

[0499] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0500] Step 1:

[0501] The user operates the terminal and inputs specific work instructions. The terminal analyzes the user's facial expressions and voice using an emotion sensor and collects emotion data. The input consists of work instructions and emotion data. This data is encrypted and sent to the server.

[0502] Step 2:

[0503] The server receives emotional data and work instructions sent from the terminal. The emotion engine is activated to analyze the received data. In this process, the input emotional data is analyzed to determine the user's psychological state. The output is the analysis results, such as the user's stress level and level of reassurance.

[0504] Step 3:

[0505] The server uses a generative AI model to generate an operation plan for the work instruction based on the received analysis results. The generated plan is dynamically adjusted based on the user's emotional state. For example, if the user is anxious, a plan with more detailed steps will be generated. The adjusted operation plan is obtained as output.

[0506] Step 4:

[0507] The server transmits a coordinated operation plan to the machine. Based on the received command, the machine begins the specified task. During this process, the machine uses sensors to monitor its surroundings. The operation plan from the server is taken as input, and the status of the task execution is generated as output.

[0508] Step 5:

[0509] If the machine detects environmental changes or unforeseen circumstances during operation, it feeds this information back to the server along with emotional data. The server then inputs this information back into the AI ​​model to generate new operation commands. The output is an updated work command.

[0510] Step 6:

[0511] Based on the updated work orders, the server checks the progress and provides feedback on the user's terminal. The progress is visually displayed to the user, incorporating features to provide reassurance. Feedback information is then generated for the user as output. This entire process optimizes work efficiency based on emotions.

[0512] (Application Example 2)

[0513] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0514] Conventional information delivery systems have difficulty providing information tailored to the emotional state of users, making it impossible to deliver information that is optimal for each individual user, thus hindering the improvement of the customer experience. Furthermore, the lack of flexible feedback functions that respond to users' emotions made it difficult to effectively improve user satisfaction.

[0515] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0516] In this invention, the server includes a receiving means for receiving operation commands, an emotion recognition means for collecting user emotion information using emotion recognition technology and providing information based on that information, and an information optimization means for optimizing and displaying information based on the user's emotions. This makes it possible to provide optimal information and effective feedback in real time according to the user's emotional state.

[0517] A "reception mechanism for receiving operation commands" is an interface that receives instruction information from the user and allows the system to start working based on that information.

[0518] "Planning generation means that generates an operation plan using a generative model" refers to a process that has an algorithm that determines the optimal work process based on specified conditions.

[0519] "Operating means for performing remote control of mechanical devices" refers to a system for controlling equipment located at a remote location and performing tasks according to instructions.

[0520] A "monitoring system for monitoring work progress and providing feedback" is a system for monitoring the current work status in real time and reporting the work status as needed.

[0521] "Adjustment mechanisms for adjusting operations in response to unforeseen circumstances" refer to functions that detect unexpected changes or failures and take appropriate measures to ensure the system continues to operate.

[0522] "An emotion recognition means that collects human emotion information using emotion recognition technology and provides information based on it" refers to a function that analyzes the user's emotional state from facial expressions, voice, etc., and provides information accordingly.

[0523] "Information optimization means that optimizes and displays information based on human emotions" refers to an algorithm that selects the most appropriate information according to the user's emotions and displays it on the user's device.

[0524] A system implementing this invention can achieve a superior user experience by providing and optimizing information based on the user's emotions. To achieve this, a server, terminal, and emotion recognition engine function as key elements.

[0525] The server first receives operation commands sent from the terminal through a receiving mechanism. Terminals equipped with emotion recognition technology use a camera and microphone to collect data such as the user's facial expressions and voice. This data is immediately transmitted to the server via a network protocol (e.g., Bluetooth or Wi-Fi).

[0526] On the server, an emotion recognition engine analyzes emotional data and uses a generative AI model to generate an optimal operation plan tailored to the user's emotions. By employing information optimization techniques, information and service content are selected based on the user's characteristics, and feedback is provided quickly. This makes it possible to provide appropriate information in real time to ensure user satisfaction.

[0527] As a concrete example, if the emotion recognition engine detects that a customer visiting a physical store appears slightly tired, the server sends a discount coupon for a drink usable in the store to the customer's device. This allows the customer to refresh themselves immediately, improving their satisfaction. An example of a prompt message to implement such a response would be, "Please advise on what products should be offered when a customer appears tired."

[0528] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0529] Step 1:

[0530] The device uses a camera to capture the user's facial expressions and a microphone to record their voice. The input data consists of facial image data and audio data, which are acquired as raw data for emotion recognition. This data is transmitted to a server via Bluetooth or Wi-Fi.

[0531] Step 2:

[0532] The server inputs the received facial image data and audio data into the emotion recognition engine. The emotion recognition engine uses a facial recognition algorithm and an audio emotion analysis algorithm to evaluate the user's emotional state. During the data analysis process, feature extraction and pattern matching are performed, and the output is a category of the emotional state (e.g., joy, surprise, fatigue).

[0533] Step 3:

[0534] The server generates an optimal action plan using a generative AI model based on the obtained emotional state categories. In this process, the AI ​​derives an appropriate response based on the prompt text. The generative model performs computational processing to select product and service information that matches the user's emotions, and as a result outputs customized information.

[0535] Step 4:

[0536] The server sends the generated operation plan and selected information to the terminal. The output information includes product information and promotional information displayed on the user's terminal. This allows the user to receive personalized information in real time and consider specific actions (e.g., purchasing a product, using a service).

[0537] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0538] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet Search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0539] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[0540] [Fourth Embodiment]

[0541] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0542] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0543] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0544] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[0545] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0546] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0547] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0548] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0549] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0550] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0551] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0552] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0553] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0554] Embodiments of the present invention will now be described. This system mainly consists of a server, a terminal, and heavy machinery, and is a mechanism that enables remote control using a generative model.

[0555] First, users issue work instructions using their devices. Specifically, they use smartphones or PCs to input work details, work location information, and desired completion time, thereby sending operational instructions to the system. This information is received by the server.

[0556] The server analyzes the work instructions received from the user and generates an optimal work plan using a generative model. This plan is automatically created by AI based on past performance data and environmental information. The server then determines which heavy machinery is best suited for use according to this plan and sends work instructions to that machinery.

[0557] The heavy machinery autonomously begins work based on instructions from the server. Utilizing its onboard sensors, the machinery accurately performs the instructed tasks while monitoring its surroundings in real time. If an unexpected problem occurs during operation, the server immediately analyzes the information and provides new instructions through a generative model, enabling a rapid response.

[0558] As a concrete example, consider a case where a user requests snow removal for a parking lot in a certain area. The user sets the location of the parking lot and the desired snow removal time on a terminal and issues a command. The server calculates the optimal route for snow removal and issues instructions to heavy machinery. The heavy machinery follows the instructions, autonomously begins snow removal work, and sends a completion report to the server. Upon receiving this completion report, the server notifies the user that the work has been completed. In this way, this system solves the problems of an elderly workforce and labor shortages, and enables efficient 24-hour operation.

[0559] The following describes the processing flow.

[0560] Step 1:

[0561] The user uses a terminal to input work instructions. Specifically, they set the location information of the area where snow removal work is to be performed, the work details, and the desired completion time, and send this information to the system. This information is then transmitted to the server.

[0562] Step 2:

[0563] The server analyzes the work instructions received from the user and creates a work plan using a generative model. The plan includes the optimal work route and motion sequence, as well as the selection of heavy machinery. At this stage, the AI ​​analyzes historical data and current environmental information to generate an efficient plan.

[0564] Step 3:

[0565] Based on the plan, the server generates specific work instructions for the appropriate heavy machinery and prepares them for remote operation. These instructions include detailed operational instructions such as route, speed, and operating procedures. These are then transmitted to the heavy machinery's control system.

[0566] Step 4:

[0567] The heavy machinery autonomously begins work based on instructions received from the server. Onboard sensors monitor the surrounding environment in real time, and the machinery performs its work while proceeding along the designated route. If any obstacles or unexpected situations occur during operation, the heavy machinery reports them to the server via data from the sensors.

[0568] Step 5:

[0569] The server analyzes data and feedback from the heavy machinery and adjusts instructions according to the new situation. It makes optimal route changes and speed adjustments in response to unforeseen circumstances and obstacles, and sends revised instructions to the heavy machinery.

[0570] Step 6:

[0571] Once the work is complete, the heavy machinery sends a completion report to the server. The server receives this report, confirms that the work was completed successfully, and then sends a completion notification to the user. This allows the user to confirm in real time that the snow removal was completed without any problems.

[0572] (Example 1)

[0573] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0574] Effective autonomous operation of machinery and equipment in remote locations, along with real-time progress feedback, requires sophisticated work planning and rapid fault response capabilities. However, in conventional systems, these processes are often performed manually, resulting in low work efficiency and delays in responding to unforeseen circumstances.

[0575] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0576] In this invention, the server includes a reception means using an information terminal to receive operation commands, a plan generation means that generates an operation plan based on past performance data and environmental information using a generation artificial intelligence model, and an operation means that transmits work instructions to selected heavy machinery using wireless communication technology. This enables autonomous operation of machinery and real-time monitoring of work status, resulting in efficient and rapid response.

[0577] An "operation command" is instructional information that specifies the exact tasks and conditions that a user should perform on a machine or device.

[0578] An "information terminal" is a technological device used by a user to input operating commands, and includes, for example, smartphones and personal computers.

[0579] A "reception method" is a technique that has the function of receiving operation commands transmitted from an information terminal and storing them in a database necessary for processing.

[0580] A "generative artificial intelligence model" is an AI model trained using algorithms based on a vast amount of historical data, and is a means of creating the optimal operation plan under specific conditions.

[0581] An "operation plan" is a plan of specific work processes and sequences that a machine should perform, generated by a generative artificial intelligence model.

[0582] "Plan generation means" refers to a function or system for formulating an operation plan using an artificial intelligence model based on operation commands.

[0583] "Heavy machinery" refers to large mechanical devices used to perform work in designated locations, and includes excavators, snowplows, and other similar equipment.

[0584] "Wireless communication technology" refers to methods for sending and receiving data between electronic devices without using cables, and examples include Wi-Fi and Bluetooth.

[0585] "Operation means" refers to a method of remotely controlling heavy machinery by issuing instructions based on a generated operation plan.

[0586] "Feedback" is an information transmission process that reports information about the progress of a machine's operation to a server in real time.

[0587] "Monitoring" is the process of continuously observing and recording the surrounding conditions and work progress of machinery and equipment using sensors.

[0588] "Operation adjustment means" refers to a function for replanning operations in order to remove factors that hinder the progress of work when unforeseen circumstances occur.

[0589] This invention provides a method for efficiently and autonomously executing user work commands using a system comprising an information terminal, a computer server, and a remotely controllable mechanical device. The embodiments for carrying out this invention are described in detail below.

[0590] Users input work instructions using information terminals such as smartphones or personal computers. These work instructions include information such as the content of the specified work, geographical area, and desired completion time. A dedicated application runs on the terminal and securely transmits the information entered by the user to the server. This process uses the SSL / TLS protocol to ensure data security.

[0591] The server generates an optimal work plan based on the received work instructions, utilizing a generative artificial intelligence model. This AI model is trained using machine learning libraries such as TensorFlow and analyzes historical and current environmental data. This enables rapid and highly accurate plan generation.

[0592] The generated work plan is transmitted to the selected heavy machinery via wireless communication technology. The heavy machinery is equipped with LiDAR sensors and cameras, which are used to monitor the surrounding environment during operation. This ensures that the work is executed accurately, and the progress is constantly fed back to the server.

[0593] As a concrete example, if a user requests snow removal work at a parking lot in a certain area, the user uses a terminal to input the location of the parking lot and the desired work time, and then issues a command. Based on this information, the server calculates the optimal snow removal route and sends instructions to the heavy machinery. The heavy machinery starts working according to the instructions and reports to the server upon completion. Upon receiving this report, the server notifies the user that the work has been completed.

[0594] An example of an input prompt for the generating AI model would be: "Optimize snow removal operations for the specified parking lot. The location information is [specific location], and the desired time is [specific time]."

[0595] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0596] Step 1:

[0597] The user enters work instructions using a terminal. Specifically, they launch a dedicated application and enter the work details, location information of the target area, and desired completion time into a form. Once this information is entered, the application encrypts the data and sends it to the server using a secure protocol (SSL / TLS). The input data generates output as a work instruction.

[0598] Step 2:

[0599] The server receives work instructions from the terminal and saves their contents to the database. The received data is first validated. This process checks for any errors in the data format or input content. Once validation is complete, the content is output as valid data for processing.

[0600] Step 3:

[0601] The server invokes a generating AI model based on the received work instructions to generate an optimal work plan. At this stage, the AI ​​model receives historical work data and current environmental data as input and performs data analysis and machine learning. This results in an optimized plan of necessary work steps and resources to be used.

[0602] Step 4:

[0603] The server selects the heavy machinery to be used based on the created work plan. It evaluates the operating status and location of the heavy machinery in real time and selects the optimal machine. Once the selection is complete, it sends work instructions to the heavy machinery using wireless communication technology (e.g., MQTT protocol). The output contains information on the selected heavy machinery and the work instructions.

[0604] Step 5:

[0605] The heavy machinery begins work according to instructions sent from the server. Using LiDAR sensors and cameras mounted on the machinery, it performs the instructed tasks while monitoring the surrounding environment in real time. This monitoring data is returned to the server as progress information during the work. The output includes work progress and environmental information.

[0606] Step 6:

[0607] Once the work is completed, the heavy machinery reports this information to the server. The server receives the report, records the work results in the database, and notifies the user based on the prompt message. The output consists of confirmation of work completion and a notification to the user.

[0608] (Application Example 1)

[0609] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0610] Conventional autonomous assembly systems suffered from insufficient optimization of work plans, making it difficult to improve the efficiency and flexibility of assembly operations. Furthermore, they lacked the means to quickly respond to obstacles and changing environments, thus creating a need for improved productivity and safety.

[0611] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0612] In this invention, the server includes a receiving means for receiving operation commands, a planning means for generating an operation plan using a generative model, and an optimization means for using a generative model to optimize assembly work. This enables efficient and safe assembly work by an autonomous device.

[0613] A "reception mechanism for receiving operation commands" is a function that receives operation instructions entered by the user and converts them into a format that can be processed within the system.

[0614] "Planning generation means for generating operation plans using generative models" refers to a function that automatically generates optimal work schedules and procedures using AI models.

[0615] "Optimization methods using generative models to optimize assembly work" refers to a function that analyzes work processes using AI models and identifies areas for improvement in order to achieve efficient assembly work.

[0616] "Operating means for performing remote control on a machine" refers to a function that remotely controls and operates a machine based on instructions from a server.

[0617] A "monitoring method that monitors the progress of work and provides feedback" is a function that constantly monitors the status of work and immediately provides necessary information back to the user or system.

[0618] "Adjustment mechanisms for adjusting operations in response to unforeseen circumstances" refer to functions that quickly correct the system's response to sudden failures or changes, thereby ensuring the continuation of normal operations.

[0619] An "autonomous device" is a device that can perform tasks automatically without external intervention.

[0620] The "real-time optimization function" is a feature that immediately derives and applies the optimal work procedure based on information obtained during the work process.

[0621] The system for realizing this invention consists of a server, a terminal, and an autonomous device. It begins with the user inputting operation commands using the terminal and sending them to the server. These operation commands include details such as the task content and deadline. The server receives this information and uses a generative AI model to create an optimal work plan. This plan is designed to maximize work efficiency, and the generative model analyzes historical data and real-time environmental information.

[0622] Based on the plan, the server sends specific instructions to the autonomous device. The autonomous device receives the instructions from the server and automatically begins the assembly work. The device is equipped with sensors that allow it to continuously monitor its surroundings. Data from the sensors is used to monitor the progress of the work in real time and is sent back to the server as feedback. If an unforeseen problem occurs during the work, the server uses a generative AI model to immediately generate new instructions and adjust the device's operation.

[0623] As a concrete example, consider the efficient assembly of bolts and nuts on an automobile parts assembly line. In this case, by sending a prompt message from a terminal saying, "Generate an efficient combination pattern for automobile engine assembly," the server can formulate a work plan and optimize the work in real time. With such a configuration, the server can manage the work efficiently and safely, improving the overall productivity of the system.

[0624] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0625] Step 1:

[0626] The user inputs operational commands using a terminal. The information entered includes the work details, start time, and destination, and the terminal transmits this data to the server.

[0627] Step 2:

[0628] The server analyzes the operation commands received from the user. Based on the input data, it automatically generates an optimal work plan using a generation AI model. It formulates an efficient work schedule by considering past work history data and environmental information. The generated work plan is obtained as output.

[0629] Step 3:

[0630] Based on the generated work plan, the server creates and sends specific operating instructions to the autonomous device. These instructions include details such as which parts should be assembled and in what order. Upon receiving these instructions, the autonomous device generates an instruction list as output.

[0631] Step 4:

[0632] The autonomous device begins assembly work according to the operation instructions received from the server. Sensors mounted on the device detect the surrounding environment and transmit progress information to the server in real time. The input is the operation instruction, and the output is the progress status.

[0633] Step 5:

[0634] If an unforeseen event occurs during operation, the server analyzes the progress information and quickly generates new operation instructions using a generation AI model. Inputs are progress information and data on unforeseen events, while output is the updated operation instructions. The generated instructions are immediately transmitted to the autonomous device and adjusted to ensure smooth operation.

[0635] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0636] Embodiments of the present invention will now be described. This system includes a function to recognize the user's emotions and dynamically adjust the operation plan based on those emotions. The main components are a server, a terminal, a mechanical device, and an emotion engine.

[0637] Users input work instructions via a terminal, and during this process, an emotion sensor reads the user's emotions from their facial expressions and voice. This emotion data is sent to the server along with the work instructions. By using emotion recognition technology, it is possible to create operation plans that reflect the user's stress levels and expectations.

[0638] The server uses an emotion engine to analyze the user's current emotional state and fine-tunes the operation plan derived from the generative model based on the results. For example, if the user is anxious, the server can provide detailed reports on the progress of the work and share real-time progress to provide reassurance. At this stage, the server selects heavy machinery suitable for the designated work area and prepares the specific instructions necessary for remote operation.

[0639] Next, the server sends work instructions to the heavy machinery, which then autonomously begins work based on the instructions from the server. During the work, various sensors on the heavy machinery monitor the surrounding environment, avoiding obstacles and proceeding with the work as planned. If an unforeseen situation occurs, a situation report including emotional data is fed back to the server, and the server uses a generative model to issue new work instructions. In this process, changes in the user's emotions are also taken into consideration, and feedback is adjusted as needed.

[0640] For example, when a user feels a sense of urgency during snow removal work in a store's parking lot during heavy snowfall, the system quickly creates a plan and visualizes the work status as needed to reassure the user. Thus, a key point of this invention is to achieve safe and efficient work execution through emotionally sensitive feedback and work adjustments.

[0641] The following describes the processing flow.

[0642] Step 1:

[0643] The user uses a terminal to input work instructions and sends the work area and desired completion time to the system. Additionally, an emotion sensor built into the terminal monitors the user's facial expressions and voice, collecting emotional data at that moment. This data is then sent to a server.

[0644] Step 2:

[0645] The server receives work instructions and emotional data from the user. The received emotional data is analyzed by an emotion engine to determine the user's emotional state. For example, if the user's voice contains anxiety or tension, this is detected and recorded as the emotional state.

[0646] Step 3:

[0647] The server uses a generative model to create a normal operation plan, but it adjusts the plan appropriately by incorporating the results of the emotion engine's analysis. Specifically, it takes measures such as increasing information sharing at important decision points to alleviate user anxiety as much as possible.

[0648] Step 4:

[0649] The server sends the adjusted operating instructions to the heavy machinery. These instructions include the route, operating speed, and avoidance procedures. The plan is optimized here, taking user sentiment into consideration.

[0650] Step 5:

[0651] The heavy machinery receives instructions from the server and autonomously begins work. During operation, the machinery monitors its surroundings with sensors and reports the situation to the server in real time. Adjustments necessary for the progress of the work are continuously made in conjunction with emotional data.

[0652] Step 6:

[0653] Once the work is complete, the heavy machinery reports completion to the server. The server confirms this and provides the user with a completion notification and sentiment-based feedback. This allows the user to confidently confirm that the work has been completed.

[0654] (Example 2)

[0655] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0656] Conventional remote control systems have problems such as failing to consider the user's psychological state and not adequately ensuring work efficiency and safety. Furthermore, the lack of real-time interaction between operation instructions and actual machine operation makes users prone to anxiety and stress. In addition, unforeseen circumstances often require simple manual intervention, placing a burden on the user.

[0657] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0658] In this invention, the server includes a receiving means for receiving operation commands, a plan generation means for generating operation plans using a generative model and dynamically adjusting them based on the user's emotions, and an emotion recognition means for analyzing the user's facial expressions and voice to collect emotion data. This enables real-time adjustment of operation plans that take the user's emotions into consideration, improving work efficiency and safety, and allowing for a rapid response to unforeseen circumstances.

[0659] An "operation command" refers to a specific work instruction that a user sends to the system via their terminal.

[0660] A "generative model" is a type of artificial intelligence model used by systems to create operation plans, enabling dynamic plan generation based on data.

[0661] A "plan generation means" is a component of a system that has the function of generating an operation plan and adjusting it based on the user's emotions.

[0662] "Emotion recognition means" refers to a component of a system that collects emotional data by analyzing the user's facial expressions and voice.

[0663] "Operating means" refers to the function of a system that performs instructed operations in order to remotely control a machine or device.

[0664] "Monitoring means" refers to a system function that monitors the progress of work and provides users with real-time feedback on the progress.

[0665] A "regulation mechanism" is a component of a system that appropriately adjusts its operation in the event of unforeseen circumstances and provides feedback based on changes in the user's emotions.

[0666] The system of this invention analyzes the user's emotions and dynamically adjusts the operation plan based on them. It mainly consists of a server, terminals, mechanical devices, and software that links them together.

[0667] The terminal provides an interface for receiving operation commands entered by the user. The terminal is equipped with an emotion sensor that analyzes the user's facial expressions and voice in real time, thereby collecting emotion data. The collected emotion data is securely encrypted and transmitted to the server.

[0668] The server uses a generative AI model to generate and dynamically adjust operation plans based on the user's emotional state. It utilizes emotion recognition techniques to analyze the user's stress levels, sense of security, and other states, and modifies the generated plan accordingly. For example, if the user is feeling anxious, the operation plan can be modified to provide more detailed progress updates.

[0669] The machinery receives operation commands transmitted from the server and begins operating autonomously. Heavy machinery, such as equipment, constantly monitors its surroundings using various sensors to perform tasks safely. In the event of an unforeseen situation, the situation, including emotional data, is quickly fed back to the server, and new work instructions are generated.

[0670] A concrete example would be a situation where a user is anxious about bad weather during garden maintenance work. In this case, the server would visually display the work progress to reassure the user. An example of a prompt to the generated AI model would be, "If the user is perceived as anxious, please tell me how to display the progress in detail," which would allow for appropriate feedback.

[0671] This system makes it possible to perform tasks safely and efficiently, tailored to the user's emotional state.

[0672] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0673] Step 1:

[0674] The user operates the terminal and inputs specific work instructions. The terminal analyzes the user's facial expressions and voice using an emotion sensor and collects emotion data. The input consists of work instructions and emotion data. This data is encrypted and sent to the server.

[0675] Step 2:

[0676] The server receives emotional data and work instructions sent from the terminal. The emotion engine is activated to analyze the received data. In this process, the input emotional data is analyzed to determine the user's psychological state. The output is the analysis results, such as the user's stress level and level of reassurance.

[0677] Step 3:

[0678] The server uses a generative AI model to generate an operation plan for the work instruction based on the received analysis results. The generated plan is dynamically adjusted based on the user's emotional state. For example, if the user is anxious, a plan with more detailed steps will be generated. The adjusted operation plan is obtained as output.

[0679] Step 4:

[0680] The server transmits a coordinated operation plan to the machine. Based on the received command, the machine begins the specified task. During this process, the machine uses sensors to monitor its surroundings. The operation plan from the server is received as input, and the status of the task execution is generated as output.

[0681] Step 5:

[0682] If the machine detects environmental changes or unforeseen circumstances during operation, it feeds this information back to the server along with emotional data. The server then inputs this information back into the AI ​​model to generate new operation commands. The output is an updated work command.

[0683] Step 6:

[0684] Based on the updated work orders, the server checks the progress and provides feedback on the user's terminal. The progress is visually displayed to the user, incorporating features to provide reassurance. Feedback information is then generated for the user as output. This entire process optimizes work efficiency based on emotions.

[0685] (Application Example 2)

[0686] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0687] Conventional information delivery systems have difficulty providing information tailored to the emotional state of users, making it impossible to deliver information that is optimal for each individual user, thus hindering the improvement of the customer experience. Furthermore, the lack of flexible feedback functions that respond to users' emotions made it difficult to effectively improve user satisfaction.

[0688] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0689] In this invention, the server includes a receiving means for receiving operation commands, an emotion recognition means for collecting user emotion information using emotion recognition technology and providing information based on that information, and an information optimization means for optimizing and displaying information based on the user's emotions. This makes it possible to provide optimal information and effective feedback in real time according to the user's emotional state.

[0690] A "reception mechanism for receiving operation commands" is an interface that receives instruction information from the user and allows the system to start working based on that information.

[0691] "Planning generation means that generates an operation plan using a generative model" refers to a process that has an algorithm that determines the optimal work process based on specified conditions.

[0692] "Operating means for performing remote control of mechanical devices" refers to a system for controlling equipment located at a remote location and performing tasks according to instructions.

[0693] A "monitoring system for monitoring work progress and providing feedback" is a system for monitoring the current work status in real time and reporting the work status as needed.

[0694] "Adjustment mechanisms for adjusting operations in response to unforeseen circumstances" refer to functions that detect unexpected changes or failures and take appropriate measures to ensure the system continues to operate.

[0695] "An emotion recognition means that collects human emotion information using emotion recognition technology and provides information based on it" refers to a function that analyzes the user's emotional state from facial expressions, voice, etc., and provides information accordingly.

[0696] "Information optimization means that optimizes and displays information based on human emotions" refers to an algorithm that selects the most appropriate information according to the user's emotions and displays it on the user's device.

[0697] A system implementing this invention can achieve a superior user experience by providing and optimizing information based on the user's emotions. To achieve this, a server, terminal, and emotion recognition engine function as key elements.

[0698] The server first receives operation commands sent from the terminal through a receiving mechanism. Terminals equipped with emotion recognition technology use a camera and microphone to collect data such as the user's facial expressions and voice. This data is immediately transmitted to the server via a network protocol (e.g., Bluetooth or Wi-Fi).

[0699] On the server, an emotion recognition engine analyzes emotional data and uses a generative AI model to generate an optimal operation plan tailored to the user's emotions. By employing information optimization techniques, information and service content are selected based on the user's characteristics, and feedback is provided quickly. This makes it possible to provide appropriate information in real time to ensure user satisfaction.

[0700] As a concrete example, if the emotion recognition engine detects that a customer visiting a physical store appears slightly tired, the server sends a discount coupon for a drink usable in the store to the customer's device. This allows the customer to refresh themselves immediately, improving their satisfaction. An example of a prompt message to implement such a response would be, "Please advise on what products should be offered when a customer appears tired."

[0701] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0702] Step 1:

[0703] The device uses a camera to capture the user's facial expressions and a microphone to record their voice. The input data consists of facial image data and audio data, which are acquired as raw data for emotion recognition. This data is transmitted to a server via Bluetooth or Wi-Fi.

[0704] Step 2:

[0705] The server inputs the received facial image data and audio data into the emotion recognition engine. The emotion recognition engine uses a facial recognition algorithm and an audio emotion analysis algorithm to evaluate the user's emotional state. During the data analysis process, feature extraction and pattern matching are performed, and the output is a category of the emotional state (e.g., joy, surprise, fatigue).

[0706] Step 3:

[0707] The server generates an optimal action plan using a generative AI model based on the obtained emotional state categories. In this process, the AI ​​derives an appropriate response based on the prompt text. The generative model performs computational processing to select product and service information that matches the user's emotions, and as a result outputs customized information.

[0708] Step 4:

[0709] The server sends the generated operation plan and selected information to the terminal. The output information includes product information and promotional information displayed on the user's terminal. This allows the user to receive personalized information in real time and consider specific actions (e.g., purchasing a product, using a service).

[0710] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0711] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0712] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[0713] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0714] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[0715] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[0716] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[0717] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[0718] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[0719] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[0720] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[0721] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[0722] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0723] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0724] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0725] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[0726] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0727] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0728] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0729] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0730] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted as being incorporated by reference.

[0731] The following is further disclosed regarding the embodiments described above.

[0732] (Claim 1)

[0733] A means for receiving operation commands,

[0734] A planning generation means that generates an operation plan using a generative model,

[0735] Operating means for performing remote control of a machine device,

[0736] A monitoring system that monitors the progress of the work and provides feedback,

[0737] An adjustment mechanism to adjust the operation in response to unforeseen circumstances,

[0738] A system that includes this.

[0739] (Claim 2)

[0740] The system according to claim 1, which receives operation instructions from a terminal device and manages the completion of work in a designated work area.

[0741] (Claim 3)

[0742] The system according to claim 1, which includes a real-time optimization function for the operation of a machine device to continue working while avoiding obstacles.

[0743] "Example 1"

[0744] (Claim 1)

[0745] A reception means using an information terminal that receives operation commands,

[0746] A plan generation means that generates an operation plan based on past performance data and environmental information using a generative artificial intelligence model,

[0747] An operating means for transmitting work instructions to selected heavy machinery using wireless communication technology,

[0748] A monitoring system that uses sensors mounted on machinery to monitor the surrounding environment and feeds back the progress of the work to a server,

[0749] An action adjustment means that generates new instructions using a generative artificial intelligence model in response to unforeseen circumstances,

[0750] A system that includes this.

[0751] (Claim 2)

[0752] The system according to claim 1, which receives operation instructions from an information terminal, manages the completion of work in a designated geographical area, and notifies the user.

[0753] (Claim 3)

[0754] The system according to claim 1, comprising a function for monitoring the progress in real time while a machine is in operation and for optimizing the operation to continue while avoiding obstacles.

[0755] "Application Example 1"

[0756] (Claim 1)

[0757] A means for receiving operation commands,

[0758] A planning generation means that generates an operation plan using a generative model,

[0759] An optimization method using a generative model to optimize assembly work,

[0760] Operating means for performing remote control of a machine device,

[0761] A monitoring system that monitors the progress of the work and provides feedback,

[0762] An adjustment mechanism to adjust the operation in response to unforeseen circumstances,

[0763] A system that includes this.

[0764] (Claim 2)

[0765] The system according to claim 1, which receives operation instructions from a terminal device and manages assembly work by an autonomous device.

[0766] (Claim 3)

[0767] The system according to claim 1, which includes a real-time optimization function for the operation of a machine device to continue assembly work while avoiding obstacles.

[0768] "Example 2 of combining an emotion engine"

[0769] (Claim 1)

[0770] A means for receiving operation commands,

[0771] A plan generation means that generates an operation plan using a generative model and dynamically adjusts it based on the user's emotions,

[0772] An emotion recognition method that collects emotion data by analyzing the user's facial expressions and voice,

[0773] Operating means for performing remote control of a machine device,

[0774] Monitoring means to monitor the progress of the work and provide feedback to the user in a transparent manner,

[0775] An adjustment mechanism that adjusts its operation in response to unforeseen circumstances and provides feedback that takes into account changes in the user's emotions,

[0776] A system that includes this.

[0777] (Claim 2)

[0778] The system according to claim 1, which receives operation instructions from a terminal device and manages the completion of work in a designated work area, taking into account the user's emotions.

[0779] (Claim 3)

[0780] The system according to claim 1, further comprising a function that allows the machine to continue working while avoiding obstacles and to optimize the work plan in real time according to the user's stress level.

[0781] "Application example 2 when combining with an emotional engine"

[0782] (Claim 1)

[0783] A means for receiving operation commands,

[0784] A planning generation means that generates an operation plan using a generative model,

[0785] Operating means for performing remote control of a machine device,

[0786] A monitoring system that monitors the progress of the work and provides feedback,

[0787] An adjustment mechanism to adjust the operation in response to unforeseen circumstances,

[0788] An emotion recognition means that collects human emotion information using emotion recognition technology and provides information based on it,

[0789] Information optimization means that optimizes and displays information based on human emotions,

[0790] A system that includes this.

[0791] (Claim 2)

[0792] The system according to claim 1, which receives operation instructions from a terminal device, manages the completion of work in a designated work area, and provides information based on collected human emotional information.

[0793] (Claim 3)

[0794] The system according to claim 1, which has a real-time optimization function for the operation of a machine device to continue working while avoiding obstacles, and also optimizes and provides information based on human emotions. [Explanation of Symbols]

[0795] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. A means for receiving operation commands, A planning generation means that generates an operation plan using a generative model, Operating means for performing remote control of a machine device, A monitoring system that monitors the progress of the work and provides feedback, An adjustment mechanism to adjust the operation in response to unforeseen circumstances, A system that includes this.

2. The system according to claim 1, which receives operation instructions from a terminal device and manages the completion of work in a designated work area.

3. The system according to claim 1, which includes a real-time optimization function for the operation of a machine device to continue working while avoiding obstacles.

Images