system

Abstract

We provide the system. [Solution] A computing device that operates a generative AI that learns heavy equipment operation patterns based on operation data acquired by a data collection device, A terminal device that receives instruction information from the aforementioned computing device and controls the heavy machinery device, A device that feeds back the operation information of heavy machinery acquired by the terminal device to the computing device, A worker support interface using a video display device that displays the aforementioned instruction information, A system that includes this.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a persona chatbot control method performed by at least one processor, the method including steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance as a response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In the construction industry, the shortage of heavy equipment operators has become serious, and there are problems that it takes a great deal of time and cost to train particularly skilled operators. In addition, heavy equipment operation by humans requires breaks and it is difficult to operate in a 24-hour system. Moreover, heavy equipment operation is often accompanied by risks, and ensuring the safety of workers is required. It is necessary to address such problems of labor shortage and safety and improve the efficiency of operations.

Means for Solving the Problems

[0005] The present invention solves the aforementioned problem by providing a system comprising a server means that operates a generative AI that learns heavy equipment operation patterns based on operation data acquired by a data collection device, and a terminal means that receives instruction information from the server means and controls the heavy equipment. Furthermore, the accuracy of operation is improved by feeding back the operation information of the heavy equipment acquired by the terminal means to the server means. In addition, by providing a management means that allows the user to monitor the operation status of the heavy equipment and switch to manual operation as needed, 24-hour operation is made possible while ensuring safety. With the present invention, it is possible to compensate for the shortage of personnel without having to train inexperienced people from scratch, and to realize efficient heavy equipment operation and a safe working environment.

[0006] A "data acquisition device" refers to devices such as sensors and cameras installed to acquire operating data from heavy machinery.

[0007] "Generative AI" is an artificial intelligence technology that learns the operating patterns of heavy machinery based on collected data and generates optimal operating instructions.

[0008] A "server system" is a computing system that operates a generation AI and processes collected data and generates instructions.

[0009] A "terminal device" is a communication device that receives instruction information from a server device and transmits that instruction to heavy machinery equipment.

[0010] "Heavy machinery" refers to mechanical equipment used for construction work, specifically including shovels and bulldozers.

[0011] A "user" is a person who monitors the operation of the system and intervenes manually as needed.

[0012] "Management means" refers to a system element that encompasses the process by which users monitor the operating status of heavy machinery in real time and switch to manual operation when an abnormality occurs. [Brief explanation of the drawing]

[0013] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This is a conceptual diagram showing an example of the essential functions of a data processing device and a smart device according to the first embodiment. [Figure 3] This is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] This is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] This is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] This is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] This is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] This is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] This shows an emotion map where multiple emotions are mapped. [Figure 10] This shows an emotion map where multiple emotions are mapped. [Figure 11] This is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] This is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] This is a sequence diagram showing the processing flow of the data processing system in Example 2, which incorporates an emotion engine. [Figure 14] This is a sequence diagram showing the processing flow of the data processing system in Application Example 2, which combines an emotion engine. [Modes for carrying out the invention]

[0014] An example of an embodiment of the system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0015] First, the terms used in the following description will be explained.

[0016] In the following embodiments, the numbered processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0017] In the following embodiments, the numbered RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0018] In the following embodiments, the numbered storage is one or more non-volatile storage devices that store various programs and various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes, etc.

[0019] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0020] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0021] [First Embodiment]

[0022] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0023] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0024] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0025] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0026] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0027] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0028] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0029] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0030] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0031] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0032] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0033] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0034] This invention is a heavy equipment operation system for supporting construction work, which achieves efficient and safe heavy equipment operation through the cooperation of a server, terminal, and user.

[0035] The server, as the central component of this invention, learns patterns of heavy equipment operation using generated AI based on heavy equipment operation data. This allows the server to understand the operation of various heavy machines and generate optimized operation instructions in real time. For example, the server determines the movement of the heavy equipment arm according to the work site conditions and continuously generates instructions in real time. "For instance, when performing debris removal work, efficient work can be achieved by optimally adjusting the arm angle and bucket position."

[0036] The terminal receives operation instructions from the server and accurately transmits those instructions to the heavy machinery. Because the terminal continuously monitors various sensor information from the heavy machinery in real time, it can improve the accuracy of operation through its collaboration with the server. For example, if the terminal detects a change in gap from the heavy machinery's accelerometer, it immediately sends this data to the server, assisting in the process of updating the operation instructions.

[0037] The user is responsible for monitoring the overall system operation and verifying that remote control is being performed correctly. The system is designed so that the user can switch to manual operation if an anomaly is detected. This ensures 24-hour operation of heavy machinery while maintaining safety. "For example, if a user detects abnormal operation of heavy machinery through the monitoring screen, they can immediately switch to manual operation to prevent accidents."

[0038] In this way, the system of the present invention can improve the efficiency and safety of heavy machinery operation and solve the problem of labor shortages. By operating the system, it is possible to significantly reduce the time and cost required to train skilled operators, and it is also expected to improve safety on site.

[0039] The following describes the processing flow.

[0040] Step 1:

[0041] The server receives operational data from data collection devices installed on heavy machinery at the site and stores it in a database. This includes information such as the location, speed, and operating status of the heavy machinery.

[0042] Step 2:

[0043] The server uses the collected operational data to train the generating AI, allowing it to learn the operating patterns of heavy machinery. This is so that the AI ​​model can generate optimized operating instructions suitable for the task.

[0044] Step 3:

[0045] The terminal receives instruction information from the server. Based on the received instructions, it controls the actuators of the heavy machinery to perform specific actions. The terminal coordinates with the server to control the movement of the heavy machinery's arm and the operation of its bucket.

[0046] Step 4:

[0047] The terminal sends real-time information obtained from various sensors during the operation of heavy machinery back to the server. This information includes data on the movement of the heavy machinery and environmental data of the work site.

[0048] Step 5:

[0049] The server analyzes the feedback received from the terminal and updates the operation instructions as needed. This allows for adjustments to be made to respond to dynamic situations.

[0050] Step 6:

[0051] The user monitors the overall system operation, checking the processes and movements of heavy machinery at each step. If an anomaly is detected, the user immediately switches to manual operation and takes appropriate action.

[0052] Step 7:

[0053] If the user does not detect any malfunctions or abnormalities, the system automatically continues with the next task or operation. This maintains the efficiency and continuity of heavy equipment operation.

[0054] (Example 1)

[0055] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0056] The present invention aims to achieve safe and efficient operation of large equipment such as heavy machinery, without relying on labor shortages or the skill level of the operator. Furthermore, it aims to enhance safety by enabling rapid responses based on real-time information from various sensors.

[0057] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0058] In this invention, the server includes an information processing device means that operates a generative AI that learns equipment operation patterns based on operation data acquired by a data recording device; a control terminal means that receives command information from the information processing device means and controls the equipment; and means that feed back equipment operation information acquired by the control terminal means to the information processing device means. This makes it possible to improve the accuracy of automated equipment operation while providing a safe and efficient work environment.

[0059] A "data recording device" is a device that collects operation logs and sensor information from equipment and stores the data necessary for analysis.

[0060] "Operation data" refers to information related to the operation of equipment, including operator instructions and data indicating the operating status of the equipment.

[0061] "Generative AI" is an artificial intelligence technology that learns patterns of device operation and generates optimized operating instructions.

[0062] "Information processing device" refers to a computing device used to process collected data and analyze necessary information.

[0063] "Command information" refers to operation instructions sent from an information processing device to a control terminal, and is necessary to control the operation of the equipment.

[0064] A "control terminal means" is a device that receives command information from an information processing device means and executes the operation of the equipment according to the instructions.

[0065] "Operational information" refers to data related to the operation of a device, including its operating status and information obtained from sensors.

[0066] A "feedback mechanism" is a function that sends operational information acquired by a control terminal back to an information processing device, which is then used for data analysis and modification of operations.

[0067] "Sensor information" refers to physical data detected by various sensors installed in equipment, and is used to improve the accuracy and safety of operations.

[0068] The present invention is a system for efficiently and safely managing the operation of heavy machinery, and is comprised of an information processing device, a control terminal, and a user monitoring device.

[0069] The server functions as an information processing device, first operating a generated AI model using operation data collected from a data recording device. This AI model learns optimal operation patterns from past operation history and generates optimized command information in real time. The hardware includes a database server and a high-performance processor, and the software uses machine learning frameworks. Specifically, software such as "TENSORFLOW®" and "PyTorch" are used.

[0070] The terminal, acting as a control terminal, receives command information from the server and is responsible for executing appropriate operations on the equipment. The terminal acquires operational and sensor information in real time from multiple sensors installed on the equipment and feeds this back to the server. The terminal communicates with the equipment's control device and transfers data using lightweight protocols such as "MQTT".

[0071] Users monitor the entire system using communication terminals as a means of management. Users can verify that operations are being performed correctly through information panels and surveillance cameras, and respond immediately by switching to manual operation if an abnormality occurs. This system ensures the safe operation of the equipment 24 hours a day.

[0072] As a concrete example of its operation, when performing debris removal work, the server passes prompt messages to the generating AI, such as "The current work site involves excavation in a narrow space. Extend the arm to its maximum extent to efficiently excavate the soil," and continues to generate operation instructions in real time. In this way, the system of the present invention realizes efficient and safe heavy equipment operation, enabling operation that does not depend on labor shortages or the skill of the operator.

[0073] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0074] Step 1:

[0075] The server receives operation data and sensor information as input from the data recording device. This data includes the position, speed, and arm angle of the heavy machinery. The server stores this data in a database and prepares it for use in subsequent processing. This data processing ensures that the necessary information is managed centrally.

[0076] Step 2:

[0077] The server inputs the accumulated operation data into a generating AI model. Based on the collected data, the AI ​​model analyzes patterns in heavy equipment operation and builds an algorithm to predict optimal actions. Through this data calculation, the AI ​​model learns operation trends from past data, enabling highly accurate predictions.

[0078] Step 3:

[0079] The server inputs prompt messages into the generating AI model. For example, a prompt such as "Generate work instructions appropriate to the current work site conditions" might be used. Based on this prompt, the AI ​​model generates real-time operation instructions, and the server receives the output. A specific example of this output might include instructions to set the arm angle to 45 degrees and advance the bucket 2 meters.

[0080] Step 4:

[0081] The terminal receives operation instructions sent from the server. The terminal transmits these instructions to the equipment's control system and executes the actual operation. The terminal verifies that the heavy machinery is operating as instructed and generates information indicating that the operation has been completed. This operation information is then used as input in subsequent verification steps.

[0082] Step 5:

[0083] The terminal feeds back operational information acquired in real time from sensors installed in the device to the server. For example, it uses data from an accelerometer to evaluate the stability of the device while it is in operation. This data feedback allows for fine-tuning of operations, improving safety.

[0084] Step 6:

[0085] The user monitors the overall operation based on operational reports and real-time video provided by the system. If an anomaly is detected, the user intervenes manually to ensure safe operation. Finally, the user sends a signal to the system indicating successful completion, and the task is finished.

[0086] (Application Example 1)

[0087] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0088] Operating heavy machinery requires skilled techniques, and labor shortages and ensuring safety are major challenges. Furthermore, the lack of appropriate information to support heavy machinery operation in real time hinders improvements in efficiency and safety at many sites. Especially in complex heavy machinery operations, workers need to receive optimal operating instructions to ensure safe operation. Therefore, new methods are needed to provide rapid and accurate support for heavy machinery operation.

[0089] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0090] In this invention, the server includes a computing device that operates a generative AI that learns heavy equipment operation patterns based on operation data acquired by a data collection device, a terminal device that receives instruction information from the computing device and controls the heavy equipment, a device that feeds back operation information of the heavy equipment acquired by the terminal device to the computing device, and a worker support interface using a video display device that displays the instruction information. This makes it possible for workers to perform their work safely and efficiently while visually confirming the operation instructions of the heavy equipment in real time.

[0091] A "data acquisition device" is a device used to acquire data related to the operation of heavy machinery.

[0092] A "computational unit" is a device that uses generated AI to learn heavy equipment operation patterns based on acquired operation data and generates instruction information.

[0093] "Generative AI" is a technology that uses artificial intelligence to learn the operating patterns of heavy machinery and generate optimized operating instructions.

[0094] A "terminal device" is a device that receives instruction information from a computing device and controls heavy machinery based on that information.

[0095] "Operation information" refers to information about the operation of heavy machinery acquired by terminal devices.

[0096] A "feedback device" is a device that has the function of sending back operational information acquired by a terminal device to the computing unit.

[0097] A "video display device" is a device that visually displays instruction information from a computing device and forms part of the operator support interface.

[0098] A "worker support interface" is a user interface used by workers to visually confirm operating instructions for heavy machinery while carrying out their tasks.

[0099] The system for carrying out this invention mainly consists of a computing device, a terminal device, a feedback device, and a video display device. By having these components work together, it is possible to improve the efficiency and safety of heavy machinery operation.

[0100] The server-side computing unit receives operation data transmitted from the data collection device and uses this data to learn heavy equipment operation patterns using generated AI. Based on the learned patterns, it generates appropriate operation instructions and transmits this instruction information to the terminal device. The computing unit performs data processing that supports safe heavy equipment operation by providing instructions optimized through processing of the AI ​​model in real time. Specifically, it creates instructions such as optimizing the bucket angle and arm movement through the analysis of operation data.

[0101] The terminal device receives instruction information transmitted from the server and controls the heavy machinery based on that information. Furthermore, it sends motion information acquired from the heavy machinery's sensors back to the computing unit using a feedback device. This operation creates an information feedback loop, improving the accuracy of the operation.

[0102] The video display device is integrated into a device such as smart glasses worn by the user, visually displaying instruction information to the operator of heavy machinery in real time. This allows the operator to perform safe and efficient operations based on direct visual information. This interface utilizes augmented reality (AR) technology, which overlays the image onto the real world, using platforms such as Unity and ARKit.

[0103] As a concrete example, in debris removal work at a construction site, the user receives instructions via smart glasses. These instructions include optimal work patterns learned by the generating AI. The user receives instructions such as "set the bucket to a specific angle," allowing them to work efficiently. An example of a prompt message to the generating AI model would be, "Generate appropriate instructions using on-site sensor information and heavy equipment operation patterns for safe and efficient heavy equipment operation."

[0104] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0105] Step 1:

[0106] The server acquires operation data from heavy machinery using data collection devices. The input includes raw data from various sensors (position, velocity, acceleration, etc.). This data is preprocessed and standardized, removing outliers, to prepare it for input into the generative AI model. This results in a clean dataset suitable for generative AI.

[0107] Step 2:

[0108] The server uses formatted data to train a generating AI model on heavy equipment operation patterns. The input in this step is the formatted operation data. The model analyzes this data and generates optimized operation instructions based on the learned patterns. The output provides specific instruction information tailored to the site conditions and generates prompt statements to instruct the operation of the heavy equipment.

[0109] Step 3:

[0110] The server sends the generated instruction information to the terminal. The input is the instruction information generated in the previous step, and the output is the data transmission to the terminal. The terminal receives this information and begins the specific action of issuing instructions to the heavy machinery's control system.

[0111] Step 4:

[0112] The terminal controls the operation of the heavy machinery according to instructions received from the server. In this step, the terminal receives instruction information as input and outputs specific operation commands to the power and control systems of the heavy machinery. The actuators of the heavy machinery follow these commands and perform the necessary actions.

[0113] Step 5:

[0114] The terminal collects operational information from sensors installed on heavy machinery via a feedback device. The input is the actual operational data of the heavy machinery, and the output is feedback data sent to the server. This operational information is analyzed as real-time data from the sensors to evaluate the accuracy and error of the operation.

[0115] Step 6:

[0116] The user visually confirms operating instructions for heavy machinery through a video display device. The input is the latest instruction information from the server. Based on these instructions, the user makes decisions for performing the work and prepares for manual operation as needed. The video display device uses AR technology to overlay instructions onto the operator's field of view, providing clear and easy-to-understand output.

[0117] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0118] This invention provides a system that further improves operational efficiency and safety by combining a remote control system for heavy machinery with an emotion engine that recognizes the user's emotions.

[0119] The server uses a generative AI to learn heavy equipment operation patterns and generates optimized operation instructions for the heavy equipment's movements. The server integrates information from terminals and users and continuously updates basic operation instructions in real time. For example, if the server receives abnormal vibration data during heavy equipment excavation work, it will quickly provide instructions to return the equipment to the normal range.

[0120] The terminal receives instruction information from the server and controls the heavy machinery based on it. The terminal also continuously feeds back the operating status of the heavy machinery to the server to ensure that optimal operation is being performed. In addition, the terminal receives instructions from the emotion engine and changes the layout of the control panel according to the user's stress level. For example, when the user is stressed, the key size is increased or guides are displayed to make operation easier.

[0121] The user can monitor the overall system operation and manually take over operation of the heavy machinery as needed. The emotion engine analyzes the user's tone of voice and facial expressions to understand their psychological state. This analysis is sent to the server, which adjusts the operation instructions and interface accordingly. For example, if the user shows high levels of stress, the server can temporarily stop the heavy machinery and issue an alarm to ensure the user's safety.

[0122] The system of this invention, through the coordinated operation of the above-mentioned components, creates a heavy equipment operating environment that reflects the user's psychological state, thereby supporting efficient and safe work. Furthermore, by reducing user stress while improving work efficiency, it is expected to be highly practical on construction sites.

[0123] The following describes the processing flow.

[0124] Step 1:

[0125] The server receives operational data from data collection devices installed on heavy machinery at the site, trains a generating AI based on that data, and learns heavy machinery operation patterns. Through this process, the server prepares optimal operational instructions for various work situations.

[0126] Step 2:

[0127] The terminal receives instruction information from the server and controls the operation of the heavy machinery. Specifically, it adjusts the operation of actuators based on the instructions received from the server, for example, to move the arm safely and effectively. To reflect changes in the site conditions, the terminal sends real-time feedback from sensors back to the server during operation.

[0128] Step 3:

[0129] The emotion engine analyzes the user's voice and facial expressions to assess their emotional state. If the user is showing high levels of stress, the engine sends this information to the server and provides instructions to the device to adjust the user interface as needed.

[0130] Step 4:

[0131] The server analyzes the user's emotional information received from the emotion engine and adjusts operating instructions and the operating status of heavy machinery based on that information. This reduces the user's psychological burden and maintains a safe and smooth working environment. For example, if stress levels are high, the server will respond quickly by temporarily pausing work or slowing down the pace.

[0132] Step 5:

[0133] Users can monitor the overall system operation and switch to manual control in the event of an unforeseen incident. They can also strive for further safety by reviewing interface adjustments suggested by the emotion engine and directly inputting instructions to the system as needed.

[0134] Step 6:

[0135] The system determines the next action based on information from sensors and the emotion engine, and adjusts the overall flow to ensure smooth operation. If user intervention is not required, the process proceeds automatically and moves on to the next task.

[0136] (Example 2)

[0137] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0138] In many heavy machinery operations, remote control is required to improve efficiency and safety. However, in reality, accidents and decreased efficiency can occur due to operational errors or psychological stress. Conventional systems lack sufficient dynamic interface adjustments and optimized operation instructions based on the user's psychological state, which poses a challenge.

[0139] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0140] In this invention, the server includes an information processing means that acquires operation data and operates a generative AI that learns machine operation patterns; a communication means that receives instruction information from the information processing means and controls the work equipment; a function that feeds back machine operation information acquired by the communication means to the information processing means; and a means that analyzes the user's psychological state and adjusts the operation interface based on the results. This makes it possible to reflect the user's psychological state in real time and enable efficient and safe machine operation.

[0141] "Operation data" refers to information that records the operating status of a machine and the user's actions.

[0142] "Generative AI" is an artificial intelligence technology that uses machine learning algorithms to learn operation patterns and generate optimal operation instructions based on new inputs.

[0143] "Information processing means" refers to hardware or software components that receive, analyze, and process data and generate results.

[0144] "Communication means" refers to an interface or protocol for sending and receiving data between different devices.

[0145] A "feedback function" is the process of sending acquired data or status information back to the original source or other system components.

[0146] "User's psychological state" refers to the state of the user's emotions, stress levels, and mental well-being.

[0147] An "operating interface" is a device or software element that allows a user to directly interact with a machine in order to operate it.

[0148] "Work equipment" refers to specialized machinery or devices used to perform a specific task.

[0149] This invention provides a remote control system for heavy machinery that improves operational efficiency and safety. Specific embodiments are described below.

[0150] The server first operates a generative AI model and learns machine operation patterns based on operational data. This includes the process of training the model using machine learning libraries such as TensorFlow. This generative AI model provides optimal operation instructions for heavy machinery in real time and dynamically. In particular, it has the ability to quickly generate corrective instructions when an anomaly is detected.

[0151] The terminal receives instructions from the server and controls the heavy machinery based on them. Hardware such as Raspberry Pi and Arduino are used for this purpose. The terminal feeds back operational information acquired from the heavy machinery's sensors to the server, and verifies that operations are performed precisely based on this information. Furthermore, it adaptively adjusts the operating interface, taking into account the user's mental state obtained from an emotion recognition engine. For example, for users experiencing high stress levels, it enlarges the size of the control keys and displays visual guidance.

[0152] Users can monitor the overall system operation and manually take over control as needed. The server assesses the user's psychological state by analyzing their tone of voice and facial expressions, and automates appropriate actions based on that data. For example, if a user shows extreme stress, the server will generate an alarm to safely stop the operation of heavy machinery.

[0153] A concrete example of a prompt message is: "Abnormal vibration data was received during heavy machinery excavation work. Please generate optimal operating instructions to return the vibration to the normal range." Such prompts allow the generating AI model to provide an appropriate control strategy.

[0154] This system embodiment optimizes the operation of heavy machinery, taking into account the user's psychological state, resulting in an efficient and safe working environment.

[0155] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0156] Step 1:

[0157] The server receives operational data transmitted from the data collection device. This data includes the operating status and operating patterns of the heavy machinery. The server analyzes this data using machine learning libraries such as TensorFlow to learn the heavy machinery operating patterns. In this process, data cleaning and feature extraction are performed to prepare the data in a format suitable for the machine learning model.

[0158] Step 2:

[0159] The server generates optimal operating instructions using a trained AI model. It uses real-time sensor information and operation data as input. Based on this, the server outputs operating instructions that allow for rapid response even when an anomaly is detected. These instructions include adjusting the operating speed of heavy machinery and modifying specific operation sequences.

[0160] Step 3:

[0161] The terminal receives operation instructions from the server and directly controls the heavy machinery. The input is instruction information from the server, and the output is the physical movement of the heavy machinery. The terminal uses a Raspberry Pi or Arduino to send instructions to actuators and sensors, controlling the actual operation. For example, it performs specific actions such as controlling the voltage of a servo motor to adjust the operating speed of the heavy machinery.

[0162] Step 4:

[0163] The terminal feeds back operational information from heavy machinery to the server in real time. The input is operational data acquired from sensors, and the output is feedback information sent to the server. This data is used to improve the accuracy of machine learning models and detect anomalies. This feedback loop allows the server to perform further optimizations.

[0164] Step 5:

[0165] The user monitors the operation of the heavy machinery and system feedback. The user can take over operation manually as needed. Inputs include operational information displayed on a display device, and outputs include user operation instructions. The user assesses the status and operational comfort of the heavy machinery and issues instructions to the system via the operation interface.

[0166] Step 6:

[0167] The server receives user psychological state data obtained from the emotion recognition engine and dynamically adjusts the interface. Inputs include user voice tone and facial expression information, while outputs include the adjusted interface layout and operation guide. Specifically, when the user is showing high levels of stress, the server takes measures such as enlarging buttons on the control panel. This process contributes to improving user work efficiency and safety.

[0168] (Application Example 2)

[0169] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0170] Conventional heavy equipment operation systems have suffered from insufficient provision of an operating environment that takes into account the user's psychological state, leading to problems such as increased operating errors and stress. In particular, in remote operation of heavy equipment, increased user stress poses a risk of reduced safety and work efficiency. To solve these problems, it is necessary to understand the user's emotions in real time and adapt the interface accordingly.

[0171] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0172] In this invention, the server includes an information processing device means that operates a generative AI that learns heavy machinery operation patterns based on operation data acquired by a data collection device; a computing device means that receives instruction information from the information processing device means and controls civil engineering machinery; a means that feeds back operation information of the civil engineering machinery acquired by the computing device means to the information processing device means; an analysis means that analyzes the user's psychological characteristics; and a means that adjusts the operation interface based on the analysis results of the analysis means. This makes it possible to monitor the user's stress level and optimize the operating environment based on it.

[0173] A "data acquisition device" is a device that measures information related to the operation and handling of heavy machinery and acquires it as digital data.

[0174] "Generative AI" is an artificial intelligence technology that learns the operation patterns of heavy machinery based on big data and generates optimal control instructions.

[0175] An "information processing device" refers to a computing device that receives, processes, records, and transmits data, and this patent primarily concerns the operation of generative AI.

[0176] A "computational device" refers to a computer or processor used to control heavy machinery based on specific instructions.

[0177] A "feedback mechanism" is a system or process that uses acquired operational information to optimize the entire system by sending it back to the server.

[0178] "Analysis methods" refer to systems and algorithms used to understand a user's psychological characteristics by analyzing their voice tone, facial expressions, and other factors.

[0179] "Means for adjusting the operating interface" refers to a process that has the function of dynamically changing the control panel and screen display according to the user's emotions and stress level.

[0180] In the system implementing this invention, the interaction between the server, terminal, and user is particularly important. The server operates a generative AI and analyzes heavy equipment operation data acquired through a data collection device. This generative AI learns the operation patterns of the heavy equipment based on past operation data and sensor data, and can generate optimal instructions. This instruction information is updated in real time and transmitted to the terminal.

[0181] The terminal controls construction machinery based on instruction information received from the server. It also feeds back the operation information of the heavy machinery to the server, promoting continuous system optimization. Furthermore, this system has analytical capabilities to analyze the user's psychological characteristics, adjusting the operating interface in real time based on emotional data obtained from the user's tone of voice, facial expressions, and other factors. For example, if the user is experiencing stress, the size and placement of buttons can be changed to make operation easier.

[0182] Users can operate the system while receiving feedback from these systems and switch to manual operation as needed. This system allows for high-level monitoring of user stress levels, resulting in improved safety and work efficiency.

[0183] For example, if an operator shows signs of fatigue at a construction site, the interface automatically adjusts, displaying guides and warnings. This helps prevent operational errors and maintains work safety.

[0184] An example of a prompt message for a generated AI model is: "When the user is under high stress, stop all operations of the heavy machinery and prompt the user to take a short break."

[0185] In this way, real-time data analysis and flexible system adjustments significantly improve the safety and efficiency of heavy equipment operation.

[0186] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0187] Step 1:

[0188] The server receives operation data and sensor data acquired from data collection devices. Using this data as input, it learns the operation patterns of heavy machinery using a generative AI model. The server uses a large dataset and applies machine learning algorithms to optimize operations. As output, it generates operation instructions and stores them in a database.

[0189] Step 2:

[0190] The server transmits the generated operation instructions to the terminal in real time. Using this instruction information as input, the terminal generates the signals necessary to control the construction machinery. The terminal then controls the actual machine operation and adjusts the position and movement of the heavy machinery.

[0191] Step 3:

[0192] The terminal continuously collects operational information from the heavy machinery and feeds it back to the server. This operational information includes the machinery's position, speed, and load status. Using this as input, the server generates additional data to further optimize the operation instructions and updates the processing in real time.

[0193] Step 4:

[0194] Sensor devices installed in the terminal capture the user's tone of voice and facial expressions to analyze their psychological characteristics. Using the acquired emotional data as input, the terminal uses a simple emotion analysis program to determine the user's stress level and emotional state. As output, it generates user emotional state data and sends it to the server.

[0195] Step 5:

[0196] The server sends instructions to the terminal to adjust the user interface based on the user's emotional state. Using this emotional data as input, it calculates instructions to adjust the layout of the user interface and the size of buttons. As a result, the terminal dynamically changes the screen layout to support the user's operation.

[0197] Step 6:

[0198] Users view the updated interface on the terminal's monitor screen and perform actions as needed. Based on sentiment analysis and interface instructions from the server, they can take appropriate actions to ensure smooth workflow. They also receive alarms and break instructions tailored to their stress levels, enabling them to take preventative measures to ensure safety.

[0199] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0200] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0201] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0202] [Second Embodiment]

[0203] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0204] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0205] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0206] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0207] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0208] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0209] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0210] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0211] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0212] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0213] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0214] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0215] This invention is a heavy equipment operation system for supporting construction work, which achieves efficient and safe heavy equipment operation through the cooperation of a server, terminal, and user.

[0216] The server, as the central component of this invention, learns patterns of heavy equipment operation using generated AI based on heavy equipment operation data. This allows the server to understand the operation of various heavy machines and generate optimized operation instructions in real time. For example, the server determines the movement of the heavy equipment arm according to the work site conditions and continuously generates instructions in real time. "For instance, when performing debris removal work, efficient work can be achieved by optimally adjusting the arm angle and bucket position."

[0217] The terminal receives operation instructions from the server and accurately transmits those instructions to the heavy machinery. Because the terminal continuously monitors various sensor information from the heavy machinery in real time, it can improve the accuracy of operation through its collaboration with the server. For example, if the terminal detects a change in gap from the heavy machinery's accelerometer, it immediately sends this data to the server, assisting in the process of updating the operation instructions.

[0218] The user is responsible for monitoring the overall system operation and verifying that remote control is being performed correctly. The system is designed so that the user can switch to manual operation if an anomaly is detected. This ensures 24-hour operation of heavy machinery while maintaining safety. "For example, if a user detects abnormal operation of heavy machinery through the monitoring screen, they can immediately switch to manual operation to prevent accidents."

[0219] In this way, the system of the present invention can improve the efficiency and safety of heavy machinery operation and solve the problem of labor shortages. By operating the system, it is possible to significantly reduce the time and cost required to train skilled operators, and it is also expected to improve safety on site.

[0220] The following describes the processing flow.

[0221] Step 1:

[0222] The server receives operational data from data collection devices installed on heavy machinery at the site and stores it in a database. This includes information such as the location, speed, and operating status of the heavy machinery.

[0223] Step 2:

[0224] The server uses the collected operational data to train the generating AI, allowing it to learn the operating patterns of heavy machinery. This is so that the AI ​​model can generate optimized operating instructions suitable for the task.

[0225] Step 3:

[0226] The terminal receives instruction information from the server. Based on the received instructions, it controls the actuators of the heavy machinery to perform specific actions. The terminal coordinates with the server to control the movement of the heavy machinery's arm and the operation of its bucket.

[0227] Step 4:

[0228] The terminal sends real-time information obtained from various sensors during the operation of heavy machinery back to the server. This information includes data on the movement of the heavy machinery and environmental data of the work site.

[0229] Step 5:

[0230] The server analyzes the feedback received from the terminal and updates the operation instructions as needed. This allows for adjustments to be made to respond to dynamic situations.

[0231] Step 6:

[0232] The user monitors the overall system operation, checking the processes and movements of heavy machinery at each step. If an anomaly is detected, the user immediately switches to manual operation and takes appropriate action.

[0233] Step 7:

[0234] If the user does not detect any malfunctions or abnormalities, the system automatically continues with the next task or operation. This maintains the efficiency and continuity of heavy equipment operation.

[0235] (Example 1)

[0236] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0237] The present invention aims to achieve safe and efficient operation of large equipment such as heavy machinery, without relying on labor shortages or the skill level of the operator. Furthermore, it aims to enhance safety by enabling rapid responses based on real-time information from various sensors.

[0238] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0239] In this invention, the server includes an information processing device means that operates a generative AI that learns equipment operation patterns based on operation data acquired by a data recording device; a control terminal means that receives command information from the information processing device means and controls the equipment; and means that feed back equipment operation information acquired by the control terminal means to the information processing device means. This makes it possible to improve the accuracy of automated equipment operation while providing a safe and efficient work environment.

[0240] A "data recording device" is a device that collects operation logs and sensor information from equipment and stores the data necessary for analysis.

[0241] "Operation data" refers to information related to the operation of equipment, including operator instructions and data indicating the operating status of the equipment.

[0242] "Generative AI" is an artificial intelligence technology that learns patterns of device operation and generates optimized operating instructions.

[0243] "Information processing device" refers to a computing device used to process collected data and analyze necessary information.

[0244] "Command information" refers to operation instructions sent from an information processing device to a control terminal, and is necessary to control the operation of the equipment.

[0245] A "control terminal means" is a device that receives command information from an information processing device means and executes the operation of the equipment according to the instructions.

[0246] "Operational information" refers to data related to the operation of a device, including its operating status and information obtained from sensors.

[0247] A "feedback mechanism" is a function that sends operational information acquired by a control terminal back to an information processing device, which is then used for data analysis and modification of operations.

[0248] "Sensor information" refers to physical data detected by various sensors installed in equipment, and is used to improve the accuracy and safety of operations.

[0249] The present invention is a system for efficiently and safely managing the operation of heavy machinery, and is comprised of an information processing device, a control terminal, and a user monitoring device.

[0250] The server functions as an information processing device, first operating a generating AI model using operation data collected from a data recording device. This AI model learns optimal operation patterns from past operation history and generates optimized command information in real time. The hardware includes a database server and a high-performance processor, and the software uses machine learning frameworks. Specifically, software such as "TensorFlow" and "PyTorch" are used.

[0251] The terminal, acting as a control terminal, receives command information from the server and is responsible for executing appropriate operations on the equipment. The terminal acquires operational and sensor information in real time from multiple sensors installed on the equipment and feeds this back to the server. The terminal communicates with the equipment's control device and transfers data using lightweight protocols such as "MQTT".

[0252] Users monitor the entire system using communication terminals as a means of management. Users can verify that operations are being performed correctly through information panels and surveillance cameras, and respond immediately by switching to manual operation if an abnormality occurs. This system ensures the safe operation of the equipment 24 hours a day.

[0253] As a concrete example of its operation, when performing debris removal work, the server passes prompt messages to the generating AI, such as "The current work site involves excavation in a narrow space. Extend the arm to its maximum extent to efficiently excavate the soil," and continues to generate operation instructions in real time. In this way, the system of the present invention realizes efficient and safe heavy equipment operation, enabling operation that does not depend on labor shortages or the skill of the operator.

[0254] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0255] Step 1:

[0256] The server receives operation data and sensor information as input from the data recording device. This data includes the position, speed, and arm angle of the heavy machinery. The server stores this data in a database and prepares it for use in subsequent processing. This data processing ensures that the necessary information is managed centrally.

[0257] Step 2:

[0258] The server inputs the accumulated operation data into a generating AI model. Based on the collected data, the AI ​​model analyzes patterns in heavy equipment operation and builds an algorithm to predict optimal actions. Through this data calculation, the AI ​​model learns operation trends from past data, enabling highly accurate predictions.

[0259] Step 3:

[0260] The server inputs prompt messages into the generating AI model. For example, a prompt such as "Generate work instructions appropriate to the current work site conditions" might be used. Based on this prompt, the AI ​​model generates real-time operation instructions, and the server receives the output. A specific example of this output might include instructions to set the arm angle to 45 degrees and advance the bucket 2 meters.

[0261] Step 4:

[0262] The terminal receives operation instructions sent from the server. The terminal transmits these instructions to the equipment's control system and executes the actual operation. The terminal verifies that the heavy machinery is operating as instructed and generates information indicating that the operation has been completed. This operation information is then used as input in subsequent verification steps.

[0263] Step 5:

[0264] The terminal feeds back operational information acquired in real time from sensors installed in the device to the server. For example, it uses data from an accelerometer to evaluate the stability of the device while it is in operation. This data feedback allows for fine-tuning of operations, improving safety.

[0265] Step 6:

[0266] The user monitors the overall operation based on operational reports and real-time video provided by the system. If an anomaly is detected, the user intervenes manually to ensure safe operation. Finally, the user sends a signal to the system indicating successful completion, and the task is finished.

[0267] (Application Example 1)

[0268] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0269] Operating heavy machinery requires skilled techniques, and labor shortages and ensuring safety are major challenges. Furthermore, the lack of appropriate information to support heavy machinery operation in real time hinders improvements in efficiency and safety at many sites. Especially in complex heavy machinery operations, workers need to receive optimal operating instructions to ensure safe operation. Therefore, new methods are needed to provide rapid and accurate support for heavy machinery operation.

[0270] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0271] In this invention, the server includes a computing device that operates a generative AI that learns heavy equipment operation patterns based on operation data acquired by a data collection device, a terminal device that receives instruction information from the computing device and controls the heavy equipment, a device that feeds back operation information of the heavy equipment acquired by the terminal device to the computing device, and a worker support interface using a video display device that displays the instruction information. This makes it possible for workers to perform their work safely and efficiently while visually confirming the operation instructions of the heavy equipment in real time.

[0272] A "data acquisition device" is a device used to acquire data related to the operation of heavy machinery.

[0273] A "computational unit" is a device that uses generated AI to learn heavy equipment operation patterns based on acquired operation data and generates instruction information.

[0274] "Generative AI" is a technology that uses artificial intelligence to learn the operating patterns of heavy machinery and generate optimized operating instructions.

[0275] A "terminal device" is a device that receives instruction information from a computing device and controls heavy machinery based on that information.

[0276] "Operation information" refers to information about the operation of heavy machinery acquired by terminal devices.

[0277] A "feedback device" is a device that has the function of sending back operational information acquired by a terminal device to the computing unit.

[0278] A "video display device" is a device that visually displays instruction information from a computing device and forms part of the operator support interface.

[0279] A "worker support interface" is a user interface used by workers to visually confirm operating instructions for heavy machinery while carrying out their tasks.

[0280] The system for carrying out this invention mainly consists of a computing device, a terminal device, a feedback device, and a video display device. By having these components work together, it is possible to improve the efficiency and safety of heavy machinery operation.

[0281] The computing device, which is a server, receives the operation data transmitted from the data collection device, and learns the heavy machinery operation pattern using the generated AI based on this data. Based on the learned pattern, it generates appropriate operation instructions and transmits this instruction information to the terminal device. The computing device performs real-time instructions optimized by the processing of the AI model and conducts data processing to support safe heavy machinery operation. Specifically, through the analysis of operation data, it creates instructions such as optimizing the angle of the bucket or the movement of the arm.

[0282] The terminal device receives the instruction information transmitted from the server and controls the heavy machinery device based on this information. Furthermore, it returns the operation information obtained from the sensors of the heavy machinery to the computing device using the feedback device. Through this operation, an information feedback loop is realized and the operation accuracy is improved.

[0283] The video display device is incorporated into a device such as smart glasses worn by the user, and visually displays the instruction information to the operator of the heavy machinery in real time. Thereby, the operator can perform safe and efficient operations based on direct visual information. This interface utilizes platforms such as Unity and ARKit and adopts AR technology that overlays and displays in the real world.

[0284] As a specific example, in the debris removal work at a construction site, the user receives instructions via smart glasses. These instructions include the optimal work pattern learned by the generated AI. The user can receive instructions such as "Set the bucket at a specific angle" and proceed with the work efficiently. Examples of prompt sentences for the generated AI model include content such as "Please generate appropriate instructions using the on-site sensor information and the heavy machinery operation pattern for safe and efficient heavy machinery operation instructions."

[0285] The flow of the specific process in Application Example 1 will be described using FIG. 12.

[0286] Step 1:

[0287] The server acquires operation data from heavy machinery using data collection devices. The input includes raw data from various sensors (position, velocity, acceleration, etc.). This data is preprocessed and standardized, removing outliers, to prepare it for input into the generative AI model. This results in a clean dataset suitable for generative AI.

[0288] Step 2:

[0289] The server uses formatted data to train a generating AI model on heavy equipment operation patterns. The input in this step is the formatted operation data. The model analyzes this data and generates optimized operation instructions based on the learned patterns. The output provides specific instruction information tailored to the site conditions and generates prompt statements to instruct the operation of the heavy equipment.

[0290] Step 3:

[0291] The server sends the generated instruction information to the terminal. The input is the instruction information generated in the previous step, and the output is the data transmission to the terminal. The terminal receives this information and begins the specific action of issuing instructions to the heavy machinery's control system.

[0292] Step 4:

[0293] The terminal controls the operation of the heavy machinery according to instructions received from the server. In this step, the terminal receives instruction information as input and outputs specific operation commands to the power and control systems of the heavy machinery. The actuators of the heavy machinery follow these commands and perform the necessary actions.

[0294] Step 5:

[0295] The terminal collects operational information from sensors installed on heavy machinery via a feedback device. The input is the actual operational data of the heavy machinery, and the output is feedback data sent to the server. This operational information is analyzed as real-time data from the sensors to evaluate the accuracy and error of the operation.

[0296] Step 6:

[0297] The user visually confirms operating instructions for heavy machinery through a video display device. The input is the latest instruction information from the server. Based on these instructions, the user makes decisions for performing the work and prepares for manual operation as needed. The video display device uses AR technology to overlay instructions onto the operator's field of view, providing clear and easy-to-understand output.

[0298] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0299] This invention provides a system that further improves operational efficiency and safety by combining a remote control system for heavy machinery with an emotion engine that recognizes the user's emotions.

[0300] The server uses a generative AI to learn heavy equipment operation patterns and generates optimized operation instructions for the heavy equipment's movements. The server integrates information from terminals and users and continuously updates basic operation instructions in real time. For example, if the server receives abnormal vibration data during heavy equipment excavation work, it will quickly provide instructions to return the equipment to the normal range.

[0301] The terminal receives instruction information from the server and controls the heavy machinery based on it. The terminal also continuously feedbacks the operating state of the heavy machinery to the server to confirm that the optimal operation is being executed. In addition, the terminal receives instructions from the emotion engine and changes the layout of the operation panel according to the user's stress level. For example, in a situation where the user is feeling stressed, the size of the keys is increased or a guide is displayed to make the operation easier.

[0302] The user can monitor the operation of the entire system and, if necessary, manually take over the operation of the heavy machinery. In this process, the emotion engine analyzes the user's voice tone and expression to grasp the psychological state. This analysis result is sent to the server, and the server adjusts the operation instructions and interface based on it. As a specific example, when the user shows high stress, the server can temporarily stop the operation of the heavy machinery and generate an alarm to ensure the safety of the user.

[0303] The system of the present invention constructs an operating environment for heavy machinery that reflects the psychological state of the user through the cooperation of each of the above components, and supports efficient and safe work. In addition, since it improves work efficiency while reducing the user's stress, it is expected to enhance the practicality at the construction site.

[0304] The process flow will be described below.

[0305] Step 1:

[0306] The server receives operation data from the data collection device installed on the heavy machinery at the site, trains the generated AI based on it, and learns the heavy machinery operation pattern. Through this process, the server prepares optimal operation instructions for various working situations.

[0307] Step 2:

[0308] The terminal receives instruction information from the server and controls the operation of the heavy machinery. Specifically, it adjusts the operation of actuators based on the instructions received from the server, for example, to move the arm safely and effectively. To reflect changes in the site conditions, the terminal sends real-time feedback from sensors back to the server during operation.

[0309] Step 3:

[0310] The emotion engine analyzes the user's voice and facial expressions to assess their emotional state. If the user is showing high levels of stress, the engine sends this information to the server and provides instructions to the device to adjust the user interface as needed.

[0311] Step 4:

[0312] The server analyzes the user's emotional information received from the emotion engine and adjusts operating instructions and the operating status of heavy machinery based on that information. This reduces the user's psychological burden and maintains a safe and smooth working environment. For example, if stress levels are high, the server will respond quickly by temporarily pausing work or slowing down the pace.

[0313] Step 5:

[0314] Users can monitor the overall system operation and switch to manual control in the event of an unforeseen incident. They can also strive for further safety by reviewing interface adjustments suggested by the emotion engine and directly inputting instructions to the system as needed.

[0315] Step 6:

[0316] The system determines the next action based on information from sensors and the emotion engine, and adjusts the overall flow to ensure smooth operation. If user intervention is not required, the process proceeds automatically and moves on to the next task.

[0317] (Example 2)

[0318] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0319] In many heavy machinery operations, remote control is required to improve efficiency and safety. However, in reality, accidents and decreased efficiency can occur due to operational errors or psychological stress. Conventional systems lack sufficient dynamic interface adjustments and optimized operation instructions based on the user's psychological state, which poses a challenge.

[0320] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0321] In this invention, the server includes an information processing means that acquires operation data and operates a generative AI that learns machine operation patterns; a communication means that receives instruction information from the information processing means and controls the work equipment; a function that feeds back machine operation information acquired by the communication means to the information processing means; and a means that analyzes the user's psychological state and adjusts the operation interface based on the results. This makes it possible to reflect the user's psychological state in real time and enable efficient and safe machine operation.

[0322] "Operation data" refers to information that records the operating status of a machine and the user's actions.

[0323] "Generative AI" is an artificial intelligence technology that uses machine learning algorithms to learn operation patterns and generate optimal operation instructions based on new inputs.

[0324] "Information processing means" refers to hardware or software components that receive, analyze, and process data and generate results.

[0325] "Communication means" refers to an interface or protocol for sending and receiving data between different devices.

[0326] A "feedback function" is the process of sending acquired data or status information back to the original source or other system components.

[0327] "User's psychological state" refers to the state of the user's emotions, stress levels, and mental well-being.

[0328] An "operating interface" is a device or software element that allows a user to directly interact with a machine in order to operate it.

[0329] "Work equipment" refers to specialized machinery or devices used to perform a specific task.

[0330] This invention provides a remote control system for heavy machinery that improves operational efficiency and safety. Specific embodiments are described below.

[0331] The server first operates a generative AI model and learns machine operation patterns based on operational data. This includes the process of training the model using machine learning libraries such as TensorFlow. This generative AI model provides optimal operation instructions for heavy machinery in real time and dynamically. In particular, it has the ability to quickly generate corrective instructions when an anomaly is detected.

[0332] The terminal receives instructions from the server and controls the heavy machinery based on them. Hardware such as Raspberry Pi and Arduino are used for this purpose. The terminal feeds back operational information acquired from the heavy machinery's sensors to the server, and verifies that operations are performed precisely based on this information. Furthermore, it adaptively adjusts the operating interface, taking into account the user's mental state obtained from an emotion recognition engine. For example, for users experiencing high stress levels, it enlarges the size of the control keys and displays visual guidance.

[0333] Users can monitor the overall system operation and manually take over control as needed. The server assesses the user's psychological state by analyzing their tone of voice and facial expressions, and automates appropriate actions based on that data. For example, if a user shows extreme stress, the server will generate an alarm to safely stop the operation of heavy machinery.

[0334] A concrete example of a prompt message is: "Abnormal vibration data was received during heavy machinery excavation work. Please generate optimal operating instructions to return the vibration to the normal range." Such prompts allow the generating AI model to provide an appropriate control strategy.

[0335] This system embodiment optimizes the operation of heavy machinery, taking into account the user's psychological state, resulting in an efficient and safe working environment.

[0336] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0337] Step 1:

[0338] The server receives operational data transmitted from the data collection device. This data includes the operating status and operating patterns of the heavy machinery. The server analyzes this data using machine learning libraries such as TensorFlow to learn the heavy machinery operating patterns. In this process, data cleaning and feature extraction are performed to prepare the data in a format suitable for the machine learning model.

[0339] Step 2:

[0340] The server generates optimal operating instructions using a trained AI model. It uses real-time sensor information and operation data as input. Based on this, the server outputs operating instructions that allow for rapid response even when an anomaly is detected. These instructions include adjusting the operating speed of heavy machinery and modifying specific operation sequences.

[0341] Step 3:

[0342] The terminal receives operation instructions from the server and directly controls the heavy machinery. The input is instruction information from the server, and the output is the physical movement of the heavy machinery. The terminal uses a Raspberry Pi or Arduino to send instructions to actuators and sensors, controlling the actual operation. For example, it performs specific actions such as controlling the voltage of a servo motor to adjust the operating speed of the heavy machinery.

[0343] Step 4:

[0344] The terminal feeds back operational information from heavy machinery to the server in real time. The input is operational data acquired from sensors, and the output is feedback information sent to the server. This data is used to improve the accuracy of machine learning models and detect anomalies. This feedback loop allows the server to perform further optimizations.

[0345] Step 5:

[0346] The user monitors the operation of the heavy machinery and system feedback. The user can take over operation manually as needed. Inputs include operational information displayed on a display device, and outputs include user operation instructions. The user assesses the status and operational comfort of the heavy machinery and issues instructions to the system via the operation interface.

[0347] Step 6:

[0348] The server receives user psychological state data obtained from the emotion recognition engine and dynamically adjusts the interface. Inputs include user voice tone and facial expression information, while outputs include the adjusted interface layout and operation guide. Specifically, when the user is showing high levels of stress, the server takes measures such as enlarging buttons on the control panel. This process contributes to improving user work efficiency and safety.

[0349] (Application Example 2)

[0350] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the smart glasses 214 as the "terminal".

[0351] Conventional heavy equipment operation systems have suffered from insufficient provision of an operating environment that takes into account the user's psychological state, leading to problems such as increased operating errors and stress. In particular, in remote operation of heavy equipment, increased user stress poses a risk of reduced safety and work efficiency. To solve these problems, it is necessary to understand the user's emotions in real time and adapt the interface accordingly.

[0352] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0353] In this invention, the server includes an information processing device means that operates a generative AI that learns heavy machinery operation patterns based on operation data acquired by a data collection device; a computing device means that receives instruction information from the information processing device means and controls civil engineering machinery; a means that feeds back operation information of the civil engineering machinery acquired by the computing device means to the information processing device means; an analysis means that analyzes the user's psychological characteristics; and a means that adjusts the operation interface based on the analysis results of the analysis means. This makes it possible to monitor the user's stress level and optimize the operating environment based on it.

[0354] A "data acquisition device" is a device that measures information related to the operation and handling of heavy machinery and acquires it as digital data.

[0355] "Generative AI" is an artificial intelligence technology that learns the operation patterns of heavy machinery based on big data and generates optimal control instructions.

[0356] An "information processing device" refers to a computing device that receives, processes, records, and transmits data, and this patent primarily concerns the operation of generative AI.

[0357] A "computational device" refers to a computer or processor used to control heavy machinery based on specific instructions.

[0358] A "feedback mechanism" is a system or process that uses acquired operational information to optimize the entire system by sending it back to the server.

[0359] "Analysis methods" refer to systems and algorithms used to understand a user's psychological characteristics by analyzing their voice tone, facial expressions, and other factors.

[0360] "Means for adjusting the operating interface" refers to a process that has the function of dynamically changing the control panel and screen display according to the user's emotions and stress level.

[0361] In the system implementing this invention, the interaction between the server, terminal, and user is particularly important. The server operates a generative AI and analyzes heavy equipment operation data acquired through a data collection device. This generative AI learns the operation patterns of the heavy equipment based on past operation data and sensor data, and can generate optimal instructions. This instruction information is updated in real time and transmitted to the terminal.

[0362] The terminal controls construction machinery based on instruction information received from the server. It also feeds back the operation information of the heavy machinery to the server, promoting continuous system optimization. Furthermore, this system has analytical capabilities to analyze the user's psychological characteristics, adjusting the operating interface in real time based on emotional data obtained from the user's tone of voice, facial expressions, and other factors. For example, if the user is experiencing stress, the size and placement of buttons can be changed to make operation easier.

[0363] Users can operate the system while receiving feedback from these systems and switch to manual operation as needed. This system allows for high-level monitoring of user stress levels, resulting in improved safety and work efficiency.

[0364] For example, if an operator shows signs of fatigue at a construction site, the interface automatically adjusts, displaying guides and warnings. This helps prevent operational errors and maintains work safety.

[0365] An example of a prompt message for a generated AI model is: "When the user is under high stress, stop all operations of the heavy machinery and prompt the user to take a short break."

[0366] In this way, real-time data analysis and flexible system adjustments significantly improve the safety and efficiency of heavy equipment operation.

[0367] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0368] Step 1:

[0369] The server receives operation data and sensor data acquired from data collection devices. Using this data as input, it learns the operation patterns of heavy machinery using a generative AI model. The server uses a large dataset and applies machine learning algorithms to optimize operations. As output, it generates operation instructions and stores them in a database.

[0370] Step 2:

[0371] The server transmits the generated operation instructions to the terminal in real time. Using this instruction information as input, the terminal generates the signals necessary to control the construction machinery. The terminal then controls the actual machine operation and adjusts the position and movement of the heavy machinery.

[0372] Step 3:

[0373] The terminal continuously collects operational information from the heavy machinery and feeds it back to the server. This operational information includes the machinery's position, speed, and load status. Using this as input, the server generates additional data to further optimize the operation instructions and updates the processing in real time.

[0374] Step 4:

[0375] Sensor devices installed in the terminal capture the user's tone of voice and facial expressions to analyze their psychological characteristics. Using the acquired emotional data as input, the terminal uses a simple emotion analysis program to determine the user's stress level and emotional state. As output, it generates user emotional state data and sends it to the server.

[0376] Step 5:

[0377] The server sends instructions to the terminal to adjust the user interface based on the user's emotional state. Using this emotional data as input, it calculates instructions to adjust the layout of the user interface and the size of buttons. As a result, the terminal dynamically changes the screen layout to support the user's operation.

[0378] Step 6:

[0379] Users view the updated interface on the terminal's monitor screen and perform actions as needed. Based on sentiment analysis and interface instructions from the server, they can take appropriate actions to ensure smooth workflow. They also receive alarms and break instructions tailored to their stress levels, enabling them to take preventative measures to ensure safety.

[0380] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0381] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0382] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0383] [Third Embodiment]

[0384] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0385] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0386] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0387] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0388] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0389] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0390] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0391] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0392] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0393] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0394] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0395] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0396] This invention is a heavy equipment operation system for supporting construction work, which achieves efficient and safe heavy equipment operation through the cooperation of a server, terminal, and user.

[0397] The server, as the central component of this invention, learns patterns of heavy equipment operation using generated AI based on heavy equipment operation data. This allows the server to understand the operation of various heavy machines and generate optimized operation instructions in real time. For example, the server determines the movement of the heavy equipment arm according to the work site conditions and continuously generates instructions in real time. "For instance, when performing debris removal work, efficient work can be achieved by optimally adjusting the arm angle and bucket position."

[0398] The terminal receives operation instructions from the server and accurately transmits those instructions to the heavy machinery. Because the terminal continuously monitors various sensor information from the heavy machinery in real time, it can improve the accuracy of operation through its collaboration with the server. For example, if the terminal detects a change in gap from the heavy machinery's accelerometer, it immediately sends this data to the server, assisting in the process of updating the operation instructions.

[0399] The user is responsible for monitoring the overall system operation and verifying that remote control is being performed correctly. The system is designed so that the user can switch to manual operation if an anomaly is detected. This ensures 24-hour operation of heavy machinery while maintaining safety. "For example, if a user detects abnormal operation of heavy machinery through the monitoring screen, they can immediately switch to manual operation to prevent accidents."

[0400] In this way, the system of the present invention can improve the efficiency and safety of heavy machinery operation and solve the problem of labor shortages. By operating the system, it is possible to significantly reduce the time and cost required to train skilled operators, and it is also expected to improve safety on site.

[0401] The following describes the processing flow.

[0402] Step 1:

[0403] The server receives operational data from data collection devices installed on heavy machinery at the site and stores it in a database. This includes information such as the location, speed, and operating status of the heavy machinery.

[0404] Step 2:

[0405] The server uses the collected operational data to train the generating AI, allowing it to learn the operating patterns of heavy machinery. This is so that the AI ​​model can generate optimized operating instructions suitable for the task.

[0406] Step 3:

[0407] The terminal receives instruction information from the server. Based on the received instructions, it controls the actuators of the heavy machinery to perform specific actions. The terminal coordinates with the server to control the movement of the heavy machinery's arm and the operation of its bucket.

[0408] Step 4:

[0409] The terminal sends real-time information obtained from various sensors during the operation of heavy machinery back to the server. This information includes data on the movement of the heavy machinery and environmental data of the work site.

[0410] Step 5:

[0411] The server analyzes the feedback received from the terminal and updates the operation instructions as needed. This allows for adjustments to be made to respond to dynamic situations.

[0412] Step 6:

[0413] The user monitors the overall system operation, checking the processes and movements of heavy machinery at each step. If an anomaly is detected, the user immediately switches to manual operation and takes appropriate action.

[0414] Step 7:

[0415] If the user does not detect any malfunctions or abnormalities, the system automatically continues with the next task or operation. This maintains the efficiency and continuity of heavy equipment operation.

[0416] (Example 1)

[0417] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0418] The present invention aims to achieve safe and efficient operation of large equipment such as heavy machinery, without relying on labor shortages or the skill level of the operator. Furthermore, it aims to enhance safety by enabling rapid responses based on real-time information from various sensors.

[0419] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0420] In this invention, the server includes an information processing device means that operates a generative AI that learns equipment operation patterns based on operation data acquired by a data recording device; a control terminal means that receives command information from the information processing device means and controls the equipment; and means that feed back equipment operation information acquired by the control terminal means to the information processing device means. This makes it possible to improve the accuracy of automated equipment operation while providing a safe and efficient work environment.

[0421] A "data recording device" is a device that collects operation logs and sensor information from equipment and stores the data necessary for analysis.

[0422] "Operation data" refers to information related to the operation of equipment, including operator instructions and data indicating the operating status of the equipment.

[0423] "Generative AI" is an artificial intelligence technology that learns patterns of device operation and generates optimized operating instructions.

[0424] "Information processing device" refers to a computing device used to process collected data and analyze necessary information.

[0425] "Command information" refers to operation instructions sent from an information processing device to a control terminal, and is necessary to control the operation of the equipment.

[0426] A "control terminal means" is a device that receives command information from an information processing device means and executes the operation of the equipment according to the instructions.

[0427] "Operational information" refers to data related to the operation of a device, including its operating status and information obtained from sensors.

[0428] A "feedback mechanism" is a function that sends operational information acquired by a control terminal back to an information processing device, which is then used for data analysis and modification of operations.

[0429] "Sensor information" refers to physical data detected by various sensors installed in equipment, and is used to improve the accuracy and safety of operations.

[0430] The present invention is a system for efficiently and safely managing the operation of heavy machinery, and is comprised of an information processing device, a control terminal, and a user monitoring device.

[0431] The server functions as an information processing device, first operating a generating AI model using operation data collected from a data recording device. This AI model learns optimal operation patterns from past operation history and generates optimized command information in real time. The hardware includes a database server and a high-performance processor, and the software uses machine learning frameworks. Specifically, software such as "TensorFlow" and "PyTorch" are used.

[0432] The terminal, acting as a control terminal, receives command information from the server and is responsible for executing appropriate operations on the equipment. The terminal acquires operational and sensor information in real time from multiple sensors installed on the equipment and feeds this back to the server. The terminal communicates with the equipment's control device and transfers data using lightweight protocols such as "MQTT".

[0433] Users monitor the entire system using communication terminals as a means of management. Users can verify that operations are being performed correctly through information panels and surveillance cameras, and respond immediately by switching to manual operation if an abnormality occurs. This system ensures the safe operation of the equipment 24 hours a day.

[0434] As a concrete example of its operation, when performing debris removal work, the server passes prompt messages to the generating AI, such as "The current work site involves excavation in a narrow space. Extend the arm to its maximum extent to efficiently excavate the soil," and continues to generate operation instructions in real time. In this way, the system of the present invention realizes efficient and safe heavy equipment operation, enabling operation that does not depend on labor shortages or the skill of the operator.

[0435] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0436] Step 1:

[0437] The server receives operation data and sensor information as input from the data recording device. This data includes the position, speed, and arm angle of the heavy machinery. The server stores this data in a database and prepares it for use in subsequent processing. This data processing ensures that the necessary information is managed centrally.

[0438] Step 2:

[0439] The server inputs the accumulated operation data into a generating AI model. Based on the collected data, the AI ​​model analyzes patterns in heavy equipment operation and builds an algorithm to predict optimal actions. Through this data calculation, the AI ​​model learns operation trends from past data, enabling highly accurate predictions.

[0440] Step 3:

[0441] The server inputs prompt messages into the generating AI model. For example, a prompt such as "Generate work instructions appropriate to the current work site conditions" might be used. Based on this prompt, the AI ​​model generates real-time operation instructions, and the server receives the output. A specific example of this output might include instructions to set the arm angle to 45 degrees and advance the bucket 2 meters.

[0442] Step 4:

[0443] The terminal receives operation instructions sent from the server. The terminal transmits these instructions to the equipment's control system and executes the actual operation. The terminal verifies that the heavy machinery is operating as instructed and generates information indicating that the operation has been completed. This operation information is then used as input in subsequent verification steps.

[0444] Step 5:

[0445] The terminal feeds back operational information acquired in real time from sensors installed in the device to the server. For example, it uses data from an accelerometer to evaluate the stability of the device while it is in operation. This data feedback allows for fine-tuning of operations, improving safety.

[0446] Step 6:

[0447] The user monitors the overall operation based on operational reports and real-time video provided by the system. If an anomaly is detected, the user intervenes manually to ensure safe operation. Finally, the user sends a signal to the system indicating successful completion, and the task is finished.

[0448] (Application Example 1)

[0449] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0450] Operating heavy machinery requires skilled techniques, and labor shortages and ensuring safety are major challenges. Furthermore, the lack of appropriate information to support heavy machinery operation in real time hinders improvements in efficiency and safety at many sites. Especially in complex heavy machinery operations, workers need to receive optimal operating instructions to ensure safe operation. Therefore, new methods are needed to provide rapid and accurate support for heavy machinery operation.

[0451] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0452] In this invention, the server includes a computing device that operates a generative AI that learns heavy equipment operation patterns based on operation data acquired by a data collection device, a terminal device that receives instruction information from the computing device and controls the heavy equipment, a device that feeds back operation information of the heavy equipment acquired by the terminal device to the computing device, and a worker support interface using a video display device that displays the instruction information. This makes it possible for workers to perform their work safely and efficiently while visually confirming the operation instructions of the heavy equipment in real time.

[0453] A "data acquisition device" is a device used to acquire data related to the operation of heavy machinery.

[0454] A "computational unit" is a device that uses generated AI to learn heavy equipment operation patterns based on acquired operation data and generates instruction information.

[0455] "Generative AI" is a technology that uses artificial intelligence to learn the operating patterns of heavy machinery and generate optimized operating instructions.

[0456] A "terminal device" is a device that receives instruction information from a computing device and controls heavy machinery based on that information.

[0457] "Operation information" refers to information about the operation of heavy machinery acquired by terminal devices.

[0458] A "feedback device" is a device that has the function of sending back operational information acquired by a terminal device to the computing unit.

[0459] A "video display device" is a device that visually displays instruction information from a computing device and forms part of the operator support interface.

[0460] A "worker support interface" is a user interface used by workers to visually confirm operating instructions for heavy machinery while carrying out their tasks.

[0461] The system for carrying out this invention mainly consists of a computing device, a terminal device, a feedback device, and a video display device. By having these components work together, it is possible to improve the efficiency and safety of heavy machinery operation.

[0462] The server-side computing unit receives operation data transmitted from the data collection device and uses this data to learn heavy equipment operation patterns using generated AI. Based on the learned patterns, it generates appropriate operation instructions and transmits this instruction information to the terminal device. The computing unit performs data processing that supports safe heavy equipment operation by providing instructions optimized through processing of the AI ​​model in real time. Specifically, it creates instructions such as optimizing the bucket angle and arm movement through the analysis of operation data.

[0463] The terminal device receives instruction information transmitted from the server and controls the heavy machinery based on that information. Furthermore, it sends motion information acquired from the heavy machinery's sensors back to the computing unit using a feedback device. This operation creates an information feedback loop, improving the accuracy of the operation.

[0464] The video display device is integrated into a device such as smart glasses worn by the user, visually displaying instruction information to the operator of heavy machinery in real time. This allows the operator to perform safe and efficient operations based on direct visual information. This interface utilizes augmented reality (AR) technology, which overlays the image onto the real world, using platforms such as Unity and ARKit.

[0465] As a concrete example, in debris removal work at a construction site, the user receives instructions via smart glasses. These instructions include optimal work patterns learned by the generating AI. The user receives instructions such as "set the bucket to a specific angle," allowing them to work efficiently. An example of a prompt message to the generating AI model would be, "Generate appropriate instructions using on-site sensor information and heavy equipment operation patterns for safe and efficient heavy equipment operation."

[0466] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0467] Step 1:

[0468] The server acquires operation data from heavy machinery using data collection devices. The input includes raw data from various sensors (position, velocity, acceleration, etc.). This data is preprocessed and standardized, removing outliers, to prepare it for input into the generative AI model. This results in a clean dataset suitable for generative AI.

[0469] Step 2:

[0470] The server uses formatted data to train a generating AI model on heavy equipment operation patterns. The input in this step is the formatted operation data. The model analyzes this data and generates optimized operation instructions based on the learned patterns. The output provides specific instruction information tailored to the site conditions and generates prompt statements to instruct the operation of the heavy equipment.

[0471] Step 3:

[0472] The server sends the generated instruction information to the terminal. The input is the instruction information generated in the previous step, and the output is the data transmission to the terminal. The terminal receives this information and begins the specific action of issuing instructions to the heavy machinery's control system.

[0473] Step 4:

[0474] The terminal controls the operation of the heavy machinery according to instructions received from the server. In this step, the terminal receives instruction information as input and outputs specific operation commands to the power and control systems of the heavy machinery. The actuators of the heavy machinery follow these commands and perform the necessary actions.

[0475] Step 5:

[0476] The terminal collects operational information from sensors installed on heavy machinery via a feedback device. The input is the actual operational data of the heavy machinery, and the output is feedback data sent to the server. This operational information is analyzed as real-time data from the sensors to evaluate the accuracy and error of the operation.

[0477] Step 6:

[0478] The user visually confirms operating instructions for heavy machinery through a video display device. The input is the latest instruction information from the server. Based on these instructions, the user makes decisions for performing the work and prepares for manual operation as needed. The video display device uses AR technology to overlay instructions onto the operator's field of view, providing clear and easy-to-understand output.

[0479] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0480] This invention provides a system that further improves operational efficiency and safety by combining a remote control system for heavy machinery with an emotion engine that recognizes the user's emotions.

[0481] The server uses a generative AI to learn heavy equipment operation patterns and generates optimized operation instructions for the heavy equipment's movements. The server integrates information from terminals and users and continuously updates basic operation instructions in real time. For example, if the server receives abnormal vibration data during heavy equipment excavation work, it will quickly provide instructions to return the equipment to the normal range.

[0482] The terminal receives instruction information from the server and controls the heavy machinery based on it. The terminal also continuously feeds back the operating status of the heavy machinery to the server to ensure that optimal operation is being performed. In addition, the terminal receives instructions from the emotion engine and changes the layout of the control panel according to the user's stress level. For example, when the user is stressed, the key size is increased or guides are displayed to make operation easier.

[0483] The user can monitor the overall system operation and manually take over operation of the heavy machinery as needed. The emotion engine analyzes the user's tone of voice and facial expressions to understand their psychological state. This analysis is sent to the server, which adjusts the operation instructions and interface accordingly. For example, if the user shows high levels of stress, the server can temporarily stop the heavy machinery and issue an alarm to ensure the user's safety.

[0484] The system of this invention, through the coordinated operation of the above-mentioned components, creates a heavy equipment operating environment that reflects the user's psychological state, thereby supporting efficient and safe work. Furthermore, by reducing user stress while improving work efficiency, it is expected to be highly practical on construction sites.

[0485] The following describes the processing flow.

[0486] Step 1:

[0487] The server receives operational data from data collection devices installed on heavy machinery at the site, trains a generating AI based on that data, and learns heavy machinery operation patterns. Through this process, the server prepares optimal operational instructions for various work situations.

[0488] Step 2:

[0489] The terminal receives instruction information from the server and controls the operation of the heavy machinery. Specifically, it adjusts the operation of actuators based on the instructions received from the server, for example, to move the arm safely and effectively. To reflect changes in the site conditions, the terminal sends real-time feedback from sensors back to the server during operation.

[0490] Step 3:

[0491] The emotion engine analyzes the user's voice and facial expressions to assess their emotional state. If the user is showing high levels of stress, the engine sends this information to the server and provides instructions to the device to adjust the user interface as needed.

[0492] Step 4:

[0493] The server analyzes the user's emotional information received from the emotion engine and adjusts operating instructions and the operating status of heavy machinery based on that information. This reduces the user's psychological burden and maintains a safe and smooth working environment. For example, if stress levels are high, the server will respond quickly by temporarily pausing work or slowing down the pace.

[0494] Step 5:

[0495] Users can monitor the overall system operation and switch to manual control in the event of an unforeseen incident. They can also strive for further safety by reviewing interface adjustments suggested by the emotion engine and directly inputting instructions to the system as needed.

[0496] Step 6:

[0497] The system determines the next action based on information from sensors and the emotion engine, and adjusts the overall flow to ensure smooth operation. If user intervention is not required, the process proceeds automatically and moves on to the next task.

[0498] (Example 2)

[0499] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0500] In many heavy machinery operations, remote control is required to improve efficiency and safety. However, in reality, accidents and decreased efficiency can occur due to operational errors or psychological stress. Conventional systems lack sufficient dynamic interface adjustments and optimized operation instructions based on the user's psychological state, which poses a challenge.

[0501] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0502] In this invention, the server includes an information processing means that acquires operation data and operates a generative AI that learns machine operation patterns; a communication means that receives instruction information from the information processing means and controls the work equipment; a function that feeds back machine operation information acquired by the communication means to the information processing means; and a means that analyzes the user's psychological state and adjusts the operation interface based on the results. This makes it possible to reflect the user's psychological state in real time and enable efficient and safe machine operation.

[0503] "Operation data" refers to information that records the operating status of a machine and the user's actions.

[0504] "Generative AI" is an artificial intelligence technology that uses machine learning algorithms to learn operation patterns and generate optimal operation instructions based on new inputs.

[0505] "Information processing means" refers to hardware or software components that receive, analyze, and process data and generate results.

[0506] "Communication means" refers to an interface or protocol for sending and receiving data between different devices.

[0507] A "feedback function" is the process of sending acquired data or status information back to the original source or other system components.

[0508] "User's psychological state" refers to the state of the user's emotions, stress levels, and mental well-being.

[0509] An "operating interface" is a device or software element that allows a user to directly interact with a machine in order to operate it.

[0510] "Work equipment" refers to specialized machinery or devices used to perform a specific task.

[0511] This invention provides a remote control system for heavy machinery that improves operational efficiency and safety. Specific embodiments are described below.

[0512] The server first operates a generative AI model and learns machine operation patterns based on operational data. This includes the process of training the model using machine learning libraries such as TensorFlow. This generative AI model provides optimal operation instructions for heavy machinery in real time and dynamically. In particular, it has the ability to quickly generate corrective instructions when an anomaly is detected.

[0513] The terminal receives instructions from the server and controls the heavy machinery based on them. Hardware such as Raspberry Pi and Arduino are used for this purpose. The terminal feeds back operational information acquired from the heavy machinery's sensors to the server, and verifies that operations are performed precisely based on this information. Furthermore, it adaptively adjusts the operating interface, taking into account the user's mental state obtained from an emotion recognition engine. For example, for users experiencing high stress levels, it enlarges the size of the control keys and displays visual guidance.

[0514] Users can monitor the overall system operation and manually take over control as needed. The server assesses the user's psychological state by analyzing their tone of voice and facial expressions, and automates appropriate actions based on that data. For example, if a user shows extreme stress, the server will generate an alarm to safely stop the operation of heavy machinery.

[0515] A concrete example of a prompt message is: "Abnormal vibration data was received during heavy machinery excavation work. Please generate optimal operating instructions to return the vibration to the normal range." Such prompts allow the generating AI model to provide an appropriate control strategy.

[0516] This system embodiment optimizes the operation of heavy machinery, taking into account the user's psychological state, resulting in an efficient and safe working environment.

[0517] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0518] Step 1:

[0519] The server receives operational data transmitted from the data collection device. This data includes the operating status and operating patterns of the heavy machinery. The server analyzes this data using machine learning libraries such as TensorFlow to learn the heavy machinery operating patterns. In this process, data cleaning and feature extraction are performed to prepare the data in a format suitable for the machine learning model.

[0520] Step 2:

[0521] The server generates optimal operating instructions using a trained AI model. It uses real-time sensor information and operation data as input. Based on this, the server outputs operating instructions that allow for rapid response even when an anomaly is detected. These instructions include adjusting the operating speed of heavy machinery and modifying specific operation sequences.

[0522] Step 3:

[0523] The terminal receives operation instructions from the server and directly controls the heavy machinery. The input is instruction information from the server, and the output is the physical movement of the heavy machinery. The terminal uses a Raspberry Pi or Arduino to send instructions to actuators and sensors, controlling the actual operation. For example, it performs specific actions such as controlling the voltage of a servo motor to adjust the operating speed of the heavy machinery.

[0524] Step 4:

[0525] The terminal feeds back operational information from heavy machinery to the server in real time. The input is operational data acquired from sensors, and the output is feedback information sent to the server. This data is used to improve the accuracy of machine learning models and detect anomalies. This feedback loop allows the server to perform further optimizations.

[0526] Step 5:

[0527] The user monitors the operation of the heavy machinery and system feedback. The user can take over operation manually as needed. Inputs include operational information displayed on a display device, and outputs include user operation instructions. The user assesses the status and operational comfort of the heavy machinery and issues instructions to the system via the operation interface.

[0528] Step 6:

[0529] The server receives user psychological state data obtained from the emotion recognition engine and dynamically adjusts the interface. Inputs include user voice tone and facial expression information, while outputs include the adjusted interface layout and operation guide. Specifically, when the user is showing high levels of stress, the server takes measures such as enlarging buttons on the control panel. This process contributes to improving user work efficiency and safety.

[0530] (Application Example 2)

[0531] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0532] Conventional heavy equipment operation systems have suffered from insufficient provision of an operating environment that takes into account the user's psychological state, leading to problems such as increased operating errors and stress. In particular, in remote operation of heavy equipment, increased user stress poses a risk of reduced safety and work efficiency. To solve these problems, it is necessary to understand the user's emotions in real time and adapt the interface accordingly.

[0533] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0534] In this invention, the server includes an information processing device means that operates a generative AI that learns heavy machinery operation patterns based on operation data acquired by a data collection device; a computing device means that receives instruction information from the information processing device means and controls civil engineering machinery; a means that feeds back operation information of the civil engineering machinery acquired by the computing device means to the information processing device means; an analysis means that analyzes the user's psychological characteristics; and a means that adjusts the operation interface based on the analysis results of the analysis means. This makes it possible to monitor the user's stress level and optimize the operating environment based on it.

[0535] A "data acquisition device" is a device that measures information related to the operation and handling of heavy machinery and acquires it as digital data.

[0536] "Generative AI" is an artificial intelligence technology that learns the operation patterns of heavy machinery based on big data and generates optimal control instructions.

[0537] An "information processing device" refers to a computing device that receives, processes, records, and transmits data, and this patent primarily concerns the operation of generative AI.

[0538] A "computational device" refers to a computer or processor used to control heavy machinery based on specific instructions.

[0539] A "feedback mechanism" is a system or process that uses acquired operational information to optimize the entire system by sending it back to the server.

[0540] "Analysis methods" refer to systems and algorithms used to understand a user's psychological characteristics by analyzing their voice tone, facial expressions, and other factors.

[0541] "Means for adjusting the operating interface" refers to a process that has the function of dynamically changing the control panel and screen display according to the user's emotions and stress level.

[0542] In the system implementing this invention, the interaction between the server, terminal, and user is particularly important. The server operates a generative AI and analyzes heavy equipment operation data acquired through a data collection device. This generative AI learns the operation patterns of the heavy equipment based on past operation data and sensor data, and can generate optimal instructions. This instruction information is updated in real time and transmitted to the terminal.

[0543] The terminal controls construction machinery based on instruction information received from the server. It also feeds back the operation information of the heavy machinery to the server, promoting continuous system optimization. Furthermore, this system has analytical capabilities to analyze the user's psychological characteristics, adjusting the operating interface in real time based on emotional data obtained from the user's tone of voice, facial expressions, and other factors. For example, if the user is experiencing stress, the size and placement of buttons can be changed to make operation easier.

[0544] Users can operate the system while receiving feedback from these systems and switch to manual operation as needed. This system allows for high-level monitoring of user stress levels, resulting in improved safety and work efficiency.

[0545] For example, if an operator shows signs of fatigue at a construction site, the interface automatically adjusts, displaying guides and warnings. This helps prevent operational errors and maintains work safety.

[0546] An example of a prompt message for a generated AI model is: "When the user is under high stress, stop all operations of the heavy machinery and prompt the user to take a short break."

[0547] In this way, real-time data analysis and flexible system adjustments significantly improve the safety and efficiency of heavy equipment operation.

[0548] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0549] Step 1:

[0550] The server receives operation data and sensor data acquired from data collection devices. Using this data as input, it learns the operation patterns of heavy machinery using a generative AI model. The server uses a large dataset and applies machine learning algorithms to optimize operations. As output, it generates operation instructions and stores them in a database.

[0551] Step 2:

[0552] The server transmits the generated operation instructions to the terminal in real time. Using this instruction information as input, the terminal generates the signals necessary to control the construction machinery. The terminal then controls the actual machine operation and adjusts the position and movement of the heavy machinery.

[0553] Step 3:

[0554] The terminal continuously collects operational information from the heavy machinery and feeds it back to the server. This operational information includes the machinery's position, speed, and load status. Using this as input, the server generates additional data to further optimize the operation instructions and updates the processing in real time.

[0555] Step 4:

[0556] Sensor devices installed in the terminal capture the user's tone of voice and facial expressions to analyze their psychological characteristics. Using the acquired emotional data as input, the terminal uses a simple emotion analysis program to determine the user's stress level and emotional state. As output, it generates user emotional state data and sends it to the server.

[0557] Step 5:

[0558] The server sends instructions to the terminal to adjust the user interface based on the user's emotional state. Using this emotional data as input, it calculates instructions to adjust the layout of the user interface and the size of buttons. As a result, the terminal dynamically changes the screen layout to support the user's operation.

[0559] Step 6:

[0560] Users view the updated interface on the terminal's monitor screen and perform actions as needed. Based on sentiment analysis and interface instructions from the server, they can take appropriate actions to ensure smooth workflow. They also receive alarms and break instructions tailored to their stress levels, enabling them to take preventative measures to ensure safety.

[0561] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0562] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0563] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[0564] [Fourth Embodiment]

[0565] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0566] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0567] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0568] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[0569] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0570] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0571] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0572] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0573] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0574] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0575] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0576] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0577] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0578] This invention is a heavy equipment operation system for supporting construction work, which achieves efficient and safe heavy equipment operation through the cooperation of a server, terminal, and user.

[0579] The server, as the central component of this invention, learns patterns of heavy equipment operation using generated AI based on heavy equipment operation data. This allows the server to understand the operation of various heavy machines and generate optimized operation instructions in real time. For example, the server determines the movement of the heavy equipment arm according to the work site conditions and continuously generates instructions in real time. "For instance, when performing debris removal work, efficient work can be achieved by optimally adjusting the arm angle and bucket position."

[0580] The terminal receives operation instructions from the server and accurately transmits those instructions to the heavy machinery. Because the terminal continuously monitors various sensor information from the heavy machinery in real time, it can improve the accuracy of operation through its collaboration with the server. For example, if the terminal detects a change in gap from the heavy machinery's accelerometer, it immediately sends this data to the server, assisting in the process of updating the operation instructions.

[0581] The user is responsible for monitoring the overall system operation and verifying that remote control is being performed correctly. The system is designed so that the user can switch to manual operation if an anomaly is detected. This ensures 24-hour operation of heavy machinery while maintaining safety. "For example, if a user detects abnormal operation of heavy machinery through the monitoring screen, they can immediately switch to manual operation to prevent accidents."

[0582] In this way, the system of the present invention can improve the efficiency and safety of heavy machinery operation and solve the problem of labor shortages. By operating the system, it is possible to significantly reduce the time and cost required to train skilled operators, and it is also expected to improve safety on site.

[0583] The following describes the processing flow.

[0584] Step 1:

[0585] The server receives operational data from data collection devices installed on heavy machinery at the site and stores it in a database. This includes information such as the location, speed, and operating status of the heavy machinery.

[0586] Step 2:

[0587] The server uses the collected operational data to train the generating AI, allowing it to learn the operating patterns of heavy machinery. This is so that the AI ​​model can generate optimized operating instructions suitable for the task.

[0588] Step 3:

[0589] The terminal receives instruction information from the server. Based on the received instructions, it controls the actuators of the heavy machinery to perform specific actions. The terminal coordinates with the server to control the movement of the heavy machinery's arm and the operation of its bucket.

[0590] Step 4:

[0591] The terminal sends real-time information obtained from various sensors during the operation of heavy machinery back to the server. This information includes data on the movement of the heavy machinery and environmental data of the work site.

[0592] Step 5:

[0593] The server analyzes the feedback received from the terminal and updates the operation instructions as needed. This allows for adjustments to be made to respond to dynamic situations.

[0594] Step 6:

[0595] The user monitors the overall system operation, checking the processes and movements of heavy machinery at each step. If an anomaly is detected, the user immediately switches to manual operation and takes appropriate action.

[0596] Step 7:

[0597] If the user does not detect any malfunctions or abnormalities, the system automatically continues with the next task or operation. This maintains the efficiency and continuity of heavy equipment operation.

[0598] (Example 1)

[0599] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0600] The present invention aims to achieve safe and efficient operation of large equipment such as heavy machinery, without relying on labor shortages or the skill level of the operator. Furthermore, it aims to enhance safety by enabling rapid responses based on real-time information from various sensors.

[0601] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0602] In this invention, the server includes an information processing device means that operates a generative AI that learns equipment operation patterns based on operation data acquired by a data recording device; a control terminal means that receives command information from the information processing device means and controls the equipment; and means that feed back equipment operation information acquired by the control terminal means to the information processing device means. This makes it possible to improve the accuracy of automated equipment operation while providing a safe and efficient work environment.

[0603] A "data recording device" is a device that collects operation logs and sensor information from equipment and stores the data necessary for analysis.

[0604] "Operation data" refers to information related to the operation of equipment, including operator instructions and data indicating the operating status of the equipment.

[0605] "Generative AI" is an artificial intelligence technology that learns patterns of device operation and generates optimized operating instructions.

[0606] "Information processing device" refers to a computing device used to process collected data and analyze necessary information.

[0607] "Command information" refers to operation instructions sent from an information processing device to a control terminal, and is necessary to control the operation of the equipment.

[0608] A "control terminal means" is a device that receives command information from an information processing device means and executes the operation of the equipment according to the instructions.

[0609] "Operational information" refers to data related to the operation of a device, including its operating status and information obtained from sensors.

[0610] A "feedback mechanism" is a function that sends operational information acquired by a control terminal back to an information processing device, which is then used for data analysis and modification of operations.

[0611] "Sensor information" refers to physical data detected by various sensors installed in equipment, and is used to improve the accuracy and safety of operations.

[0612] The present invention is a system for efficiently and safely managing the operation of heavy machinery, and is comprised of an information processing device, a control terminal, and a user monitoring device.

[0613] The server functions as an information processing device, first operating a generating AI model using operation data collected from a data recording device. This AI model learns optimal operation patterns from past operation history and generates optimized command information in real time. The hardware includes a database server and a high-performance processor, and the software uses machine learning frameworks. Specifically, software such as "TensorFlow" and "PyTorch" are used.

[0614] The terminal, acting as a control terminal, receives command information from the server and is responsible for executing appropriate operations on the equipment. The terminal acquires operational and sensor information in real time from multiple sensors installed on the equipment and feeds this back to the server. The terminal communicates with the equipment's control device and transfers data using lightweight protocols such as "MQTT".

[0615] Users monitor the entire system using communication terminals as a means of management. Users can verify that operations are being performed correctly through information panels and surveillance cameras, and respond immediately by switching to manual operation if an abnormality occurs. This system ensures the safe operation of the equipment 24 hours a day.

[0616] As a concrete example of its operation, when performing debris removal work, the server passes prompt messages to the generating AI, such as "The current work site involves excavation in a narrow space. Extend the arm to its maximum extent to efficiently excavate the soil," and continues to generate operation instructions in real time. In this way, the system of the present invention realizes efficient and safe heavy equipment operation, enabling operation that does not depend on labor shortages or the skill of the operator.

[0617] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0618] Step 1:

[0619] The server receives operation data and sensor information as input from the data recording device. This data includes the position, speed, and arm angle of the heavy machinery. The server stores this data in a database and prepares it for use in subsequent processing. This data processing ensures that the necessary information is managed centrally.

[0620] Step 2:

[0621] The server inputs the accumulated operation data into a generating AI model. Based on the collected data, the AI ​​model analyzes patterns in heavy equipment operation and builds an algorithm to predict optimal actions. Through this data calculation, the AI ​​model learns operation trends from past data, enabling highly accurate predictions.

[0622] Step 3:

[0623] The server inputs prompt messages into the generating AI model. For example, a prompt such as "Generate work instructions appropriate to the current work site conditions" might be used. Based on this prompt, the AI ​​model generates real-time operation instructions, and the server receives the output. A specific example of this output might include instructions to set the arm angle to 45 degrees and advance the bucket 2 meters.

[0624] Step 4:

[0625] The terminal receives operation instructions sent from the server. The terminal transmits these instructions to the equipment's control system and executes the actual operation. The terminal verifies that the heavy machinery is operating as instructed and generates information indicating that the operation has been completed. This operation information is then used as input in subsequent verification steps.

[0626] Step 5:

[0627] The terminal feeds back operational information acquired in real time from sensors installed in the device to the server. For example, it uses data from an accelerometer to evaluate the stability of the device while it is in operation. This data feedback allows for fine-tuning of operations, improving safety.

[0628] Step 6:

[0629] The user monitors the overall operation based on operational reports and real-time video provided by the system. If an anomaly is detected, the user intervenes manually to ensure safe operation. Finally, the user sends a signal to the system indicating successful completion, and the task is finished.

[0630] (Application Example 1)

[0631] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0632] Operating heavy machinery requires skilled techniques, and labor shortages and ensuring safety are major challenges. Furthermore, the lack of appropriate information to support heavy machinery operation in real time hinders improvements in efficiency and safety at many sites. Especially in complex heavy machinery operations, workers need to receive optimal operating instructions to ensure safe operation. Therefore, new methods are needed to provide rapid and accurate support for heavy machinery operation.

[0633] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0634] In this invention, the server includes a computing device that operates a generative AI that learns heavy equipment operation patterns based on operation data acquired by a data collection device, a terminal device that receives instruction information from the computing device and controls the heavy equipment, a device that feeds back operation information of the heavy equipment acquired by the terminal device to the computing device, and a worker support interface using a video display device that displays the instruction information. This makes it possible for workers to perform their work safely and efficiently while visually confirming the operation instructions of the heavy equipment in real time.

[0635] A "data acquisition device" is a device used to acquire data related to the operation of heavy machinery.

[0636] A "computational unit" is a device that uses generated AI to learn heavy equipment operation patterns based on acquired operation data and generates instruction information.

[0637] "Generative AI" is a technology that uses artificial intelligence to learn the operating patterns of heavy machinery and generate optimized operating instructions.

[0638] A "terminal device" is a device that receives instruction information from a computing device and controls heavy machinery based on that information.

[0639] "Operation information" refers to information about the operation of heavy machinery acquired by terminal devices.

[0640] A "feedback device" is a device that has the function of sending back operational information acquired by a terminal device to the computing unit.

[0641] A "video display device" is a device that visually displays instruction information from a computing device and forms part of the operator support interface.

[0642] A "worker support interface" is a user interface used by workers to visually confirm operating instructions for heavy machinery while carrying out their tasks.

[0643] The system for carrying out this invention mainly consists of a computing device, a terminal device, a feedback device, and a video display device. By having these components work together, it is possible to improve the efficiency and safety of heavy machinery operation.

[0644] The server-side computing unit receives operation data transmitted from the data collection device and uses this data to learn heavy equipment operation patterns using generated AI. Based on the learned patterns, it generates appropriate operation instructions and transmits this instruction information to the terminal device. The computing unit performs data processing that supports safe heavy equipment operation by providing instructions optimized through processing of the AI ​​model in real time. Specifically, it creates instructions such as optimizing the bucket angle and arm movement through the analysis of operation data.

[0645] The terminal device receives instruction information transmitted from the server and controls the heavy machinery based on that information. Furthermore, it sends motion information acquired from the heavy machinery's sensors back to the computing unit using a feedback device. This operation creates an information feedback loop, improving the accuracy of the operation.

[0646] The video display device is integrated into a device such as smart glasses worn by the user, visually displaying instruction information to the operator of heavy machinery in real time. This allows the operator to perform safe and efficient operations based on direct visual information. This interface utilizes augmented reality (AR) technology, which overlays the image onto the real world, using platforms such as Unity and ARKit.

[0647] As a concrete example, in debris removal work at a construction site, the user receives instructions via smart glasses. These instructions include optimal work patterns learned by the generating AI. The user receives instructions such as "set the bucket to a specific angle," allowing them to work efficiently. An example of a prompt message to the generating AI model would be, "Generate appropriate instructions using on-site sensor information and heavy equipment operation patterns for safe and efficient heavy equipment operation."

[0648] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0649] Step 1:

[0650] The server acquires operation data from heavy machinery using data collection devices. The input includes raw data from various sensors (position, velocity, acceleration, etc.). This data is preprocessed and standardized, removing outliers, to prepare it for input into the generative AI model. This results in a clean dataset suitable for generative AI.

[0651] Step 2:

[0652] The server uses formatted data to train a generating AI model on heavy equipment operation patterns. The input in this step is the formatted operation data. The model analyzes this data and generates optimized operation instructions based on the learned patterns. The output provides specific instruction information tailored to the site conditions and generates prompt statements to instruct the operation of the heavy equipment.

[0653] Step 3:

[0654] The server sends the generated instruction information to the terminal. The input is the instruction information generated in the previous step, and the output is the data transmission to the terminal. The terminal receives this information and begins the specific action of issuing instructions to the heavy machinery's control system.

[0655] Step 4:

[0656] The terminal controls the operation of the heavy machinery according to instructions received from the server. In this step, the terminal receives instruction information as input and outputs specific operation commands to the power and control systems of the heavy machinery. The actuators of the heavy machinery follow these commands and perform the necessary actions.

[0657] Step 5:

[0658] The terminal collects operational information from sensors installed on heavy machinery via a feedback device. The input is the actual operational data of the heavy machinery, and the output is feedback data sent to the server. This operational information is analyzed as real-time data from the sensors to evaluate the accuracy and error of the operation.

[0659] Step 6:

[0660] The user visually confirms operating instructions for heavy machinery through a video display device. The input is the latest instruction information from the server. Based on these instructions, the user makes decisions for performing the work and prepares for manual operation as needed. The video display device uses AR technology to overlay instructions onto the operator's field of view, providing clear and easy-to-understand output.

[0661] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0662] This invention provides a system that further improves operational efficiency and safety by combining a remote control system for heavy machinery with an emotion engine that recognizes the user's emotions.

[0663] The server uses a generative AI to learn heavy equipment operation patterns and generates optimized operation instructions for the heavy equipment's movements. The server integrates information from terminals and users and continuously updates basic operation instructions in real time. For example, if the server receives abnormal vibration data during heavy equipment excavation work, it will quickly provide instructions to return the equipment to the normal range.

[0664] The terminal receives instruction information from the server and controls the heavy machinery based on it. The terminal also continuously feeds back the operating status of the heavy machinery to the server to ensure that optimal operation is being performed. In addition, the terminal receives instructions from the emotion engine and changes the layout of the control panel according to the user's stress level. For example, when the user is stressed, the key size is increased or guides are displayed to make operation easier.

[0665] The user can monitor the overall system operation and manually take over operation of the heavy machinery as needed. The emotion engine analyzes the user's tone of voice and facial expressions to understand their psychological state. This analysis is sent to the server, which adjusts the operation instructions and interface accordingly. For example, if the user shows high levels of stress, the server can temporarily stop the heavy machinery and issue an alarm to ensure the user's safety.

[0666] The system of this invention, through the coordinated operation of the above-mentioned components, creates a heavy equipment operating environment that reflects the user's psychological state, thereby supporting efficient and safe work. Furthermore, by reducing user stress while improving work efficiency, it is expected to be highly practical on construction sites.

[0667] The following describes the processing flow.

[0668] Step 1:

[0669] The server receives operational data from data collection devices installed on heavy machinery at the site, trains a generating AI based on that data, and learns heavy machinery operation patterns. Through this process, the server prepares optimal operational instructions for various work situations.

[0670] Step 2:

[0671] The terminal receives instruction information from the server and controls the operation of the heavy machinery. Specifically, it adjusts the operation of actuators based on the instructions received from the server, for example, to move the arm safely and effectively. To reflect changes in the site conditions, the terminal sends real-time feedback from sensors back to the server during operation.

[0672] Step 3:

[0673] The emotion engine analyzes the user's voice and facial expressions to assess their emotional state. If the user is showing high levels of stress, the engine sends this information to the server and provides instructions to the device to adjust the user interface as needed.

[0674] Step 4:

[0675] The server analyzes the user's emotional information received from the emotion engine and adjusts operating instructions and the operating status of heavy machinery based on that information. This reduces the user's psychological burden and maintains a safe and smooth working environment. For example, if stress levels are high, the server will respond quickly by temporarily pausing work or slowing down the pace.

[0676] Step 5:

[0677] Users can monitor the overall system operation and switch to manual control in the event of an unforeseen incident. They can also strive for further safety by reviewing interface adjustments suggested by the emotion engine and directly inputting instructions to the system as needed.

[0678] Step 6:

[0679] The system determines the next action based on information from sensors and the emotion engine, and adjusts the overall flow to ensure smooth operation. If user intervention is not required, the process proceeds automatically and moves on to the next task.

[0680] (Example 2)

[0681] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0682] In many heavy machinery operations, remote control is required to improve efficiency and safety. However, in reality, accidents and decreased efficiency can occur due to operational errors or psychological stress. Conventional systems lack sufficient dynamic interface adjustments and optimized operation instructions based on the user's psychological state, which poses a challenge.

[0683] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0684] In this invention, the server includes an information processing means that acquires operation data and operates a generative AI that learns machine operation patterns; a communication means that receives instruction information from the information processing means and controls the work equipment; a function that feeds back machine operation information acquired by the communication means to the information processing means; and a means that analyzes the user's psychological state and adjusts the operation interface based on the results. This makes it possible to reflect the user's psychological state in real time and enable efficient and safe machine operation.

[0685] "Operation data" refers to information that records the operating status of a machine and the user's actions.

[0686] "Generative AI" is an artificial intelligence technology that uses machine learning algorithms to learn operation patterns and generate optimal operation instructions based on new inputs.

[0687] "Information processing means" refers to hardware or software components that receive, analyze, and process data and generate results.

[0688] "Communication means" refers to an interface or protocol for sending and receiving data between different devices.

[0689] A "feedback function" is the process of sending acquired data or status information back to the original source or other system components.

[0690] "User's psychological state" refers to the state of the user's emotions, stress levels, and mental well-being.

[0691] An "operating interface" is a device or software element that allows a user to directly interact with a machine in order to operate it.

[0692] "Work equipment" refers to specialized machinery or devices used to perform a specific task.

[0693] This invention provides a remote control system for heavy machinery that improves operational efficiency and safety. Specific embodiments are described below.

[0694] The server first operates a generative AI model and learns machine operation patterns based on operational data. This includes the process of training the model using machine learning libraries such as TensorFlow. This generative AI model provides optimal operation instructions for heavy machinery in real time and dynamically. In particular, it has the ability to quickly generate corrective instructions when an anomaly is detected.

[0695] The terminal receives instructions from the server and controls the heavy machinery based on them. Hardware such as Raspberry Pi and Arduino are used for this purpose. The terminal feeds back operational information acquired from the heavy machinery's sensors to the server, and verifies that operations are performed precisely based on this information. Furthermore, it adaptively adjusts the operating interface, taking into account the user's mental state obtained from an emotion recognition engine. For example, for users experiencing high stress levels, it enlarges the size of the control keys and displays visual guidance.

[0696] Users can monitor the overall system operation and manually take over control as needed. The server assesses the user's psychological state by analyzing their tone of voice and facial expressions, and automates appropriate actions based on that data. For example, if a user shows extreme stress, the server will generate an alarm to safely stop the operation of heavy machinery.

[0697] A concrete example of a prompt message is: "Abnormal vibration data was received during heavy machinery excavation work. Please generate optimal operating instructions to return the vibration to the normal range." Such prompts allow the generating AI model to provide an appropriate control strategy.

[0698] This system embodiment optimizes the operation of heavy machinery, taking into account the user's psychological state, resulting in an efficient and safe working environment.

[0699] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0700] Step 1:

[0701] The server receives operational data transmitted from the data collection device. This data includes the operating status and operating patterns of the heavy machinery. The server analyzes this data using machine learning libraries such as TensorFlow to learn the heavy machinery operating patterns. In this process, data cleaning and feature extraction are performed to prepare the data in a format suitable for the machine learning model.

[0702] Step 2:

[0703] The server generates optimal operating instructions using a trained AI model. It uses real-time sensor information and operation data as input. Based on this, the server outputs operating instructions that allow for rapid response even when an anomaly is detected. These instructions include adjusting the operating speed of heavy machinery and modifying specific operation sequences.

[0704] Step 3:

[0705] The terminal receives operation instructions from the server and directly controls the heavy machinery. The input is instruction information from the server, and the output is the physical movement of the heavy machinery. The terminal uses a Raspberry Pi or Arduino to send instructions to actuators and sensors, controlling the actual operation. For example, it performs specific actions such as controlling the voltage of a servo motor to adjust the operating speed of the heavy machinery.

[0706] Step 4:

[0707] The terminal feeds back operational information from heavy machinery to the server in real time. The input is operational data acquired from sensors, and the output is feedback information sent to the server. This data is used to improve the accuracy of machine learning models and detect anomalies. This feedback loop allows the server to perform further optimizations.

[0708] Step 5:

[0709] The user monitors the operation of the heavy machinery and system feedback. The user can take over operation manually as needed. Inputs include operational information displayed on a display device, and outputs include user operation instructions. The user assesses the status and operational comfort of the heavy machinery and issues instructions to the system via the operation interface.

[0710] Step 6:

[0711] The server receives user psychological state data obtained from the emotion recognition engine and dynamically adjusts the interface. Inputs include user voice tone and facial expression information, while outputs include the adjusted interface layout and operation guide. Specifically, when the user is showing high levels of stress, the server takes measures such as enlarging buttons on the control panel. This process contributes to improving user work efficiency and safety.

[0712] (Application Example 2)

[0713] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0714] Conventional heavy equipment operation systems have suffered from insufficient provision of an operating environment that takes into account the user's psychological state, leading to problems such as increased operating errors and stress. In particular, in remote operation of heavy equipment, increased user stress poses a risk of reduced safety and work efficiency. To solve these problems, it is necessary to understand the user's emotions in real time and adapt the interface accordingly.

[0715] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0716] In this invention, the server includes an information processing device means that operates a generative AI that learns heavy machinery operation patterns based on operation data acquired by a data collection device; a computing device means that receives instruction information from the information processing device means and controls civil engineering machinery; a means that feeds back operation information of the civil engineering machinery acquired by the computing device means to the information processing device means; an analysis means that analyzes the user's psychological characteristics; and a means that adjusts the operation interface based on the analysis results of the analysis means. This makes it possible to monitor the user's stress level and optimize the operating environment based on it.

[0717] A "data acquisition device" is a device that measures information related to the operation and handling of heavy machinery and acquires it as digital data.

[0718] "Generative AI" is an artificial intelligence technology that learns the operation patterns of heavy machinery based on big data and generates optimal control instructions.

[0719] An "information processing device" refers to a computing device that receives, processes, records, and transmits data, and this patent primarily concerns the operation of generative AI.

[0720] A "computational device" refers to a computer or processor used to control heavy machinery based on specific instructions.

[0721] A "feedback mechanism" is a system or process that uses acquired operational information to optimize the entire system by sending it back to the server.

[0722] "Analysis methods" refer to systems and algorithms used to understand a user's psychological characteristics by analyzing their voice tone, facial expressions, and other factors.

[0723] "Means for adjusting the operating interface" refers to a process that has the function of dynamically changing the control panel and screen display according to the user's emotions and stress level.

[0724] In the system implementing this invention, the interaction between the server, terminal, and user is particularly important. The server operates a generative AI and analyzes heavy equipment operation data acquired through a data collection device. This generative AI learns the operation patterns of the heavy equipment based on past operation data and sensor data, and can generate optimal instructions. This instruction information is updated in real time and transmitted to the terminal.

[0725] The terminal controls construction machinery based on instruction information received from the server. It also feeds back the operation information of the heavy machinery to the server, promoting continuous system optimization. Furthermore, this system has analytical capabilities to analyze the user's psychological characteristics, adjusting the operating interface in real time based on emotional data obtained from the user's tone of voice, facial expressions, and other factors. For example, if the user is experiencing stress, the size and placement of buttons can be changed to make operation easier.

[0726] Users can operate the system while receiving feedback from these systems and switch to manual operation as needed. This system allows for high-level monitoring of user stress levels, resulting in improved safety and work efficiency.

[0727] For example, if an operator shows signs of fatigue at a construction site, the interface automatically adjusts, displaying guides and warnings. This helps prevent operational errors and maintains work safety.

[0728] An example of a prompt message for a generated AI model is: "When the user is under high stress, stop all operations of the heavy machinery and prompt the user to take a short break."

[0729] In this way, real-time data analysis and flexible system adjustments significantly improve the safety and efficiency of heavy equipment operation.

[0730] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0731] Step 1:

[0732] The server receives operation data and sensor data acquired from data collection devices. Using this data as input, it learns the operation patterns of heavy machinery using a generative AI model. The server uses a large dataset and applies machine learning algorithms to optimize operations. As output, it generates operation instructions and stores them in a database.

[0733] Step 2:

[0734] The server transmits the generated operation instructions to the terminal in real time. Using this instruction information as input, the terminal generates the signals necessary to control the construction machinery. The terminal then controls the actual machine operation and adjusts the position and movement of the heavy machinery.

[0735] Step 3:

[0736] The terminal continuously collects operational information from the heavy machinery and feeds it back to the server. This operational information includes the machinery's position, speed, and load status. Using this as input, the server generates additional data to further optimize the operation instructions and updates the processing in real time.

[0737] Step 4:

[0738] Sensor devices installed in the terminal capture the user's tone of voice and facial expressions to analyze their psychological characteristics. Using the acquired emotional data as input, the terminal uses a simple emotion analysis program to determine the user's stress level and emotional state. As output, it generates user emotional state data and sends it to the server.

[0739] Step 5:

[0740] The server sends instructions to the terminal to adjust the user interface based on the user's emotional state. Using this emotional data as input, it calculates instructions to adjust the layout of the user interface and the size of buttons. As a result, the terminal dynamically changes the screen layout to support the user's operation.

[0741] Step 6:

[0742] Users view the updated interface on the terminal's monitor screen and perform actions as needed. Based on sentiment analysis and interface instructions from the server, they can take appropriate actions to ensure smooth workflow. They also receive alarms and break instructions tailored to their stress levels, enabling them to take preventative measures to ensure safety.

[0743] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0744] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0745] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[0746] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0747] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[0748] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[0749] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[0750] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[0751] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[0752] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[0753] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[0754] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[0755] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0756] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0757] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0758] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[0759] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0760] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0761] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0762] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0763] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[0764] The following is further disclosed regarding the embodiments described above.

[0765] (Claim 1)

[0766] A server system that operates a generative AI that learns heavy equipment operation patterns based on operation data acquired by a data collection device,

[0767] A terminal means that receives instruction information from the server means and controls the heavy machinery device,

[0768] A means for feeding back the operation information of the heavy machinery acquired by the terminal means to the server means,

[0769] A system that includes this.

[0770] (Claim 2)

[0771] The system according to claim 1, further comprising a management means that allows the user to monitor the operating status of heavy machinery and switch to manual operation as necessary.

[0772] (Claim 3)

[0773] The system according to claim 1, wherein the generating AI has means for learning operation patterns for heavy machinery dismantling work or debris removal work and generating optimized operation instructions.

[0774] "Example 1"

[0775] (Claim 1)

[0776] An information processing device means that operates a generative AI that learns equipment operation patterns based on operation data acquired by a data recording device,

[0777] A control terminal means that receives command information from the aforementioned information processing device means and controls the equipment,

[0778] A means for feeding back the operation information of the device acquired by the control terminal means to the information processing means,

[0779] The control terminal means monitors sensor information of the device and transmits this data to the information processing device.

[0780] A system that includes this.

[0781] (Claim 2)

[0782] The system according to claim 1, further comprising a management means that allows a communication terminal to monitor the operating status of the equipment and switch to manual operation if an abnormality is detected.

[0783] (Claim 3)

[0784] The system according to claim 1, wherein the generating AI has means for learning operation patterns targeting specific tasks or object removal tasks of equipment and generating optimized operation instructions.

[0785] "Application Example 1"

[0786] (Claim 1)

[0787] A computing device that operates a generative AI that learns heavy equipment operation patterns based on operation data acquired by a data collection device,

[0788] A terminal device that receives instruction information from the aforementioned computing device and controls the heavy machinery device,

[0789] A device that feeds back the operation information of heavy machinery acquired by the terminal device to the computing device,

[0790] A worker support interface using a video display device that displays the aforementioned instruction information,

[0791] A system that includes this.

[0792] (Claim 2)

[0793] The system according to claim 1, further comprising a management system that allows an operator to monitor the operating status of heavy machinery and switch to manual operation as necessary.

[0794] (Claim 3)

[0795] The system according to claim 1, wherein the generating AI has a process for learning operation patterns for specific tasks of heavy machinery and generating optimized operation instructions, and further, the instruction information can be visually confirmed by the operator using a video display device.

[0796] "Example 2 of combining an emotion engine"

[0797] (Claim 1)

[0798] An information processing means that acquires operation data and operates a generative AI that learns machine operation patterns,

[0799] A communication means that receives instruction information from the aforementioned information processing means and controls the work equipment,

[0800] A function to feed back machine operation information acquired by the communication means to the information processing means,

[0801] A means for analyzing the user's psychological state and adjusting the operating interface based on the results,

[0802] ...

[0803] A system that includes this.

[0804] (Claim 2)

[0805] The system according to claim 1, further comprising monitoring means that allow the user to monitor the operating status of the machine and switch to manual operation as necessary.

[0806] (Claim 3)

[0807] The system according to claim 1, wherein the generating AI includes means for learning the operation patterns of work equipment and generating optimized operation instructions.

[0808] "Application example 2 when combining with an emotional engine"

[0809] (Claim 1)

[0810] An information processing device that operates a generative AI that learns heavy equipment operation patterns based on operation data acquired by a data collection device,

[0811] A computing device that receives instruction information from the aforementioned information processing device and controls the civil engineering machinery device,

[0812] A means for feeding back the operation information of the civil engineering machinery acquired by the computing device means to the information processing device means,

[0813] An analytical method for analyzing the psychological characteristics of users,

[0814] A means for adjusting the operation interface based on the analysis results of the aforementioned analysis means,

[0815] A system that includes this.

[0816] (Claim 2)

[0817] The system according to claim 1, further comprising control means that allows the user to monitor the operating status of the construction machinery and switch to manual operation as necessary.

[0818] (Claim 3)

[0819] The system according to claim 1, wherein the generating AI has means for learning operation patterns for dismantling or debris removal work of civil engineering machinery and generating optimized operation instructions. [Explanation of Symbols]

[0820] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. A computing device that operates a generative AI that learns heavy equipment operation patterns based on operation data acquired by a data collection device, A terminal device that receives instruction information from the aforementioned computing device and controls the heavy machinery device, A device that feeds back the operation information of heavy machinery acquired by the terminal device to the computing device, A worker support interface using a video display device that displays the aforementioned instruction information, A system that includes this.

2. The system according to claim 1, further comprising a management system that allows an operator to monitor the operating status of heavy machinery and switch to manual operation as needed.

3. The system according to claim 1, wherein the generating AI has a process for learning operation patterns for specific tasks of heavy machinery and generating optimized operation instructions, and further, the instruction information can be visually confirmed by the operator using a video display device.

Images