system

Abstract

We provide the system. [Solution] A server means for collecting and pre-processing operational data, A server means that builds a machine learning model based on preprocessed data and generates commands for operating heavy machinery, A terminal means for remotely operating heavy machinery based on commands received from a server, A user means to monitor the operation of heavy machinery and enable adjustments according to the situation, A system that includes this.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a method for controlling a persona chatbot performed by at least one processor, including steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of the chatbot's character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance as a response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In the construction industry, the shortage of highly specialized heavy equipment operators has become a serious problem. In particular, with the decline and aging of the labor force, it has become difficult to secure personnel responsible for heavy equipment operation. In addition, those engaged in heavy equipment work are accompanied by long working hours and dangerous working environments, which are exacerbating the personnel shortage. Therefore, it is an urgent task to realize remote operation and full automation to achieve labor saving and safety improvement in heavy equipment work.

Means for Solving the Problems

[0005] The present invention solves the aforementioned problem by providing a system that enables the remote operation of heavy machinery. This system includes server means for collecting and pre-processing operation data, and server means for building a machine learning model based on this data and generating commands for operating heavy machinery. Furthermore, it includes terminal means for remotely operating heavy machinery based on commands received from the server, and also has user means for monitoring the operation of heavy machinery and enabling adjustments according to the situation. This makes it possible to realize the safe and efficient operation of unmanned heavy machinery.

[0006] "Operation data" refers to information regarding the operating status and operation history of heavy machinery.

[0007] "Server means" refers to an information processing device that collects and processes data and generates commands for operating heavy machinery.

[0008] A "machine learning model" refers to a set of computational rules that are trained using operational data and used for the automated operation of heavy machinery.

[0009] "Commands" refer to control information sent from a server to a terminal to control the specific actions of heavy machinery.

[0010] "Terminal means" refers to a device that remotely operates heavy machinery based on commands from a server.

[0011] "User means" refers to management devices or people that monitor the entire system and adjust its operation as needed.

[0012] "Remote control" refers to the technology of operating equipment from a distance without directly touching it. [Brief explanation of the drawing]

[0013] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This is a conceptual diagram showing an example of the essential functions of a data processing device and a smart device according to the first embodiment. [Figure 3] It is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] It is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] It is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] It is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] It is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] It is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] It shows an emotion map to which multiple emotions are mapped. [Figure 10] It shows an emotion map to which multiple emotions are mapped. [Figure 11] It is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] It is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] It is a sequence diagram showing the processing flow of the data processing system in Example 2 when an emotion engine is combined. [Figure 14] It is a sequence diagram showing the processing flow of the data processing system in Application Example 2 when an emotion engine is combined.

Embodiments for Carrying Out the Invention

[0014] Hereinafter, an example of an embodiment of a system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0015] First, the language used in the following description will be explained.

[0016] In the following embodiments, the numbered processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0017] In the following embodiments, the numbered RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0018] In the following embodiments, the numbered storage is one or more non-volatile storage devices that store various programs and various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes, etc.

[0019] In the following embodiments, the numbered communication I / F (Interface) is an interface including a communication processor and an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark), etc.

[0020] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0021] [First Embodiment]

[0022] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0023] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0024] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0025] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0026] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0027] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0028] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0029] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0030] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0031] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0032] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0033] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0034] This invention relates to a remote control system for realizing unmanned operation of heavy machinery, and a specific embodiment is shown below. This system is assembled from three main components: a server, a terminal, and a user.

[0035] The server collects operational data from various sensors attached to the heavy machinery. This data includes the position, speed, orientation, and status of the arm and shovel. The server preprocesses this raw data and extracts features to be input into a machine learning model. This process builds a machine learning model that mimics the operation of the heavy machinery. Furthermore, the server uses this model to generate optimal operational commands tailored to the site conditions.

[0036] The terminal receives commands from the server and remotely controls the heavy machinery. Based on the commands, it transmits specific control signals to the heavy machinery, precisely controlling operations such as arm movement, rotation, and forward movement. The terminal also transmits the operating status of the heavy machinery and surrounding environmental data to the server in real time, providing feedback.

[0037] The user is responsible for monitoring the operation of heavy machinery from the control room. Based on monitoring data transmitted from the terminal, the user confirms that the operation of the heavy machinery is proceeding as planned. In the event of an unexpected situation, the user can also correct the situation by performing manual operations. After the operation, the user evaluates the results and provides feedback to the server to promote optimal automation of operations in various situations.

[0038] As a concrete example, consider debris removal work at a construction site. The server recognizes the shape and location of the debris piles using data from sensors and calculates the optimal removal method. The terminal controls heavy machinery based on the calculation results, performing operations to remove the debris safely and efficiently. The user monitors the entire operation and makes adjustments as needed.

[0039] In this way, this system can highly automate the unmanned operation of heavy machinery, thereby addressing the labor shortage problem and improving the working environment.

[0040] The following describes the processing flow.

[0041] Step 1:

[0042] The server collects operational data in real time from sensors mounted on the heavy machinery. This data includes location information, speed, and video and audio data showing the surrounding environment.

[0043] Step 2:

[0044] The server preprocesses the collected data, performing noise reduction and data normalization, and converting it into a format usable by machine learning models.

[0045] Step 3:

[0046] The server updates the machine learning model using the pre-processed data. This model learns an optimization algorithm for heavy equipment operation and generates the necessary operation commands.

[0047] Step 4:

[0048] The server sends the generated operation commands to the terminal. The commands include specific instructions for movement, such as changing the angle of the arm or the direction of movement of the vehicle body.

[0049] Step 5:

[0050] The terminal remotely controls heavy machinery based on commands received from the server. It generates appropriate control signals to execute the operations of the heavy machinery.

[0051] Step 6:

[0052] The terminal feeds back the operating status of the heavy machinery to the server. Based on the actual operating data, it provides the information necessary to generate the next command.

[0053] Step 7:

[0054] The user monitors the operating status in real time. They verify that the system is working stably and take manual control if an abnormality occurs.

[0055] Step 8:

[0056] After completing a task, the user evaluates the operation history and results of the heavy machinery. This allows them to provide feedback to the server for improving future operations.

[0057] (Example 1)

[0058] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0059] To safely and efficiently operate large machinery such as heavy equipment remotely and unmanned, real-time situational awareness and the generation of appropriate operating commands are required. Furthermore, a feedback mechanism capable of responding to unforeseen circumstances is necessary. However, conventional systems struggle to meet these requirements, resulting in challenges such as decreased labor productivity and safety risks.

[0060] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0061] In this invention, the server includes an information processing device means that collects operational information and performs preprocessing including noise reduction and normalization; an information processing device means that extracts characteristics based on the preprocessed information, constructs a learning model using a generation AI algorithm, and generates machine operation commands; and a control device means that remotely operates the machine based on the commands received from the information processing device and generates specific control signals. This enables efficient and safe operation of heavy machinery in real time from a remote location.

[0062] "Operational information" refers to data that indicates the state and behavior of the machine, including position information, speed, and arm movement.

[0063] "Information processing device means" refers to a device used to receive operation information and perform noise reduction, normalization, and characteristic extraction.

[0064] A "generative AI algorithm" refers to an algorithm that uses data to perform pattern recognition and prediction, and generates commands that are appropriate for new situations.

[0065] A "learning model" refers to a mathematical model that is built using past data and used to optimize future operations.

[0066] "Control device means" refers to a device for remotely operating heavy machinery and other machines based on received commands.

[0067] "Supervision device means" refers to a device that monitors the operating status of a machine and allows for manual operation in response to unforeseen circumstances.

[0068] A "feedback mechanism" refers to a system that compares actual actions with command content and updates the learning model based on the results.

[0069] This invention relates to an advanced automation system for realizing unmanned operation of heavy machinery. This system mainly consists of a server, terminals, and users, and each element works together to achieve efficient machine operation.

[0070] The server collects data from sensors attached to heavy machinery. This data includes the machinery's position, speed, and arm movement status. The server receives this data in real time and performs preprocessing such as noise reduction and data normalization. Furthermore, it uses machine learning frameworks such as TENSORFLOW® and PyTorch to extract characteristics from the data and build a learning model. This model uses generative AI algorithms to generate optimal operating commands for the heavy machinery.

[0071] The terminal receives operation commands transmitted from the server and generates specific control signals to control the heavy machinery based on those commands. This allows for precise remote operation of various aspects of the heavy machinery, such as arm movement, body rotation, and direction of travel adjustment. The terminal continuously feeds back the heavy machinery's operation data to the server, providing information to improve the accuracy of the operation.

[0072] In the control room, the user monitors the operation data of the heavy machinery provided from the terminal. The user can verify that the heavy machinery is operating as planned and can intervene manually if an anomaly is detected. This plays a crucial role in ensuring operational safety. Furthermore, after completing a task, the user reviews the results and provides feedback to the server, contributing to the optimization of future tasks.

[0073] As a concrete example, in the case of debris removal work at a construction site, the server analyzes the shape and location of the debris based on data from sensors and calculates the optimal removal method. The terminal operates heavy machinery according to these instructions to efficiently remove the debris. The user monitors this work and makes adjustments as needed. An example of a prompt message is, "Please tell me how to design an AI model that generates optimal heavy machinery operation instructions based on the shape and location of the pile of debris."

[0074] This invention aims to efficiently and safely achieve unmanned operation of heavy machinery through the coordination of these elements, thereby addressing labor shortages and improving working conditions.

[0075] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0076] Step 1:

[0077] The server collects operational information in real time from various sensors attached to heavy machinery. Inputs include location information from GPS sensors, velocity information from accelerometers, and arm movement status information from motion sensors. The server aggregates this input data, removes noise and outliers, and outputs it as normalized data. This process ensures data quality.

[0078] Step 2:

[0079] The server extracts features using preprocessed data. The input is the denoised data from step 1, from which characteristics related to the operation of heavy machinery are extracted. For example, this includes the angle of the arm and the open / closed state of the shovel at a specific time. The output is a dataset containing these features, which is used as input data for the generative AI model. This process also performs dimensionality reduction of the data as needed.

[0080] Step 3:

[0081] The server builds a machine learning model based on the extracted features. The feature data obtained in step 2 is given as input, and a machine learning framework such as TensorFlow is used to generate the trained model. The output is a trained model for predicting and optimizing the operation of heavy machinery. The model is used to generate operation commands according to site conditions.

[0082] Step 4:

[0083] The server generates optimal operating commands using the constructed learning model. The model generated in step 3 receives real-time field conditions as input data and calculates how the heavy machinery should operate. The output is a set of specific operating commands, which are sent to the terminal. These commands include instructions for arm movement and the direction of the machine's movement.

[0084] Step 5:

[0085] The terminal receives operation commands transmitted from the server and remotely controls the heavy machinery based on them. The input is the operation command generated in step 4, and the terminal generates specific control signals based on this. The output is the control signal to the heavy machinery, precisely controlling the operation of the arm and the movement of the machine. Furthermore, the terminal monitors the situation and feeds back the operation data to the server.

[0086] Step 6:

[0087] The user monitors the operating status of heavy machinery based on feedback data from the terminal. The input is real-time operation data provided by the terminal, which the user refers to to verify that the heavy machinery is operating as planned. If a problem occurs, the user intervenes manually and adjusts the operation. The output consists of user evaluations and correction instructions, ensuring the safety and efficiency of the system.

[0088] (Application Example 1)

[0089] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0090] Conventional remote control systems for heavy machinery lack methods for visualizing operating conditions, immediately detecting anomalies, and enabling rapid response by site supervisors. Therefore, there is a need to improve safety and work efficiency on site. Furthermore, remote control systems face the challenge of not being able to flexibly adapt to diverse environments.

[0091] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0092] In this invention, the server includes an information processing means for collecting and pre-processing operational information, an information processing means for constructing a mathematical model based on the pre-processed information and generating instructions for operating heavy machinery, and a communication device means for remotely operating heavy machinery based on instructions received from the information processing device. This makes it possible to visualize the operation of heavy machinery, respond immediately when an anomaly is detected, and allow site supervisors to quickly correct instructions. Furthermore, it enables safe and efficient work in a variety of site environments.

[0093] "Operational information" refers to all data related to the operation of heavy machinery, including information such as position, speed, orientation, and operating status obtained from sensors and other devices.

[0094] "Information processing means" refers to a device or software equipped with data processing functions for pre-processing collected operational information and using it to construct mathematical models or generate operation instructions.

[0095] A "mathematical model" refers to a model constructed using mathematical methods based on operational information in order to mimic and optimize the operation of heavy machinery.

[0096] "Communication device means" refers to a device or system that has a communication function for transmitting operation instructions received from an information processing device to heavy machinery and for remotely controlling the heavy machinery.

[0097] "Human-operated means" refers to means that provide an interface for human operators to monitor the operation of heavy machinery, enabling adjustments and modifications of instructions as needed.

[0098] "Mobile devices" refer to portable electronic devices such as smartphones and tablets that site supervisors use to visualize the operation status of heavy machinery and to correct manual instructions.

[0099] "Display means" refers to a function or device for visually displaying the operating status of heavy machinery or information from sensors on a mobile device.

[0100] "Alarming device" refers to a device or program that has an alert function to detect abnormalities or dangers in real time and immediately notify the user.

[0101] The system for implementing this invention combines information processing means, communication device means, human user means, mobile terminal and display means to realize unmanned remote operation of heavy machinery.

[0102] The server collects operational information from various sensors installed on heavy machinery and preprocesses this data. The preprocessed data is used to build a mathematical model, and after the model is built, optimal instructions for operating the heavy machinery are generated. This process utilizes generative AI models as data processing and computing technology. Furthermore, the server uses information management software to analyze operational information in real time and generate optimal instructions.

[0103] The terminal receives instructions generated by the server via a communication line and controls various operations of the heavy machinery. The communication device converts these instructions into specific control signals, enabling remote control of the heavy machinery.

[0104] Users can use their mobile devices to check operational status data transmitted from the server in real time. The devices are equipped with display devices that visually show the operating status of heavy machinery and environmental information. In addition, they allow for human intervention, such as fine-tuning of operations and manual intervention in case of abnormalities.

[0105] Anomaly detection is a mechanism in which the server quickly detects anomalies and notifies the user through an alarm system on a mobile device. For example, if a mechanical anomaly or safety risk occurs during operation, an alert is immediately issued, allowing the user to respond quickly.

[0106] As a concrete example, when removing rubble, the server acquires shape data in real time and suggests the most efficient work route to the user. By using this as a reference, the user can improve the safety and efficiency of the work.

[0107] An example of a prompt message for a generated AI model is, "Based on the current terrain data, please propose an efficient method for debris removal." This allows the server to provide sophisticated, data-driven instructions.

[0108] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0109] Step 1:

[0110] The server collects operational information such as position, speed, direction, and surrounding environment information in real time from various sensors installed on heavy machinery. The input is raw data from the sensors, and the output is pre-processed data. The data is shaped through noise filtering and normalization, and processed into a form suitable for mathematical models.

[0111] Step 2:

[0112] The server builds a generative AI model using pre-processed data. Based on this model, it generates optimal operating instructions for heavy machinery. The input is pre-processed data, and the output is specific operating instructions. Feature extraction techniques and machine learning algorithms are utilized to propose optimal actions tailored to on-site conditions.

[0113] Step 3:

[0114] Operation instructions generated from the server are transmitted to the terminal via the communication line. The input is the operation instructions generated by the server, and the output is the control command to the terminal. The instructions are encoded in a format that the terminal can understand, according to the communication protocol.

[0115] Step 4:

[0116] The terminal remotely controls heavy machinery via communication devices based on received instructions. The input is the received control command, and the output is the motion of the heavy machinery. The control signals are converted, and specific actions are performed to control the various actuators of the heavy machinery.

[0117] Step 5:

[0118] Users monitor the operation of heavy machinery and the surrounding conditions in real time via a mobile device display. Inputs are operational information from the heavy machinery and environmental data, while output is information displayed on the user interface. The visualized information allows users to quickly grasp the current situation.

[0119] Step 6:

[0120] The server uses an anomaly detection algorithm to monitor for potential anomalies during operation and immediately sends an alarm to the mobile device upon detection. Inputs are operational data and environmental data, and output is an alarm notification. Based on anomaly analysis, it draws attention and requests prompt action from the user.

[0121] Step 7:

[0122] When an anomaly is reported from a mobile device, the user takes immediate action. The input for the anomaly is an alarm notification, and the output is a corrected operation instruction. Depending on the situation, the user can issue manual instructions and coordinate with the server to replan operations.

[0123] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0124] This invention relates to a remote control system that combines an emotion engine to achieve unmanned operation of heavy machinery, and a specific embodiment is shown below. This system consists of four main components: a server, a terminal, a user, and an emotion engine.

[0125] The server collects operational data from sensors mounted on heavy machinery, preprocesses this data, and builds a machine learning model. This model generates commands to optimize the operation of the heavy machinery. The server also includes an algorithm to acquire the user's emotional state and reflect it in the operational commands.

[0126] The terminal remotely operates heavy machinery based on commands received from the server. It generates the control signals necessary for the operation of the heavy machinery and performs the specific operations. Furthermore, the terminal feeds back the operating status of the heavy machinery and the user's emotional state to the server, contributing to improved processing accuracy.

[0127] The user acts as a monitor in the control room, while their emotional state is evaluated in real time by an emotion engine. By taking into account the user's stress and attention levels, it supports safe and effective operation. In the event of an anomaly, manual intervention can be performed immediately, but the emotion engine ensures that the intervention is even more appropriate.

[0128] The emotion engine analyzes the user's emotions in real time based on their facial expressions, tone of voice, and word choice. If the obtained emotion data affects the overall system operation, the server immediately suggests the optimal solution and adjusts the operation commands as needed. In this way, the operation of the heavy machinery and the user's psychological burden are optimized simultaneously.

[0129] A concrete example is debris removal work using heavy machinery at a construction site. The server calculates the optimal operation of the heavy machinery, taking into account data from sensors and the user's stress level. The terminal controls the heavy machinery based on this calculation, achieving both safety and efficiency simultaneously. The user can monitor the entire operation with the support of the emotion engine and intervene manually in emergencies.

[0130] This configuration allows the system to significantly advance the unmanned operation of heavy machinery while also ensuring the safety and psychological well-being of users.

[0131] The following describes the processing flow.

[0132] Step 1:

[0133] The server collects operational data in real time from sensors mounted on the heavy machinery. This includes the machinery's position, speed, tilt, and information about the surrounding environment. Simultaneously, it also collects user emotion data from the emotion engine.

[0134] Step 2:

[0135] The server preprocesses the collected operational and sentiment data. It removes noise from the data and converts it into a format suitable for input to the machine learning model to be used.

[0136] Step 3:

[0137] The server updates the machine learning model based on pre-processed data. This model is used to generate optimal operating commands for heavy machinery, taking into account the environment and emotional state.

[0138] Step 4:

[0139] The server sends the generated operation commands to the terminal. The commands include specific control information such as the movement and path of the heavy machinery's arm.

[0140] Step 5:

[0141] The terminal remotely controls heavy machinery based on operation commands received from the server. It sends necessary control signals to ensure the heavy machinery operates according to the program.

[0142] Step 6:

[0143] The terminal reports the operating status of the heavy machinery to the server. At the same time, the emotion engine also feeds back user emotion data that it has analyzed to the server.

[0144] Step 7:

[0145] Users can monitor the operating status of heavy machinery in real time through the monitoring device. They can determine if the system is functioning correctly and perform manual operations if necessary.

[0146] Step 8:

[0147] The server continuously evaluates the user's emotional state, and if any stressors or decreased concentration detected affect operations, it generates new instructions for adjustment.

[0148] Step 9:

[0149] Based on user feedback and operational data, the server readjusts its machine learning models for subsequent tasks, improving overall operational efficiency.

[0150] (Example 2)

[0151] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0152] Conventional heavy equipment operation systems struggle to achieve both efficiency and safety in unmanned operation, and furthermore, the emotional stress of the operator can affect system performance. This has resulted in limitations in optimizing operations and preventing accidents.

[0153] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0154] In this invention, the server includes a computing device means for collecting and preprocessing operational information, a computing device means for constructing a predictive model based on the preprocessed information and generating machine operation commands, and a communication device means for remotely controlling the machine based on the commands received from the computing device. This enhances efficiency and safety in the unmanned operation of heavy machinery, reduces emotional stress on the operator, and enables optimal operation.

[0155] "Operational information" refers to data related to the operation of a machine, collected from sensors and other devices.

[0156] "Preprocessing" refers to removing noise and outliers from collected data and formatting it to make it easier to analyze.

[0157] "Computing device means" refers to a device that processes data and performs calculations for analysis and prediction.

[0158] A "predictive model" is a mathematical model used to generate instructions for optimizing machine operation based on collected data.

[0159] "Communication device means" refers to a device for sending and receiving information between a server and a machine or terminal in a remote location.

[0160] "Monitoring tools" are means for monitoring the state of a system and intervening or adjusting as needed.

[0161] "User's psychological state" refers to the mental state of the person operating the system, including their emotions, stress levels, and attention span.

[0162] "Emotional analysis methods" are techniques for analyzing a user's psychological state based on facial expressions, tone of voice, and other factors.

[0163] "Feedback" refers to providing information based on the system's operation and user status to facilitate further improvements and adjustments.

[0164] This invention is a remote control system that enables unmanned operation of heavy machinery while reducing the psychological stress on operators. The system mainly consists of a server, terminals, users, and an emotion analysis module.

[0165] The server plays a central role in this system, collecting operational information in real time from various sensors mounted on the heavy machinery. This data includes GPS location information, engine status, and speed sensor data. The server preprocesses this data, removing noise and correcting for outliers. It also generates a predictive model based on the preprocessed data. This predictive model is built using machine learning techniques on a specific platform, utilizing software such as Python and TensorFlow.

[0166] The terminal receives commands transmitted from the server and precisely controls the operation of heavy machinery. For example, in excavation work at a construction site, the terminal accurately moves the arm of the heavy machinery to a specified coordinate, ensuring safe and efficient work. This operation is performed by control signals transmitted via a communication network.

[0167] The user has the role of monitoring the entire system through their terminal and intervening as needed. The user's psychological state is analyzed by an emotion analysis module. This module uses cameras and microphones to evaluate the user's facial expressions, tone of voice, posture, etc., and quantifies stress levels and attention. As a result, the system can generate operational commands that take the operator's emotional state into account.

[0168] As a concrete example, in debris removal work, the server calculates the optimal operation command considering location information obtained from sensors and the user's stress level. The terminal then controls heavy machinery based on this, improving safety and efficiency. The user can monitor the system's operation in real time and intervene manually if an unforeseen situation occurs.

[0169] An example of a prompt for a generating AI model is, "Suggest how to adjust heavy equipment operation commands when the user is under high stress." This prompt prompts the AI ​​model to generate the optimal operation strategy for the situation, improving system performance.

[0170] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0171] Step 1:

[0172] The server collects operational information in real time from sensors mounted on heavy machinery. Inputs include GPS location data, acceleration data, engine status, and other sensor data. The server preprocesses this data, performing noise reduction and correcting for outliers, to output a clean dataset. This processing prepares data suitable for analysis.

[0173] Step 2:

[0174] The server builds a predictive model using a generative AI model based on preprocessed data. The input is the clean data obtained in step 1, and based on this, the server creates a model to generate optimal operation commands for heavy machinery. The output is the optimal operation command for a specific task. This command is dynamically updated through a machine learning algorithm.

[0175] Step 3:

[0176] The terminal receives operation commands transmitted from the server and generates specific control signals. The input is the operation command obtained in step 2, and the terminal remotely controls each component of the heavy machinery based on it. The output is specific control signals regarding the direction and speed of the heavy machinery. This ensures safe and precise operation of the heavy machinery.

[0177] Step 4:

[0178] The user is monitored in real time through an emotion analysis module, and their psychological state is evaluated. Inputs are biometric data such as the user's facial expressions and voice tone, which the server uses to analyze the user's stress level and attention span. Outputs are feedback data based on the user's psychological state. This allows the system to adjust operational commands to take the user's condition into consideration.

[0179] Step 5:

[0180] The terminal sends data on the operating status of the heavy machinery and the user's psychological state as feedback to the server. The input is the current operating status of the heavy machinery and environmental data, which the terminal analyzes to generate feedback information to improve the accuracy of the equipment's operation. The output is evaluation data that helps further optimize the entire system. This feedback enables dynamic adjustments to the system and reduces the burden on the operator.

[0181] (Application Example 2)

[0182] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0183] In unmanned operation of heavy machinery, it is necessary to consider the psychological state of the human operator in order to improve safety and efficiency. However, current systems have difficulty adjusting operations in real time to reflect the user's emotions and stress levels, which can increase the user's psychological burden. In addition, there is a lack of feedback mechanisms to achieve optimal operation, making it a challenge to improve the precision of heavy machinery operation.

[0184] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0185] In this invention, the server includes a computing device means for collecting and preprocessing operational information, a computing device means for constructing a mathematical learning model based on the preprocessed information and generating commands for operating heavy machinery, and an emotion analysis means for evaluating the user's psychological state and adjusting the operation commands accordingly. This makes it possible to optimize heavy machinery operation appropriately, taking the user's emotions into consideration.

[0186] "Operational information" refers to various data related to the operation of heavy machinery, including location information, environmental conditions, and operation history.

[0187] A "computational device" is a device that processes collected operational information and constructs the necessary mathematical learning model.

[0188] A "mathematical learning model" is a model constructed based on collected data and is used to generate optimal commands for operating heavy machinery.

[0189] "Device means" refers to a device that controls heavy machinery based on operating commands received from a computing device, enabling remote operation.

[0190] "User means" refers to a user interface that monitors the operation of heavy machinery and makes adjustments as needed.

[0191] "Psychological state" refers to the user's internal state, including emotions, stress levels, and attention span, and is evaluated in real time using emotion analysis tools.

[0192] An "emotional analysis tool" is an analytical mechanism that evaluates the user's psychological state and adjusts the operation commands based on that evaluation.

[0193] A "feedback mechanism" is a system that compares the operation history of heavy machinery with the command content and updates the mathematical learning model as needed.

[0194] The system for realizing this invention mainly consists of a server including a computing device, device means, user means, and emotion analysis means. The server collects operation information from various sensors mounted on the heavy machinery and preprocesses its relationships. Next, it constructs a mathematical learning model based on the preprocessed information and generates optimal heavy machinery operation commands. The computing device uses Python or TensorFlow to process data and build models.

[0195] The device remotely controls heavy machinery based on generated operating commands. In this system, a user interface using React Native provides the user with information on the operating status of the heavy machinery and the content of the commands.

[0196] The user's control system allows for real-time monitoring of the heavy machinery's operation and manual adjustments as needed. Furthermore, an emotion analysis system evaluates the user's psychological state and appropriately adjusts the driving commands if the user experiences significant stress or decreased attention.

[0197] As an example, consider crane operation during the construction of a high-rise building at a large-scale construction site. In this scenario, the emotion analysis system analyzes the user's facial expressions and tone of voice in real time. If the user begins to feel stressed, the server automatically adjusts the crane's operating speed based on this information. This reduces the psychological burden on the user while ensuring safety at the site.

[0198] An example of a prompt for a generated AI model might be, "Explain how to adjust the crane's operation if the operator is under high stress." This sentence provides guidance for the model to make optimal operational adjustments that respond to the user's psychological state.

[0199] This configuration allows for the optimization and unmanned operation of heavy machinery, taking into account the user's psychological state, thereby improving safety and efficiency.

[0200] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0201] Step 1:

[0202] The server collects operational information in real time from various sensors mounted on the heavy machinery. The input is data acquired from the sensors, including location information, speed, and operation history. The server preprocesses this data, removing noise and correcting outliers to generate clean and reliable data. The output is a cleaned dataset.

[0203] Step 2:

[0204] The server builds a mathematical learning model using a cleaned dataset. Here, machine learning libraries such as TensorFlow are utilized to learn the optimal operating commands for heavy machinery from the dataset. The input is pre-processed data, and the output is the optimal operating commands for the heavy machinery. This process allows, for example, the detection of specific operating patterns and the generation of efficient operating procedures based on them.

[0205] Step 3:

[0206] The calculated operating commands are transmitted to the equipment. The terminal remotely controls the heavy machinery based on these operating commands. The input is the operating commands, and the output is the actual operation of the heavy machinery. Specifically, the position of the heavy machinery's arm and crane are adjusted to perform optimal operation.

[0207] Step 4:

[0208] The user's psychological state is evaluated through an emotion analysis system. Emotion-related data, such as the user's facial expressions and voice tone, is used as input, and the system measures stress and attention in real time based on this data. The output is an evaluation result regarding the user's psychological state.

[0209] Step 5:

[0210] The server adjusts the operating commands as needed based on the evaluation results obtained by the emotion analysis system. The input is the user's psychological evaluation result, and the output is the adjusted operating command. As a result, if the user is experiencing stress, the commands are adjusted so that the heavy machinery operates more safely.

[0211] Step 6:

[0212] Based on the overall system operation history and user feedback, the server dynamically updates its mathematical learning model. The input is the operation history and user feedback, which are used to learn new heavy equipment operation patterns. The output is the updated mathematical learning model. This feedback mechanism allows the system to improve accuracy over time, enabling safer and more efficient operation.

[0213] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0214] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0215] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0216] [Second Embodiment]

[0217] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0218] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0219] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0220] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0221] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0222] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0223] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0224] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0225] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0226] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0227] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0228] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0229] This invention relates to a remote control system for realizing unmanned operation of heavy machinery, and a specific embodiment is shown below. This system is assembled from three main components: a server, a terminal, and a user.

[0230] The server collects operational data from various sensors attached to the heavy machinery. This data includes the position, speed, orientation, and status of the arm and shovel. The server preprocesses this raw data and extracts features to be input into a machine learning model. This process builds a machine learning model that mimics the operation of the heavy machinery. Furthermore, the server uses this model to generate optimal operational commands tailored to the site conditions.

[0231] The terminal receives commands from the server and remotely controls the heavy machinery. Based on the commands, it transmits specific control signals to the heavy machinery, precisely controlling operations such as arm movement, rotation, and forward movement. The terminal also transmits the operating status of the heavy machinery and surrounding environmental data to the server in real time, providing feedback.

[0232] The user is responsible for monitoring the operation of heavy machinery from the control room. Based on monitoring data transmitted from the terminal, the user confirms that the operation of the heavy machinery is proceeding as planned. In the event of an unexpected situation, the user can also correct the situation by performing manual operations. After the operation, the user evaluates the results and provides feedback to the server to promote optimal automation of operations in various situations.

[0233] As a concrete example, consider debris removal work at a construction site. The server recognizes the shape and location of the debris piles using data from sensors and calculates the optimal removal method. The terminal controls heavy machinery based on the calculation results, performing operations to remove the debris safely and efficiently. The user monitors the entire operation and makes adjustments as needed.

[0234] In this way, this system can highly automate the unmanned operation of heavy machinery, thereby addressing the labor shortage problem and improving the working environment.

[0235] The following describes the processing flow.

[0236] Step 1:

[0237] The server collects operational data in real time from sensors mounted on the heavy machinery. This data includes location information, speed, and video and audio data showing the surrounding environment.

[0238] Step 2:

[0239] The server preprocesses the collected data, performing noise reduction and data normalization, and converting it into a format usable by machine learning models.

[0240] Step 3:

[0241] The server updates the machine learning model using the pre-processed data. This model learns an optimization algorithm for heavy equipment operation and generates the necessary operation commands.

[0242] Step 4:

[0243] The server sends the generated operation commands to the terminal. The commands include specific instructions for movement, such as changing the angle of the arm or the direction of movement of the vehicle body.

[0244] Step 5:

[0245] The terminal remotely controls heavy machinery based on commands received from the server. It generates appropriate control signals to execute the operations of the heavy machinery.

[0246] Step 6:

[0247] The terminal feeds back the operating status of the heavy machinery to the server. Based on the actual operating data, it provides the information necessary to generate the next command.

[0248] Step 7:

[0249] The user monitors the operating status in real time. They verify that the system is working stably and take manual control if an abnormality occurs.

[0250] Step 8:

[0251] After completing a task, the user evaluates the operation history and results of the heavy machinery. This allows them to provide feedback to the server for improving future operations.

[0252] (Example 1)

[0253] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0254] To safely and efficiently operate large machinery such as heavy equipment remotely and unmanned, real-time situational awareness and the generation of appropriate operating commands are required. Furthermore, a feedback mechanism capable of responding to unforeseen circumstances is necessary. However, conventional systems struggle to meet these requirements, resulting in challenges such as decreased labor productivity and safety risks.

[0255] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0256] In this invention, the server includes an information processing device means that collects operational information and performs preprocessing including noise reduction and normalization; an information processing device means that extracts characteristics based on the preprocessed information, constructs a learning model using a generation AI algorithm, and generates machine operation commands; and a control device means that remotely operates the machine based on the commands received from the information processing device and generates specific control signals. This enables efficient and safe operation of heavy machinery in real time from a remote location.

[0257] "Operational information" refers to data that indicates the state and behavior of the machine, including position information, speed, and arm movement.

[0258] "Information processing device means" refers to a device used to receive operation information and perform noise reduction, normalization, and characteristic extraction.

[0259] A "generative AI algorithm" refers to an algorithm that uses data to perform pattern recognition and prediction, and generates commands that are appropriate for new situations.

[0260] A "learning model" refers to a mathematical model that is built using past data and used to optimize future operations.

[0261] "Control device means" refers to a device for remotely operating heavy machinery and other machines based on received commands.

[0262] "Supervision device means" refers to a device that monitors the operating status of a machine and allows for manual operation in response to unforeseen circumstances.

[0263] A "feedback mechanism" refers to a system that compares actual actions with command content and updates the learning model based on the results.

[0264] This invention relates to an advanced automation system for realizing unmanned operation of heavy machinery. This system mainly consists of a server, terminals, and users, and each element works together to achieve efficient machine operation.

[0265] The server collects data from sensors attached to heavy machinery. This data includes the machinery's position, speed, and arm movement. The server receives this data in real time and performs preprocessing such as noise reduction and data normalization. Furthermore, it uses machine learning frameworks such as TensorFlow and PyTorch to extract characteristics from the data and build a learning model. This model uses generative AI algorithms to generate optimal operating commands for the heavy machinery.

[0266] The terminal receives operation commands transmitted from the server and generates specific control signals to control the heavy machinery based on those commands. This allows for precise remote operation of various aspects of the heavy machinery, such as arm movement, body rotation, and direction of travel adjustment. The terminal continuously feeds back the heavy machinery's operation data to the server, providing information to improve the accuracy of the operation.

[0267] In the control room, the user monitors the operation data of the heavy machinery provided from the terminal. The user can verify that the heavy machinery is operating as planned and can intervene manually if an anomaly is detected. This plays a crucial role in ensuring operational safety. Furthermore, after completing a task, the user reviews the results and provides feedback to the server, contributing to the optimization of future tasks.

[0268] As a concrete example, in the case of debris removal work at a construction site, the server analyzes the shape and location of the debris based on data from sensors and calculates the optimal removal method. The terminal operates heavy machinery according to these instructions to efficiently remove the debris. The user monitors this work and makes adjustments as needed. An example of a prompt message is, "Please tell me how to design an AI model that generates optimal heavy machinery operation instructions based on the shape and location of the pile of debris."

[0269] This invention aims to efficiently and safely achieve unmanned operation of heavy machinery through the coordination of these elements, thereby addressing labor shortages and improving working conditions.

[0270] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0271] Step 1:

[0272] The server collects operational information in real time from various sensors attached to heavy machinery. Inputs include location information from GPS sensors, velocity information from accelerometers, and arm movement status information from motion sensors. The server aggregates this input data, removes noise and outliers, and outputs it as normalized data. This process ensures data quality.

[0273] Step 2:

[0274] The server extracts features using preprocessed data. The input is the denoised data from step 1, from which characteristics related to the operation of heavy machinery are extracted. For example, this includes the angle of the arm and the open / closed state of the shovel at a specific time. The output is a dataset containing these features, which is used as input data for the generative AI model. This process also performs dimensionality reduction of the data as needed.

[0275] Step 3:

[0276] The server builds a machine learning model based on the extracted features. The feature data obtained in step 2 is given as input, and a machine learning framework such as TensorFlow is used to generate the trained model. The output is a trained model for predicting and optimizing the operation of heavy machinery. The model is used to generate operation commands according to site conditions.

[0277] Step 4:

[0278] The server generates optimal operating commands using the constructed learning model. The model generated in step 3 receives real-time field conditions as input data and calculates how the heavy machinery should operate. The output is a set of specific operating commands, which are sent to the terminal. These commands include instructions for arm movement and the direction of the machine's movement.

[0279] Step 5:

[0280] The terminal receives the operation commands sent from the server and remotely operates the heavy machinery based on them. The input is the operation commands generated in step 4, and the terminal generates specific control signals based on this. The output is the control signals to the heavy machinery, which accurately control the operation of the arm and the movement of the body. Furthermore, the terminal monitors the situation and feeds back the operation data to the server.

[0281] Step 6:

[0282] The user monitors the operating state of the heavy machinery based on the feedback data from the terminal. The input is the real-time operation data provided by the terminal, and the user refers to this to confirm whether the heavy machinery is operating as planned. If a problem occurs, the user intervenes manually and adjusts the operation. The output is the evaluation and correction instructions by the user, ensuring the safety and efficiency of the system.

[0283] (Application Example 1)

[0284] Next, Application Example 1 will be described. In the following description, the data processing device 12 is referred to as the "server", and the smart glasses 214 are referred to as the "terminal".

[0285] In the conventional remote operation system for heavy machinery, there is a lack of a method that enables visualization of the operation situation, immediate detection of abnormalities, and further enables rapid response by on-site supervisors. Therefore, improvement in on-site safety and work efficiency is required. Furthermore, there is an issue that it is difficult for the remote operation system to flexibly respond to various environments.

[0286] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following respective means.

[0287] In this invention, the server includes an information processing means for collecting and pre-processing operational information, an information processing means for constructing a mathematical model based on the pre-processed information and generating instructions for operating heavy machinery, and a communication device means for remotely operating heavy machinery based on instructions received from the information processing device. This makes it possible to visualize the operation of heavy machinery, respond immediately when an anomaly is detected, and allow site supervisors to quickly correct instructions. Furthermore, it enables safe and efficient work in a variety of site environments.

[0288] "Operational information" refers to all data related to the operation of heavy machinery, including information such as position, speed, orientation, and operating status obtained from sensors and other devices.

[0289] "Information processing means" refers to a device or software equipped with data processing functions for pre-processing collected operational information and using it to construct mathematical models or generate operation instructions.

[0290] A "mathematical model" refers to a model constructed using mathematical methods based on operational information in order to mimic and optimize the operation of heavy machinery.

[0291] "Communication device means" refers to a device or system that has a communication function for transmitting operation instructions received from an information processing device to heavy machinery and for remotely controlling the heavy machinery.

[0292] "Human-operated means" refers to means that provide an interface for human operators to monitor the operation of heavy machinery, enabling adjustments and modifications of instructions as needed.

[0293] "Mobile devices" refer to portable electronic devices such as smartphones and tablets that site supervisors use to visualize the operation status of heavy machinery and to correct manual instructions.

[0294] "Display means" refers to a function or device for visually displaying the operating status of heavy machinery or information from sensors on a mobile device.

[0295] "Alarming device" refers to a device or program that has an alert function to detect abnormalities or dangers in real time and immediately notify the user.

[0296] The system for implementing this invention combines information processing means, communication device means, human user means, mobile terminal and display means to realize unmanned remote operation of heavy machinery.

[0297] The server collects operational information from various sensors installed on heavy machinery and preprocesses this data. The preprocessed data is used to build a mathematical model, and after the model is built, optimal instructions for operating the heavy machinery are generated. This process utilizes generative AI models as data processing and computing technology. Furthermore, the server uses information management software to analyze operational information in real time and generate optimal instructions.

[0298] The terminal receives instructions generated by the server via a communication line and controls various operations of the heavy machinery. The communication device converts these instructions into specific control signals, enabling remote control of the heavy machinery.

[0299] Users can use their mobile devices to check operational status data transmitted from the server in real time. The devices are equipped with display devices that visually show the operating status of heavy machinery and environmental information. In addition, they allow for human intervention, such as fine-tuning of operations and manual intervention in case of abnormalities.

[0300] Anomaly detection is a mechanism in which the server quickly detects anomalies and notifies the user through an alarm system on a mobile device. For example, if a mechanical anomaly or safety risk occurs during operation, an alert is immediately issued, allowing the user to respond quickly.

[0301] As a specific example, when performing debris removal work, the server acquires shape data in real time and proposes the most efficient work route to the user. By referring to this, the user can improve the safety and efficiency of the work when proceeding with the work.

[0302] As an example of the prompt sentence for the generation AI model, the content is something like "Please propose an efficient debris removal method based on the current terrain data." Thus, it becomes possible for the server to provide advanced operation instructions based on the data.

[0303] The flow of the specific process in Application Example 1 will be described using FIG. 12.

[0304] Step 1:

[0305] The server collects operation information such as position, speed, direction, and surrounding environment information in real time from various sensors installed on the heavy machinery. The input is raw data from the sensors, and the output is preprocessed data. The data is shaped through noise filtering and normalization and processed into a form suitable for the mathematical model.

[0306] Step 2:

[0307] The server constructs a generation AI model using the preprocessed data. Based on this model, it generates optimal operation instructions for the heavy machinery. The input is the preprocessed data, and the output is specific operation instructions. Utilize feature extraction technology and machine learning algorithms to propose optimal operations according to the on-site conditions.

[0308] Step 3:

[0309] The operation instructions generated by the server are transmitted to the terminal through the communication line. The input is the operation instructions generated by the server, and the output is a control command to the terminal. Encoding according to the communication protocol, the instruction content is encoded into a form that the terminal can understand.

[0310] Step 4:

[0311] The terminal remotely controls heavy machinery via communication devices based on received instructions. The input is the received control command, and the output is the motion of the heavy machinery. The control signals are converted, and specific actions are performed to control the various actuators of the heavy machinery.

[0312] Step 5:

[0313] Users monitor the operation of heavy machinery and the surrounding conditions in real time via a mobile device display. Inputs are operational information from the heavy machinery and environmental data, while output is information displayed on the user interface. The visualized information allows users to quickly grasp the current situation.

[0314] Step 6:

[0315] The server uses an anomaly detection algorithm to monitor for potential anomalies during operation and immediately sends an alarm to the mobile device upon detection. Inputs are operational data and environmental data, and output is an alarm notification. Based on anomaly analysis, it draws attention and requests prompt action from the user.

[0316] Step 7:

[0317] When an anomaly is reported from a mobile device, the user takes immediate action. The input for the anomaly is an alarm notification, and the output is a corrected operation instruction. Depending on the situation, the user can issue manual instructions and coordinate with the server to replan operations.

[0318] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0319] This invention relates to a remote control system that combines an emotion engine to achieve unmanned operation of heavy machinery, and a specific embodiment is shown below. This system consists of four main components: a server, a terminal, a user, and an emotion engine.

[0320] The server collects operational data from sensors mounted on heavy machinery, preprocesses this data, and builds a machine learning model. This model generates commands to optimize the operation of the heavy machinery. The server also includes an algorithm to acquire the user's emotional state and reflect it in the operational commands.

[0321] The terminal remotely operates heavy machinery based on commands received from the server. It generates the control signals necessary for the operation of the heavy machinery and performs the specific operations. Furthermore, the terminal feeds back the operating status of the heavy machinery and the user's emotional state to the server, contributing to improved processing accuracy.

[0322] The user acts as a monitor in the control room, while their emotional state is evaluated in real time by an emotion engine. By taking into account the user's stress and attention levels, it supports safe and effective operation. In the event of an anomaly, manual intervention can be performed immediately, but the emotion engine ensures that the intervention is even more appropriate.

[0323] The emotion engine analyzes the user's emotions in real time based on their facial expressions, tone of voice, and word choice. If the obtained emotion data affects the overall system operation, the server immediately suggests the optimal solution and adjusts the operation commands as needed. In this way, the operation of the heavy machinery and the user's psychological burden are optimized simultaneously.

[0324] A concrete example is debris removal work using heavy machinery at a construction site. The server calculates the optimal operation of the heavy machinery, taking into account data from sensors and the user's stress level. The terminal controls the heavy machinery based on this calculation, achieving both safety and efficiency simultaneously. The user can monitor the entire operation with the support of the emotion engine and intervene manually in emergencies.

[0325] This configuration allows the system to significantly advance the unmanned operation of heavy machinery while also ensuring the safety and psychological well-being of users.

[0326] The following describes the processing flow.

[0327] Step 1:

[0328] The server collects operational data in real time from sensors mounted on the heavy machinery. This includes the machinery's position, speed, tilt, and information about the surrounding environment. Simultaneously, it also collects user emotion data from the emotion engine.

[0329] Step 2:

[0330] The server preprocesses the collected operational and sentiment data. It removes noise from the data and converts it into a format suitable for input to the machine learning model to be used.

[0331] Step 3:

[0332] The server updates the machine learning model based on pre-processed data. This model is used to generate optimal operating commands for heavy machinery, taking into account the environment and emotional state.

[0333] Step 4:

[0334] The server sends the generated operation commands to the terminal. The commands include specific control information such as the movement and path of the heavy machinery's arm.

[0335] Step 5:

[0336] The terminal remotely controls heavy machinery based on operation commands received from the server. It sends necessary control signals to ensure the heavy machinery operates according to the program.

[0337] Step 6:

[0338] The terminal reports the operating status of the heavy machinery to the server. At the same time, the emotion engine also feeds back user emotion data that it has analyzed to the server.

[0339] Step 7:

[0340] Users can monitor the operating status of heavy machinery in real time through the monitoring device. They can determine if the system is functioning correctly and perform manual operations if necessary.

[0341] Step 8:

[0342] The server continuously evaluates the user's emotional state, and if any stressors or decreased concentration detected affect operations, it generates new instructions for adjustment.

[0343] Step 9:

[0344] Based on user feedback and operational data, the server readjusts its machine learning models for subsequent tasks, improving overall operational efficiency.

[0345] (Example 2)

[0346] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0347] Conventional heavy equipment operation systems struggle to achieve both efficiency and safety in unmanned operation, and furthermore, the emotional stress of the operator can affect system performance. This has resulted in limitations in optimizing operations and preventing accidents.

[0348] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0349] In this invention, the server includes a computing device means for collecting and preprocessing operational information, a computing device means for constructing a predictive model based on the preprocessed information and generating machine operation commands, and a communication device means for remotely controlling the machine based on the commands received from the computing device. This enhances efficiency and safety in the unmanned operation of heavy machinery, reduces emotional stress on the operator, and enables optimal operation.

[0350] "Operational information" refers to data related to the operation of a machine, collected from sensors and other devices.

[0351] "Preprocessing" refers to removing noise and outliers from collected data and formatting it to make it easier to analyze.

[0352] "Computing device means" refers to a device that processes data and performs calculations for analysis and prediction.

[0353] A "predictive model" is a mathematical model used to generate instructions for optimizing machine operation based on collected data.

[0354] "Communication device means" refers to a device for sending and receiving information between a server and a machine or terminal in a remote location.

[0355] "Monitoring tools" are means for monitoring the state of a system and intervening or adjusting as needed.

[0356] "User's psychological state" refers to the mental state of the person operating the system, including their emotions, stress levels, and attention span.

[0357] "Emotional analysis methods" are techniques for analyzing a user's psychological state based on facial expressions, tone of voice, and other factors.

[0358] "Feedback" refers to providing information based on the system's operation and user status to facilitate further improvements and adjustments.

[0359] This invention is a remote control system that enables unmanned operation of heavy machinery while reducing the psychological stress on operators. The system mainly consists of a server, terminals, users, and an emotion analysis module.

[0360] The server plays a central role in this system, collecting operational information in real time from various sensors mounted on the heavy machinery. This data includes GPS location information, engine status, and speed sensor data. The server preprocesses this data, removing noise and correcting for outliers. It also generates a predictive model based on the preprocessed data. This predictive model is built using machine learning techniques on a specific platform, utilizing software such as Python and TensorFlow.

[0361] The terminal receives commands transmitted from the server and precisely controls the operation of heavy machinery. For example, in excavation work at a construction site, the terminal accurately moves the arm of the heavy machinery to a specified coordinate, ensuring safe and efficient work. This operation is performed by control signals transmitted via a communication network.

[0362] The user has the role of monitoring the entire system through their terminal and intervening as needed. The user's psychological state is analyzed by an emotion analysis module. This module uses cameras and microphones to evaluate the user's facial expressions, tone of voice, posture, etc., and quantifies stress levels and attention. As a result, the system can generate operational commands that take the operator's emotional state into account.

[0363] As a concrete example, in debris removal work, the server calculates the optimal operation command considering location information obtained from sensors and the user's stress level. The terminal then controls heavy machinery based on this, improving safety and efficiency. The user can monitor the system's operation in real time and intervene manually if an unforeseen situation occurs.

[0364] An example of a prompt for a generating AI model is, "Suggest how to adjust heavy equipment operation commands when the user is under high stress." This prompt prompts the AI ​​model to generate the optimal operation strategy for the situation, improving system performance.

[0365] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0366] Step 1:

[0367] The server collects operational information in real time from sensors mounted on heavy machinery. Inputs include GPS location data, acceleration data, engine status, and other sensor data. The server preprocesses this data, performing noise reduction and correcting for outliers, to output a clean dataset. This processing prepares data suitable for analysis.

[0368] Step 2:

[0369] The server builds a predictive model using a generative AI model based on preprocessed data. The input is the clean data obtained in step 1, and based on this, the server creates a model to generate optimal operation commands for heavy machinery. The output is the optimal operation command for a specific task. This command is dynamically updated through a machine learning algorithm.

[0370] Step 3:

[0371] The terminal receives operation commands transmitted from the server and generates specific control signals. The input is the operation command obtained in step 2, and the terminal remotely controls each component of the heavy machinery based on it. The output is specific control signals regarding the direction and speed of the heavy machinery. This ensures safe and precise operation of the heavy machinery.

[0372] Step 4:

[0373] The user is monitored in real time through an emotion analysis module, and their psychological state is evaluated. Inputs are biometric data such as the user's facial expressions and voice tone, which the server uses to analyze the user's stress level and attention span. Outputs are feedback data based on the user's psychological state. This allows the system to adjust operational commands to take the user's condition into consideration.

[0374] Step 5:

[0375] The terminal sends data on the operating status of the heavy machinery and the user's psychological state as feedback to the server. The input is the current operating status of the heavy machinery and environmental data, which the terminal analyzes to generate feedback information to improve the accuracy of the equipment's operation. The output is evaluation data that helps further optimize the entire system. This feedback enables dynamic adjustments to the system and reduces the burden on the operator.

[0376] (Application Example 2)

[0377] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0378] In unmanned operation of heavy machinery, it is necessary to consider the psychological state of the human operator in order to improve safety and efficiency. However, current systems have difficulty adjusting operations in real time to reflect the user's emotions and stress levels, which can increase the user's psychological burden. In addition, there is a lack of feedback mechanisms to achieve optimal operation, making it a challenge to improve the precision of heavy machinery operation.

[0379] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0380] In this invention, the server includes a computing device means for collecting and preprocessing operational information, a computing device means for constructing a mathematical learning model based on the preprocessed information and generating commands for operating heavy machinery, and an emotion analysis means for evaluating the user's psychological state and adjusting the operation commands accordingly. This makes it possible to optimize heavy machinery operation appropriately, taking the user's emotions into consideration.

[0381] "Operational information" refers to various data related to the operation of heavy machinery, including location information, environmental conditions, and operation history.

[0382] A "computational device" is a device that processes collected operational information and constructs the necessary mathematical learning model.

[0383] A "mathematical learning model" is a model constructed based on collected data and is used to generate optimal commands for operating heavy machinery.

[0384] "Device means" refers to a device that controls heavy machinery based on operating commands received from a computing device, enabling remote operation.

[0385] "User means" refers to a user interface that monitors the operation of heavy machinery and makes adjustments as needed.

[0386] "Psychological state" refers to the user's internal state, including emotions, stress levels, and attention span, and is evaluated in real time using emotion analysis tools.

[0387] An "emotional analysis tool" is an analytical mechanism that evaluates the user's psychological state and adjusts the operation commands based on that evaluation.

[0388] A "feedback mechanism" is a system that compares the operation history of heavy machinery with the command content and updates the mathematical learning model as needed.

[0389] The system for realizing this invention mainly consists of a server including a computing device, device means, user means, and emotion analysis means. The server collects operation information from various sensors mounted on the heavy machinery and preprocesses its relationships. Next, it constructs a mathematical learning model based on the preprocessed information and generates optimal heavy machinery operation commands. The computing device uses Python or TensorFlow to process data and build models.

[0390] The device remotely controls heavy machinery based on generated operating commands. In this system, a user interface using React Native provides the user with information on the operating status of the heavy machinery and the content of the commands.

[0391] The user's control system allows for real-time monitoring of the heavy machinery's operation and manual adjustments as needed. Furthermore, an emotion analysis system evaluates the user's psychological state and appropriately adjusts the driving commands if the user experiences significant stress or decreased attention.

[0392] As an example, consider crane operation during the construction of a high-rise building at a large-scale construction site. In this scenario, the emotion analysis system analyzes the user's facial expressions and tone of voice in real time. If the user begins to feel stressed, the server automatically adjusts the crane's operating speed based on this information. This reduces the psychological burden on the user while ensuring safety at the site.

[0393] An example of a prompt for a generated AI model might be, "Explain how to adjust the crane's operation if the operator is under high stress." This sentence provides guidance for the model to make optimal operational adjustments that respond to the user's psychological state.

[0394] This configuration allows for the optimization and unmanned operation of heavy machinery, taking into account the user's psychological state, thereby improving safety and efficiency.

[0395] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0396] Step 1:

[0397] The server collects operational information in real time from various sensors mounted on the heavy machinery. The input is data acquired from the sensors, including location information, speed, and operation history. The server preprocesses this data, removing noise and correcting outliers to generate clean and reliable data. The output is a cleaned dataset.

[0398] Step 2:

[0399] The server builds a mathematical learning model using a cleaned dataset. Here, machine learning libraries such as TensorFlow are utilized to learn the optimal operating commands for heavy machinery from the dataset. The input is pre-processed data, and the output is the optimal operating commands for the heavy machinery. This process allows, for example, the detection of specific operating patterns and the generation of efficient operating procedures based on them.

[0400] Step 3:

[0401] The calculated operating commands are transmitted to the equipment. The terminal remotely controls the heavy machinery based on these operating commands. The input is the operating commands, and the output is the actual operation of the heavy machinery. Specifically, the position of the heavy machinery's arm and crane are adjusted to perform optimal operation.

[0402] Step 4:

[0403] The user's psychological state is evaluated through an emotion analysis system. Emotion-related data, such as the user's facial expressions and voice tone, is used as input, and the system measures stress and attention in real time based on this data. The output is an evaluation result regarding the user's psychological state.

[0404] Step 5:

[0405] The server adjusts the operating commands as needed based on the evaluation results obtained by the emotion analysis system. The input is the user's psychological evaluation result, and the output is the adjusted operating command. As a result, if the user is experiencing stress, the commands are adjusted so that the heavy machinery operates more safely.

[0406] Step 6:

[0407] Based on the overall system operation history and user feedback, the server dynamically updates its mathematical learning model. The input is the operation history and user feedback, which are used to learn new heavy equipment operation patterns. The output is the updated mathematical learning model. This feedback mechanism allows the system to improve accuracy over time, enabling safer and more efficient operation.

[0408] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0409] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0410] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0411] [Third Embodiment]

[0412] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0413] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0414] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0415] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0416] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0417] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0418] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0419] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0420] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0421] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0422] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0423] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0424] This invention relates to a remote control system for realizing unmanned operation of heavy machinery, and a specific embodiment is shown below. This system is assembled from three main components: a server, a terminal, and a user.

[0425] The server collects operational data from various sensors attached to the heavy machinery. This data includes the position, speed, orientation, and status of the arm and shovel. The server preprocesses this raw data and extracts features to be input into a machine learning model. This process builds a machine learning model that mimics the operation of the heavy machinery. Furthermore, the server uses this model to generate optimal operational commands tailored to the site conditions.

[0426] The terminal receives commands from the server and remotely controls the heavy machinery. Based on the commands, it transmits specific control signals to the heavy machinery, precisely controlling operations such as arm movement, rotation, and forward movement. The terminal also transmits the operating status of the heavy machinery and surrounding environmental data to the server in real time, providing feedback.

[0427] The user is responsible for monitoring the operation of heavy machinery from the control room. Based on monitoring data transmitted from the terminal, the user confirms that the operation of the heavy machinery is proceeding as planned. In the event of an unexpected situation, the user can also correct the situation by performing manual operations. After the operation, the user evaluates the results and provides feedback to the server to promote optimal automation of operations in various situations.

[0428] As a concrete example, consider debris removal work at a construction site. The server recognizes the shape and location of the debris piles using data from sensors and calculates the optimal removal method. The terminal controls heavy machinery based on the calculation results, performing operations to remove the debris safely and efficiently. The user monitors the entire operation and makes adjustments as needed.

[0429] In this way, this system can highly automate the unmanned operation of heavy machinery, thereby addressing the labor shortage problem and improving the working environment.

[0430] The following describes the processing flow.

[0431] Step 1:

[0432] The server collects operational data in real time from sensors mounted on the heavy machinery. This data includes location information, speed, and video and audio data showing the surrounding environment.

[0433] Step 2:

[0434] The server preprocesses the collected data, performing noise reduction and data normalization, and converting it into a format usable by machine learning models.

[0435] Step 3:

[0436] The server updates the machine learning model using the pre-processed data. This model learns an optimization algorithm for heavy equipment operation and generates the necessary operation commands.

[0437] Step 4:

[0438] The server sends the generated operation commands to the terminal. The commands include specific instructions for movement, such as changing the angle of the arm or the direction of movement of the vehicle body.

[0439] Step 5:

[0440] The terminal remotely controls heavy machinery based on commands received from the server. It generates appropriate control signals to execute the operations of the heavy machinery.

[0441] Step 6:

[0442] The terminal feeds back the operating status of the heavy machinery to the server. Based on the actual operating data, it provides the information necessary to generate the next command.

[0443] Step 7:

[0444] The user monitors the operating status in real time. They verify that the system is working stably and take manual control if an abnormality occurs.

[0445] Step 8:

[0446] After completing a task, the user evaluates the operation history and results of the heavy machinery. This allows them to provide feedback to the server for improving future operations.

[0447] (Example 1)

[0448] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0449] To safely and efficiently operate large machinery such as heavy equipment remotely and unmanned, real-time situational awareness and the generation of appropriate operating commands are required. Furthermore, a feedback mechanism capable of responding to unforeseen circumstances is necessary. However, conventional systems struggle to meet these requirements, resulting in challenges such as decreased labor productivity and safety risks.

[0450] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0451] In this invention, the server includes an information processing device means that collects operational information and performs preprocessing including noise reduction and normalization; an information processing device means that extracts characteristics based on the preprocessed information, constructs a learning model using a generation AI algorithm, and generates machine operation commands; and a control device means that remotely operates the machine based on the commands received from the information processing device and generates specific control signals. This enables efficient and safe operation of heavy machinery in real time from a remote location.

[0452] "Operational information" refers to data that indicates the state and behavior of the machine, including position information, speed, and arm movement.

[0453] "Information processing device means" refers to a device used to receive operation information and perform noise reduction, normalization, and characteristic extraction.

[0454] A "generative AI algorithm" refers to an algorithm that uses data to perform pattern recognition and prediction, and generates commands that are appropriate for new situations.

[0455] A "learning model" refers to a mathematical model that is built using past data and used to optimize future operations.

[0456] "Control device means" refers to a device for remotely operating heavy machinery and other machines based on received commands.

[0457] "Supervision device means" refers to a device that monitors the operating status of a machine and allows for manual operation in response to unforeseen circumstances.

[0458] A "feedback mechanism" refers to a system that compares actual actions with command content and updates the learning model based on the results.

[0459] This invention relates to an advanced automation system for realizing unmanned operation of heavy machinery. This system mainly consists of a server, terminals, and users, and each element works together to achieve efficient machine operation.

[0460] The server collects data from sensors attached to heavy machinery. This data includes the machinery's position, speed, and arm movement. The server receives this data in real time and performs preprocessing such as noise reduction and data normalization. Furthermore, it uses machine learning frameworks such as TensorFlow and PyTorch to extract characteristics from the data and build a learning model. This model uses generative AI algorithms to generate optimal operating commands for the heavy machinery.

[0461] The terminal receives operation commands transmitted from the server and generates specific control signals to control the heavy machinery based on those commands. This allows for precise remote operation of various aspects of the heavy machinery, such as arm movement, body rotation, and direction of travel adjustment. The terminal continuously feeds back the heavy machinery's operation data to the server, providing information to improve the accuracy of the operation.

[0462] In the control room, the user monitors the operation data of the heavy machinery provided from the terminal. The user can verify that the heavy machinery is operating as planned and can intervene manually if an anomaly is detected. This plays a crucial role in ensuring operational safety. Furthermore, after completing a task, the user reviews the results and provides feedback to the server, contributing to the optimization of future tasks.

[0463] As a concrete example, in the case of debris removal work at a construction site, the server analyzes the shape and location of the debris based on data from sensors and calculates the optimal removal method. The terminal operates heavy machinery according to these instructions to efficiently remove the debris. The user monitors this work and makes adjustments as needed. An example of a prompt message is, "Please tell me how to design an AI model that generates optimal heavy machinery operation instructions based on the shape and location of the pile of debris."

[0464] This invention aims to efficiently and safely achieve unmanned operation of heavy machinery through the coordination of these elements, thereby addressing labor shortages and improving working conditions.

[0465] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0466] Step 1:

[0467] The server collects operational information in real time from various sensors attached to heavy machinery. Inputs include location information from GPS sensors, velocity information from accelerometers, and arm movement status information from motion sensors. The server aggregates this input data, removes noise and outliers, and outputs it as normalized data. This process ensures data quality.

[0468] Step 2:

[0469] The server extracts features using preprocessed data. The input is the denoised data from step 1, from which characteristics related to the operation of heavy machinery are extracted. For example, this includes the angle of the arm and the open / closed state of the shovel at a specific time. The output is a dataset containing these features, which is used as input data for the generative AI model. This process also performs dimensionality reduction of the data as needed.

[0470] Step 3:

[0471] The server builds a machine learning model based on the extracted features. The feature data obtained in step 2 is given as input, and a machine learning framework such as TensorFlow is used to generate the trained model. The output is a trained model for predicting and optimizing the operation of heavy machinery. The model is used to generate operation commands according to site conditions.

[0472] Step 4:

[0473] The server generates optimal operating commands using the constructed learning model. The model generated in step 3 receives real-time field conditions as input data and calculates how the heavy machinery should operate. The output is a set of specific operating commands, which are sent to the terminal. These commands include instructions for arm movement and the direction of the machine's movement.

[0474] Step 5:

[0475] The terminal receives operation commands transmitted from the server and remotely controls the heavy machinery based on them. The input is the operation command generated in step 4, and the terminal generates specific control signals based on this. The output is the control signal to the heavy machinery, precisely controlling the operation of the arm and the movement of the machine. Furthermore, the terminal monitors the situation and feeds back the operation data to the server.

[0476] Step 6:

[0477] The user monitors the operating status of heavy machinery based on feedback data from the terminal. The input is real-time operation data provided by the terminal, which the user refers to to verify that the heavy machinery is operating as planned. If a problem occurs, the user intervenes manually and adjusts the operation. The output consists of user evaluations and correction instructions, ensuring the safety and efficiency of the system.

[0478] (Application Example 1)

[0479] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0480] Conventional remote control systems for heavy machinery lack methods for visualizing operating conditions, immediately detecting anomalies, and enabling rapid response by site supervisors. Therefore, there is a need to improve safety and work efficiency on site. Furthermore, remote control systems face the challenge of not being able to flexibly adapt to diverse environments.

[0481] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0482] In this invention, the server includes an information processing means for collecting and pre-processing operational information, an information processing means for constructing a mathematical model based on the pre-processed information and generating instructions for operating heavy machinery, and a communication device means for remotely operating heavy machinery based on instructions received from the information processing device. This makes it possible to visualize the operation of heavy machinery, respond immediately when an anomaly is detected, and allow site supervisors to quickly correct instructions. Furthermore, it enables safe and efficient work in a variety of site environments.

[0483] "Operational information" refers to all data related to the operation of heavy machinery, including information such as position, speed, orientation, and operating status obtained from sensors and other devices.

[0484] "Information processing means" refers to a device or software equipped with data processing functions for pre-processing collected operational information and using it to construct mathematical models or generate operation instructions.

[0485] A "mathematical model" refers to a model constructed using mathematical methods based on operational information in order to mimic and optimize the operation of heavy machinery.

[0486] "Communication device means" refers to a device or system that has a communication function for transmitting operation instructions received from an information processing device to heavy machinery and for remotely controlling the heavy machinery.

[0487] "Human-operated means" refers to means that provide an interface for human operators to monitor the operation of heavy machinery, enabling adjustments and modifications of instructions as needed.

[0488] "Mobile devices" refer to portable electronic devices such as smartphones and tablets that site supervisors use to visualize the operation status of heavy machinery and to correct manual instructions.

[0489] "Display means" refers to a function or device for visually displaying the operating status of heavy machinery or information from sensors on a mobile device.

[0490] "Alarming device" refers to a device or program that has an alert function to detect abnormalities or dangers in real time and immediately notify the user.

[0491] The system for implementing this invention combines information processing means, communication device means, human user means, mobile terminal and display means to realize unmanned remote operation of heavy machinery.

[0492] The server collects operational information from various sensors installed on heavy machinery and preprocesses this data. The preprocessed data is used to build a mathematical model, and after the model is built, optimal instructions for operating the heavy machinery are generated. This process utilizes generative AI models as data processing and computing technology. Furthermore, the server uses information management software to analyze operational information in real time and generate optimal instructions.

[0493] The terminal receives instructions generated by the server via a communication line and controls various operations of the heavy machinery. The communication device converts these instructions into specific control signals, enabling remote control of the heavy machinery.

[0494] Users can use their mobile devices to check operational status data transmitted from the server in real time. The devices are equipped with display devices that visually show the operating status of heavy machinery and environmental information. In addition, they allow for human intervention, such as fine-tuning of operations and manual intervention in case of abnormalities.

[0495] Anomaly detection is a mechanism in which the server quickly detects anomalies and notifies the user through an alarm system on a mobile device. For example, if a mechanical anomaly or safety risk occurs during operation, an alert is immediately issued, allowing the user to respond quickly.

[0496] As a concrete example, when removing rubble, the server acquires shape data in real time and suggests the most efficient work route to the user. By using this as a reference, the user can improve the safety and efficiency of the work.

[0497] An example of a prompt message for a generated AI model is, "Based on the current terrain data, please propose an efficient method for debris removal." This allows the server to provide sophisticated, data-driven instructions.

[0498] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0499] Step 1:

[0500] The server collects operational information such as position, speed, direction, and surrounding environment information in real time from various sensors installed on heavy machinery. The input is raw data from the sensors, and the output is pre-processed data. The data is shaped through noise filtering and normalization, and processed into a form suitable for mathematical models.

[0501] Step 2:

[0502] The server builds a generative AI model using pre-processed data. Based on this model, it generates optimal operating instructions for heavy machinery. The input is pre-processed data, and the output is specific operating instructions. Feature extraction techniques and machine learning algorithms are utilized to propose optimal actions tailored to on-site conditions.

[0503] Step 3:

[0504] Operation instructions generated from the server are transmitted to the terminal via the communication line. The input is the operation instructions generated by the server, and the output is the control command to the terminal. The instructions are encoded in a format that the terminal can understand, according to the communication protocol.

[0505] Step 4:

[0506] The terminal remotely controls heavy machinery via communication devices based on received instructions. The input is the received control command, and the output is the motion of the heavy machinery. The control signals are converted, and specific actions are performed to control the various actuators of the heavy machinery.

[0507] Step 5:

[0508] Users monitor the operation of heavy machinery and the surrounding conditions in real time via a mobile device display. Inputs are operational information from the heavy machinery and environmental data, while output is information displayed on the user interface. The visualized information allows users to quickly grasp the current situation.

[0509] Step 6:

[0510] The server uses an anomaly detection algorithm to monitor for potential anomalies during operation and immediately sends an alarm to the mobile device upon detection. Inputs are operational data and environmental data, and output is an alarm notification. Based on anomaly analysis, it draws attention and requests prompt action from the user.

[0511] Step 7:

[0512] When an anomaly is reported from a mobile device, the user takes immediate action. The input for the anomaly is an alarm notification, and the output is a corrected operation instruction. Depending on the situation, the user can issue manual instructions and coordinate with the server to replan operations.

[0513] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0514] This invention relates to a remote control system that combines an emotion engine to achieve unmanned operation of heavy machinery, and a specific embodiment is shown below. This system consists of four main components: a server, a terminal, a user, and an emotion engine.

[0515] The server collects operational data from sensors mounted on heavy machinery, preprocesses this data, and builds a machine learning model. This model generates commands to optimize the operation of the heavy machinery. The server also includes an algorithm to acquire the user's emotional state and reflect it in the operational commands.

[0516] The terminal remotely operates heavy machinery based on commands received from the server. It generates the control signals necessary for the operation of the heavy machinery and performs the specific operations. Furthermore, the terminal feeds back the operating status of the heavy machinery and the user's emotional state to the server, contributing to improved processing accuracy.

[0517] The user acts as a monitor in the control room, while their emotional state is evaluated in real time by an emotion engine. By taking into account the user's stress and attention levels, it supports safe and effective operation. In the event of an anomaly, manual intervention can be performed immediately, but the emotion engine ensures that the intervention is even more appropriate.

[0518] The emotion engine analyzes the user's emotions in real time based on their facial expressions, tone of voice, and word choice. If the obtained emotion data affects the overall system operation, the server immediately suggests the optimal solution and adjusts the operation commands as needed. In this way, the operation of the heavy machinery and the user's psychological burden are optimized simultaneously.

[0519] A concrete example is debris removal work using heavy machinery at a construction site. The server calculates the optimal operation of the heavy machinery, taking into account data from sensors and the user's stress level. The terminal controls the heavy machinery based on this calculation, achieving both safety and efficiency simultaneously. The user can monitor the entire operation with the support of the emotion engine and intervene manually in emergencies.

[0520] This configuration allows the system to significantly advance the unmanned operation of heavy machinery while also ensuring the safety and psychological well-being of users.

[0521] The following describes the processing flow.

[0522] Step 1:

[0523] The server collects operational data in real time from sensors mounted on the heavy machinery. This includes the machinery's position, speed, tilt, and information about the surrounding environment. Simultaneously, it also collects user emotion data from the emotion engine.

[0524] Step 2:

[0525] The server preprocesses the collected operational and sentiment data. It removes noise from the data and converts it into a format suitable for input to the machine learning model to be used.

[0526] Step 3:

[0527] The server updates the machine learning model based on pre-processed data. This model is used to generate optimal operating commands for heavy machinery, taking into account the environment and emotional state.

[0528] Step 4:

[0529] The server sends the generated operation commands to the terminal. The commands include specific control information such as the movement and path of the heavy machinery's arm.

[0530] Step 5:

[0531] The terminal remotely controls heavy machinery based on operation commands received from the server. It sends necessary control signals to ensure the heavy machinery operates according to the program.

[0532] Step 6:

[0533] The terminal reports the operating status of the heavy machinery to the server. At the same time, the emotion engine also feeds back user emotion data that it has analyzed to the server.

[0534] Step 7:

[0535] Users can monitor the operating status of heavy machinery in real time through the monitoring device. They can determine if the system is functioning correctly and perform manual operations if necessary.

[0536] Step 8:

[0537] The server continuously evaluates the user's emotional state, and if any stressors or decreased concentration detected affect operations, it generates new instructions for adjustment.

[0538] Step 9:

[0539] Based on user feedback and operational data, the server readjusts its machine learning models for subsequent tasks, improving overall operational efficiency.

[0540] (Example 2)

[0541] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0542] Conventional heavy equipment operation systems struggle to achieve both efficiency and safety in unmanned operation, and furthermore, the emotional stress of the operator can affect system performance. This has resulted in limitations in optimizing operations and preventing accidents.

[0543] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0544] In this invention, the server includes a computing device means for collecting and preprocessing operational information, a computing device means for constructing a predictive model based on the preprocessed information and generating machine operation commands, and a communication device means for remotely controlling the machine based on the commands received from the computing device. This enhances efficiency and safety in the unmanned operation of heavy machinery, reduces emotional stress on the operator, and enables optimal operation.

[0545] "Operational information" refers to data related to the operation of a machine, collected from sensors and other devices.

[0546] "Preprocessing" refers to removing noise and outliers from collected data and formatting it to make it easier to analyze.

[0547] "Computing device means" refers to a device that processes data and performs calculations for analysis and prediction.

[0548] A "predictive model" is a mathematical model used to generate instructions for optimizing machine operation based on collected data.

[0549] "Communication device means" refers to a device for sending and receiving information between a server and a machine or terminal in a remote location.

[0550] "Monitoring tools" are means for monitoring the state of a system and intervening or adjusting as needed.

[0551] "User's psychological state" refers to the mental state of the person operating the system, including their emotions, stress levels, and attention span.

[0552] "Emotional analysis methods" are techniques for analyzing a user's psychological state based on facial expressions, tone of voice, and other factors.

[0553] "Feedback" refers to providing information based on the system's operation and user status to facilitate further improvements and adjustments.

[0554] This invention is a remote control system that enables unmanned operation of heavy machinery while reducing the psychological stress on operators. The system mainly consists of a server, terminals, users, and an emotion analysis module.

[0555] The server plays a central role in this system, collecting operational information in real time from various sensors mounted on the heavy machinery. This data includes GPS location information, engine status, and speed sensor data. The server preprocesses this data, removing noise and correcting for outliers. It also generates a predictive model based on the preprocessed data. This predictive model is built using machine learning techniques on a specific platform, utilizing software such as Python and TensorFlow.

[0556] The terminal receives commands transmitted from the server and precisely controls the operation of heavy machinery. For example, in excavation work at a construction site, the terminal accurately moves the arm of the heavy machinery to a specified coordinate, ensuring safe and efficient work. This operation is performed by control signals transmitted via a communication network.

[0557] The user has the role of monitoring the entire system through their terminal and intervening as needed. The user's psychological state is analyzed by an emotion analysis module. This module uses cameras and microphones to evaluate the user's facial expressions, tone of voice, posture, etc., and quantifies stress levels and attention. As a result, the system can generate operational commands that take the operator's emotional state into account.

[0558] As a concrete example, in debris removal work, the server calculates the optimal operation command considering location information obtained from sensors and the user's stress level. The terminal then controls heavy machinery based on this, improving safety and efficiency. The user can monitor the system's operation in real time and intervene manually if an unforeseen situation occurs.

[0559] An example of a prompt for a generating AI model is, "Suggest how to adjust heavy equipment operation commands when the user is under high stress." This prompt prompts the AI ​​model to generate the optimal operation strategy for the situation, improving system performance.

[0560] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0561] Step 1:

[0562] The server collects operational information in real time from sensors mounted on heavy machinery. Inputs include GPS location data, acceleration data, engine status, and other sensor data. The server preprocesses this data, performing noise reduction and correcting for outliers, to output a clean dataset. This processing prepares data suitable for analysis.

[0563] Step 2:

[0564] The server builds a predictive model using a generative AI model based on preprocessed data. The input is the clean data obtained in step 1, and based on this, the server creates a model to generate optimal operation commands for heavy machinery. The output is the optimal operation command for a specific task. This command is dynamically updated through a machine learning algorithm.

[0565] Step 3:

[0566] The terminal receives operation commands transmitted from the server and generates specific control signals. The input is the operation command obtained in step 2, and the terminal remotely controls each component of the heavy machinery based on it. The output is specific control signals regarding the direction and speed of the heavy machinery. This ensures safe and precise operation of the heavy machinery.

[0567] Step 4:

[0568] The user is monitored in real time through an emotion analysis module, and their psychological state is evaluated. Inputs are biometric data such as the user's facial expressions and voice tone, which the server uses to analyze the user's stress level and attention span. Outputs are feedback data based on the user's psychological state. This allows the system to adjust operational commands to take the user's condition into consideration.

[0569] Step 5:

[0570] The terminal sends data on the operating status of the heavy machinery and the user's psychological state as feedback to the server. The input is the current operating status of the heavy machinery and environmental data, which the terminal analyzes to generate feedback information to improve the accuracy of the equipment's operation. The output is evaluation data that helps further optimize the entire system. This feedback enables dynamic adjustments to the system and reduces the burden on the operator.

[0571] (Application Example 2)

[0572] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0573] In unmanned operation of heavy machinery, it is necessary to consider the psychological state of the human operator in order to improve safety and efficiency. However, current systems have difficulty adjusting operations in real time to reflect the user's emotions and stress levels, which can increase the user's psychological burden. In addition, there is a lack of feedback mechanisms to achieve optimal operation, making it a challenge to improve the precision of heavy machinery operation.

[0574] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0575] In this invention, the server includes a computing device means for collecting and preprocessing operational information, a computing device means for constructing a mathematical learning model based on the preprocessed information and generating commands for operating heavy machinery, and an emotion analysis means for evaluating the user's psychological state and adjusting the operation commands accordingly. This makes it possible to optimize heavy machinery operation appropriately, taking the user's emotions into consideration.

[0576] "Operational information" refers to various data related to the operation of heavy machinery, including location information, environmental conditions, and operation history.

[0577] A "computational device" is a device that processes collected operational information and constructs the necessary mathematical learning model.

[0578] A "mathematical learning model" is a model constructed based on collected data and is used to generate optimal commands for operating heavy machinery.

[0579] "Device means" refers to a device that controls heavy machinery based on operating commands received from a computing device, enabling remote operation.

[0580] "User means" refers to a user interface that monitors the operation of heavy machinery and makes adjustments as needed.

[0581] "Psychological state" refers to the user's internal state, including emotions, stress levels, and attention span, and is evaluated in real time using emotion analysis tools.

[0582] An "emotional analysis tool" is an analytical mechanism that evaluates the user's psychological state and adjusts the operation commands based on that evaluation.

[0583] A "feedback mechanism" is a system that compares the operation history of heavy machinery with the command content and updates the mathematical learning model as needed.

[0584] The system for realizing this invention mainly consists of a server including a computing device, device means, user means, and emotion analysis means. The server collects operation information from various sensors mounted on the heavy machinery and preprocesses its relationships. Next, it constructs a mathematical learning model based on the preprocessed information and generates optimal heavy machinery operation commands. The computing device uses Python or TensorFlow to process data and build models.

[0585] The device remotely controls heavy machinery based on generated operating commands. In this system, a user interface using React Native provides the user with information on the operating status of the heavy machinery and the content of the commands.

[0586] The user's control system allows for real-time monitoring of the heavy machinery's operation and manual adjustments as needed. Furthermore, an emotion analysis system evaluates the user's psychological state and appropriately adjusts the driving commands if the user experiences significant stress or decreased attention.

[0587] As an example, consider crane operation during the construction of a high-rise building at a large-scale construction site. In this scenario, the emotion analysis system analyzes the user's facial expressions and tone of voice in real time. If the user begins to feel stressed, the server automatically adjusts the crane's operating speed based on this information. This reduces the psychological burden on the user while ensuring safety at the site.

[0588] An example of a prompt for a generated AI model might be, "Explain how to adjust the crane's operation if the operator is under high stress." This sentence provides guidance for the model to make optimal operational adjustments that respond to the user's psychological state.

[0589] This configuration allows for the optimization and unmanned operation of heavy machinery, taking into account the user's psychological state, thereby improving safety and efficiency.

[0590] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0591] Step 1:

[0592] The server collects operational information in real time from various sensors mounted on the heavy machinery. The input is data acquired from the sensors, including location information, speed, and operation history. The server preprocesses this data, removing noise and correcting outliers to generate clean and reliable data. The output is a cleaned dataset.

[0593] Step 2:

[0594] The server builds a mathematical learning model using a cleaned dataset. Here, machine learning libraries such as TensorFlow are utilized to learn the optimal operating commands for heavy machinery from the dataset. The input is pre-processed data, and the output is the optimal operating commands for the heavy machinery. This process allows, for example, the detection of specific operating patterns and the generation of efficient operating procedures based on them.

[0595] Step 3:

[0596] The calculated operating commands are transmitted to the equipment. The terminal remotely controls the heavy machinery based on these operating commands. The input is the operating commands, and the output is the actual operation of the heavy machinery. Specifically, the position of the heavy machinery's arm and crane are adjusted to perform optimal operation.

[0597] Step 4:

[0598] The user's psychological state is evaluated through an emotion analysis system. Emotion-related data, such as the user's facial expressions and voice tone, is used as input, and the system measures stress and attention in real time based on this data. The output is an evaluation result regarding the user's psychological state.

[0599] Step 5:

[0600] The server adjusts the operating commands as needed based on the evaluation results obtained by the emotion analysis system. The input is the user's psychological evaluation result, and the output is the adjusted operating command. As a result, if the user is experiencing stress, the commands are adjusted so that the heavy machinery operates more safely.

[0601] Step 6:

[0602] Based on the overall system operation history and user feedback, the server dynamically updates its mathematical learning model. The input is the operation history and user feedback, which are used to learn new heavy equipment operation patterns. The output is the updated mathematical learning model. This feedback mechanism allows the system to improve accuracy over time, enabling safer and more efficient operation.

[0603] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0604] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0605] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[0606] [Fourth Embodiment]

[0607] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0608] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0609] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0610] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[0611] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0612] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0613] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0614] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0615] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0616] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0617] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0618] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0619] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0620] This invention relates to a remote control system for realizing unmanned operation of heavy machinery, and a specific embodiment is shown below. This system is assembled from three main components: a server, a terminal, and a user.

[0621] The server collects operational data from various sensors attached to the heavy machinery. This data includes the position, speed, orientation, and status of the arm and shovel. The server preprocesses this raw data and extracts features to be input into a machine learning model. This process builds a machine learning model that mimics the operation of the heavy machinery. Furthermore, the server uses this model to generate optimal operational commands tailored to the site conditions.

[0622] The terminal receives commands from the server and remotely controls the heavy machinery. Based on the commands, it transmits specific control signals to the heavy machinery, precisely controlling operations such as arm movement, rotation, and forward movement. The terminal also transmits the operating status of the heavy machinery and surrounding environmental data to the server in real time, providing feedback.

[0623] The user is responsible for monitoring the operation of heavy machinery from the control room. Based on monitoring data transmitted from the terminal, the user confirms that the operation of the heavy machinery is proceeding as planned. In the event of an unexpected situation, the user can also correct the situation by performing manual operations. After the operation, the user evaluates the results and provides feedback to the server to promote optimal automation of operations in various situations.

[0624] As a concrete example, consider debris removal work at a construction site. The server recognizes the shape and location of the debris piles using data from sensors and calculates the optimal removal method. The terminal controls heavy machinery based on the calculation results, performing operations to remove the debris safely and efficiently. The user monitors the entire operation and makes adjustments as needed.

[0625] In this way, this system can highly automate the unmanned operation of heavy machinery, thereby addressing the labor shortage problem and improving the working environment.

[0626] The following describes the processing flow.

[0627] Step 1:

[0628] The server collects operational data in real time from sensors mounted on the heavy machinery. This data includes location information, speed, and video and audio data showing the surrounding environment.

[0629] Step 2:

[0630] The server preprocesses the collected data, performing noise reduction and data normalization, and converting it into a format usable by machine learning models.

[0631] Step 3:

[0632] The server updates the machine learning model using the pre-processed data. This model learns an optimization algorithm for heavy equipment operation and generates the necessary operation commands.

[0633] Step 4:

[0634] The server sends the generated operation commands to the terminal. The commands include specific instructions for movement, such as changing the angle of the arm or the direction of movement of the vehicle body.

[0635] Step 5:

[0636] The terminal remotely controls heavy machinery based on commands received from the server. It generates appropriate control signals to execute the operations of the heavy machinery.

[0637] Step 6:

[0638] The terminal feeds back the operating status of the heavy machinery to the server. Based on the actual operating data, it provides the information necessary to generate the next command.

[0639] Step 7:

[0640] The user monitors the operating status in real time. They verify that the system is working stably and take manual control if an abnormality occurs.

[0641] Step 8:

[0642] After completing a task, the user evaluates the operation history and results of the heavy machinery. This allows them to provide feedback to the server for improving future operations.

[0643] (Example 1)

[0644] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0645] To safely and efficiently operate large machinery such as heavy equipment remotely and unmanned, real-time situational awareness and the generation of appropriate operating commands are required. Furthermore, a feedback mechanism capable of responding to unforeseen circumstances is necessary. However, conventional systems struggle to meet these requirements, resulting in challenges such as decreased labor productivity and safety risks.

[0646] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0647] In this invention, the server includes an information processing device means that collects operational information and performs preprocessing including noise reduction and normalization; an information processing device means that extracts characteristics based on the preprocessed information, constructs a learning model using a generation AI algorithm, and generates machine operation commands; and a control device means that remotely operates the machine based on the commands received from the information processing device and generates specific control signals. This enables efficient and safe operation of heavy machinery in real time from a remote location.

[0648] "Operational information" refers to data that indicates the state and behavior of the machine, including position information, speed, and arm movement.

[0649] "Information processing device means" refers to a device used to receive operation information and perform noise reduction, normalization, and characteristic extraction.

[0650] A "generative AI algorithm" refers to an algorithm that uses data to perform pattern recognition and prediction, and generates commands that are appropriate for new situations.

[0651] A "learning model" refers to a mathematical model that is built using past data and used to optimize future operations.

[0652] "Control device means" refers to a device for remotely operating heavy machinery and other machines based on received commands.

[0653] "Supervision device means" refers to a device that monitors the operating status of a machine and allows for manual operation in response to unforeseen circumstances.

[0654] A "feedback mechanism" refers to a system that compares actual actions with command content and updates the learning model based on the results.

[0655] This invention relates to an advanced automation system for realizing unmanned operation of heavy machinery. This system mainly consists of a server, terminals, and users, and each element works together to achieve efficient machine operation.

[0656] The server collects data from sensors attached to heavy machinery. This data includes the machinery's position, speed, and arm movement. The server receives this data in real time and performs preprocessing such as noise reduction and data normalization. Furthermore, it uses machine learning frameworks such as TensorFlow and PyTorch to extract characteristics from the data and build a learning model. This model uses generative AI algorithms to generate optimal operating commands for the heavy machinery.

[0657] The terminal receives operation commands transmitted from the server and generates specific control signals to control the heavy machinery based on those commands. This allows for precise remote operation of various aspects of the heavy machinery, such as arm movement, body rotation, and direction of travel adjustment. The terminal continuously feeds back the heavy machinery's operation data to the server, providing information to improve the accuracy of the operation.

[0658] In the control room, the user monitors the operation data of the heavy machinery provided from the terminal. The user can verify that the heavy machinery is operating as planned and can intervene manually if an anomaly is detected. This plays a crucial role in ensuring operational safety. Furthermore, after completing a task, the user reviews the results and provides feedback to the server, contributing to the optimization of future tasks.

[0659] As a concrete example, in the case of debris removal work at a construction site, the server analyzes the shape and location of the debris based on data from sensors and calculates the optimal removal method. The terminal operates heavy machinery according to these instructions to efficiently remove the debris. The user monitors this work and makes adjustments as needed. An example of a prompt message is, "Please tell me how to design an AI model that generates optimal heavy machinery operation instructions based on the shape and location of the pile of debris."

[0660] This invention aims to efficiently and safely achieve unmanned operation of heavy machinery through the coordination of these elements, thereby addressing labor shortages and improving working conditions.

[0661] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0662] Step 1:

[0663] The server collects operational information in real time from various sensors attached to heavy machinery. Inputs include location information from GPS sensors, velocity information from accelerometers, and arm movement status information from motion sensors. The server aggregates this input data, removes noise and outliers, and outputs it as normalized data. This process ensures data quality.

[0664] Step 2:

[0665] The server extracts features using preprocessed data. The input is the denoised data from step 1, from which characteristics related to the operation of heavy machinery are extracted. For example, this includes the angle of the arm and the open / closed state of the shovel at a specific time. The output is a dataset containing these features, which is used as input data for the generative AI model. This process also performs dimensionality reduction of the data as needed.

[0666] Step 3:

[0667] The server builds a machine learning model based on the extracted features. The feature data obtained in step 2 is given as input, and a machine learning framework such as TensorFlow is used to generate the trained model. The output is a trained model for predicting and optimizing the operation of heavy machinery. The model is used to generate operation commands according to site conditions.

[0668] Step 4:

[0669] The server generates optimal operating commands using the constructed learning model. The model generated in step 3 receives real-time field conditions as input data and calculates how the heavy machinery should operate. The output is a set of specific operating commands, which are sent to the terminal. These commands include instructions for arm movement and the direction of the machine's movement.

[0670] Step 5:

[0671] The terminal receives operation commands transmitted from the server and remotely controls the heavy machinery based on them. The input is the operation command generated in step 4, and the terminal generates specific control signals based on this. The output is the control signal to the heavy machinery, precisely controlling the operation of the arm and the movement of the machine. Furthermore, the terminal monitors the situation and feeds back the operation data to the server.

[0672] Step 6:

[0673] The user monitors the operating status of heavy machinery based on feedback data from the terminal. The input is real-time operation data provided by the terminal, which the user refers to to verify that the heavy machinery is operating as planned. If a problem occurs, the user intervenes manually and adjusts the operation. The output consists of user evaluations and correction instructions, ensuring the safety and efficiency of the system.

[0674] (Application Example 1)

[0675] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0676] Conventional remote control systems for heavy machinery lack methods for visualizing operating conditions, immediately detecting anomalies, and enabling rapid response by site supervisors. Therefore, there is a need to improve safety and work efficiency on site. Furthermore, remote control systems face the challenge of not being able to flexibly adapt to diverse environments.

[0677] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0678] In this invention, the server includes an information processing means for collecting and pre-processing operational information, an information processing means for constructing a mathematical model based on the pre-processed information and generating instructions for operating heavy machinery, and a communication device means for remotely operating heavy machinery based on instructions received from the information processing device. This makes it possible to visualize the operation of heavy machinery, respond immediately when an anomaly is detected, and allow site supervisors to quickly correct instructions. Furthermore, it enables safe and efficient work in a variety of site environments.

[0679] "Operational information" refers to all data related to the operation of heavy machinery, including information such as position, speed, orientation, and operating status obtained from sensors and other devices.

[0680] "Information processing means" refers to a device or software equipped with data processing functions for pre-processing collected operational information and using it to construct mathematical models or generate operation instructions.

[0681] A "mathematical model" refers to a model constructed using mathematical methods based on operational information in order to mimic and optimize the operation of heavy machinery.

[0682] "Communication device means" refers to a device or system that has a communication function for transmitting operation instructions received from an information processing device to heavy machinery and for remotely controlling the heavy machinery.

[0683] "Human-operated means" refers to means that provide an interface for human operators to monitor the operation of heavy machinery, enabling adjustments and modifications of instructions as needed.

[0684] "Mobile devices" refer to portable electronic devices such as smartphones and tablets that site supervisors use to visualize the operation status of heavy machinery and to correct manual instructions.

[0685] "Display means" refers to a function or device for visually displaying the operating status of heavy machinery or information from sensors on a mobile device.

[0686] "Alarming device" refers to a device or program that has an alert function to detect abnormalities or dangers in real time and immediately notify the user.

[0687] The system for implementing this invention combines information processing means, communication device means, human user means, mobile terminal and display means to realize unmanned remote operation of heavy machinery.

[0688] The server collects operational information from various sensors installed on heavy machinery and preprocesses this data. The preprocessed data is used to build a mathematical model, and after the model is built, optimal instructions for operating the heavy machinery are generated. This process utilizes generative AI models as data processing and computing technology. Furthermore, the server uses information management software to analyze operational information in real time and generate optimal instructions.

[0689] The terminal receives instructions generated by the server via a communication line and controls various operations of the heavy machinery. The communication device converts these instructions into specific control signals, enabling remote control of the heavy machinery.

[0690] Users can use their mobile devices to check operational status data transmitted from the server in real time. The devices are equipped with display devices that visually show the operating status of heavy machinery and environmental information. In addition, they allow for human intervention, such as fine-tuning of operations and manual intervention in case of abnormalities.

[0691] Anomaly detection is a mechanism in which the server quickly detects anomalies and notifies the user through an alarm system on a mobile device. For example, if a mechanical anomaly or safety risk occurs during operation, an alert is immediately issued, allowing the user to respond quickly.

[0692] As a concrete example, when removing rubble, the server acquires shape data in real time and suggests the most efficient work route to the user. By using this as a reference, the user can improve the safety and efficiency of the work.

[0693] An example of a prompt message for a generated AI model is, "Based on the current terrain data, please propose an efficient method for debris removal." This allows the server to provide sophisticated, data-driven instructions.

[0694] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0695] Step 1:

[0696] The server collects operational information such as position, speed, direction, and surrounding environment information in real time from various sensors installed on heavy machinery. The input is raw data from the sensors, and the output is pre-processed data. The data is shaped through noise filtering and normalization, and processed into a form suitable for mathematical models.

[0697] Step 2:

[0698] The server builds a generative AI model using pre-processed data. Based on this model, it generates optimal operating instructions for heavy machinery. The input is pre-processed data, and the output is specific operating instructions. Feature extraction techniques and machine learning algorithms are utilized to propose optimal actions tailored to on-site conditions.

[0699] Step 3:

[0700] Operation instructions generated from the server are transmitted to the terminal via the communication line. The input is the operation instructions generated by the server, and the output is the control command to the terminal. The instructions are encoded in a format that the terminal can understand, according to the communication protocol.

[0701] Step 4:

[0702] The terminal remotely controls heavy machinery via communication devices based on received instructions. The input is the received control command, and the output is the motion of the heavy machinery. The control signals are converted, and specific actions are performed to control the various actuators of the heavy machinery.

[0703] Step 5:

[0704] Users monitor the operation of heavy machinery and the surrounding conditions in real time via a mobile device display. Inputs are operational information from the heavy machinery and environmental data, while output is information displayed on the user interface. The visualized information allows users to quickly grasp the current situation.

[0705] Step 6:

[0706] The server uses an anomaly detection algorithm to monitor for potential anomalies during operation and immediately sends an alarm to the mobile device upon detection. Inputs are operational data and environmental data, and output is an alarm notification. Based on anomaly analysis, it draws attention and requests prompt action from the user.

[0707] Step 7:

[0708] When an anomaly is reported from a mobile device, the user takes immediate action. The input for the anomaly is an alarm notification, and the output is a corrected operation instruction. Depending on the situation, the user can issue manual instructions and coordinate with the server to replan operations.

[0709] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0710] This invention relates to a remote control system that combines an emotion engine to achieve unmanned operation of heavy machinery, and a specific embodiment is shown below. This system consists of four main components: a server, a terminal, a user, and an emotion engine.

[0711] The server collects operational data from sensors mounted on heavy machinery, preprocesses this data, and builds a machine learning model. This model generates commands to optimize the operation of the heavy machinery. The server also includes an algorithm to acquire the user's emotional state and reflect it in the operational commands.

[0712] The terminal remotely operates heavy machinery based on commands received from the server. It generates the control signals necessary for the operation of the heavy machinery and performs the specific operations. Furthermore, the terminal feeds back the operating status of the heavy machinery and the user's emotional state to the server, contributing to improved processing accuracy.

[0713] The user acts as a monitor in the control room, while their emotional state is evaluated in real time by an emotion engine. By taking into account the user's stress and attention levels, it supports safe and effective operation. In the event of an anomaly, manual intervention can be performed immediately, but the emotion engine ensures that the intervention is even more appropriate.

[0714] The emotion engine analyzes the user's emotions in real time based on their facial expressions, tone of voice, and word choice. If the obtained emotion data affects the overall system operation, the server immediately suggests the optimal solution and adjusts the operation commands as needed. In this way, the operation of the heavy machinery and the user's psychological burden are optimized simultaneously.

[0715] A concrete example is debris removal work using heavy machinery at a construction site. The server calculates the optimal operation of the heavy machinery, taking into account data from sensors and the user's stress level. The terminal controls the heavy machinery based on this calculation, achieving both safety and efficiency simultaneously. The user can monitor the entire operation with the support of the emotion engine and intervene manually in emergencies.

[0716] This configuration allows the system to significantly advance the unmanned operation of heavy machinery while also ensuring the safety and psychological well-being of users.

[0717] The following describes the processing flow.

[0718] Step 1:

[0719] The server collects operational data in real time from sensors mounted on the heavy machinery. This includes the machinery's position, speed, tilt, and information about the surrounding environment. Simultaneously, it also collects user emotion data from the emotion engine.

[0720] Step 2:

[0721] The server preprocesses the collected operational and sentiment data. It removes noise from the data and converts it into a format suitable for input to the machine learning model to be used.

[0722] Step 3:

[0723] The server updates the machine learning model based on pre-processed data. This model is used to generate optimal operating commands for heavy machinery, taking into account the environment and emotional state.

[0724] Step 4:

[0725] The server sends the generated operation commands to the terminal. The commands include specific control information such as the movement and path of the heavy machinery's arm.

[0726] Step 5:

[0727] The terminal remotely controls heavy machinery based on operation commands received from the server. It sends necessary control signals to ensure the heavy machinery operates according to the program.

[0728] Step 6:

[0729] The terminal reports the operating status of the heavy machinery to the server. At the same time, the emotion engine also feeds back user emotion data that it has analyzed to the server.

[0730] Step 7:

[0731] Users can monitor the operating status of heavy machinery in real time through the monitoring device. They can determine if the system is functioning correctly and perform manual operations if necessary.

[0732] Step 8:

[0733] The server continuously evaluates the user's emotional state, and if any stressors or decreased concentration detected affect operations, it generates new instructions for adjustment.

[0734] Step 9:

[0735] Based on user feedback and operational data, the server readjusts its machine learning models for subsequent tasks, improving overall operational efficiency.

[0736] (Example 2)

[0737] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0738] Conventional heavy equipment operation systems struggle to achieve both efficiency and safety in unmanned operation, and furthermore, the emotional stress of the operator can affect system performance. This has resulted in limitations in optimizing operations and preventing accidents.

[0739] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0740] In this invention, the server includes a computing device means for collecting and preprocessing operational information, a computing device means for constructing a predictive model based on the preprocessed information and generating machine operation commands, and a communication device means for remotely controlling the machine based on the commands received from the computing device. This enhances efficiency and safety in the unmanned operation of heavy machinery, reduces emotional stress on the operator, and enables optimal operation.

[0741] "Operational information" refers to data related to the operation of a machine, collected from sensors and other devices.

[0742] "Preprocessing" refers to removing noise and outliers from collected data and formatting it to make it easier to analyze.

[0743] "Computing device means" refers to a device that processes data and performs calculations for analysis and prediction.

[0744] A "predictive model" is a mathematical model used to generate instructions for optimizing machine operation based on collected data.

[0745] "Communication device means" refers to a device for sending and receiving information between a server and a machine or terminal in a remote location.

[0746] "Monitoring tools" are means for monitoring the state of a system and intervening or adjusting as needed.

[0747] "User's psychological state" refers to the mental state of the person operating the system, including their emotions, stress levels, and attention span.

[0748] "Emotional analysis methods" are techniques for analyzing a user's psychological state based on facial expressions, tone of voice, and other factors.

[0749] "Feedback" refers to providing information based on the system's operation and user status to facilitate further improvements and adjustments.

[0750] This invention is a remote control system that enables unmanned operation of heavy machinery while reducing the psychological stress on operators. The system mainly consists of a server, terminals, users, and an emotion analysis module.

[0751] The server plays a central role in this system, collecting operational information in real time from various sensors mounted on the heavy machinery. This data includes GPS location information, engine status, and speed sensor data. The server preprocesses this data, removing noise and correcting for outliers. It also generates a predictive model based on the preprocessed data. This predictive model is built using machine learning techniques on a specific platform, utilizing software such as Python and TensorFlow.

[0752] The terminal receives commands transmitted from the server and precisely controls the operation of heavy machinery. For example, in excavation work at a construction site, the terminal accurately moves the arm of the heavy machinery to a specified coordinate, ensuring safe and efficient work. This operation is performed by control signals transmitted via a communication network.

[0753] The user has the role of monitoring the entire system through their terminal and intervening as needed. The user's psychological state is analyzed by an emotion analysis module. This module uses cameras and microphones to evaluate the user's facial expressions, tone of voice, posture, etc., and quantifies stress levels and attention. As a result, the system can generate operational commands that take the operator's emotional state into account.

[0754] As a concrete example, in debris removal work, the server calculates the optimal operation command considering location information obtained from sensors and the user's stress level. The terminal then controls heavy machinery based on this, improving safety and efficiency. The user can monitor the system's operation in real time and intervene manually if an unforeseen situation occurs.

[0755] An example of a prompt for a generating AI model is, "Suggest how to adjust heavy equipment operation commands when the user is under high stress." This prompt prompts the AI ​​model to generate the optimal operation strategy for the situation, improving system performance.

[0756] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0757] Step 1:

[0758] The server collects operational information in real time from sensors mounted on heavy machinery. Inputs include GPS location data, acceleration data, engine status, and other sensor data. The server preprocesses this data, performing noise reduction and correcting for outliers, to output a clean dataset. This processing prepares data suitable for analysis.

[0759] Step 2:

[0760] The server builds a predictive model using a generative AI model based on preprocessed data. The input is the clean data obtained in step 1, and based on this, the server creates a model to generate optimal operation commands for heavy machinery. The output is the optimal operation command for a specific task. This command is dynamically updated through a machine learning algorithm.

[0761] Step 3:

[0762] The terminal receives operation commands transmitted from the server and generates specific control signals. The input is the operation command obtained in step 2, and the terminal remotely controls each component of the heavy machinery based on it. The output is specific control signals regarding the direction and speed of the heavy machinery. This ensures safe and precise operation of the heavy machinery.

[0763] Step 4:

[0764] The user is monitored in real time through an emotion analysis module, and their psychological state is evaluated. Inputs are biometric data such as the user's facial expressions and voice tone, which the server uses to analyze the user's stress level and attention span. Outputs are feedback data based on the user's psychological state. This allows the system to adjust operational commands to take the user's condition into consideration.

[0765] Step 5:

[0766] The terminal sends data on the operating status of the heavy machinery and the user's psychological state as feedback to the server. The input is the current operating status of the heavy machinery and environmental data, which the terminal analyzes to generate feedback information to improve the accuracy of the equipment's operation. The output is evaluation data that helps further optimize the entire system. This feedback enables dynamic adjustments to the system and reduces the burden on the operator.

[0767] (Application Example 2)

[0768] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0769] In unmanned operation of heavy machinery, it is necessary to consider the psychological state of the human operator in order to improve safety and efficiency. However, current systems have difficulty adjusting operations in real time to reflect the user's emotions and stress levels, which can increase the user's psychological burden. In addition, there is a lack of feedback mechanisms to achieve optimal operation, making it a challenge to improve the precision of heavy machinery operation.

[0770] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0771] In this invention, the server includes a computing device means for collecting and preprocessing operational information, a computing device means for constructing a mathematical learning model based on the preprocessed information and generating commands for operating heavy machinery, and an emotion analysis means for evaluating the user's psychological state and adjusting the operation commands accordingly. This makes it possible to optimize heavy machinery operation appropriately, taking the user's emotions into consideration.

[0772] "Operational information" refers to various data related to the operation of heavy machinery, including location information, environmental conditions, and operation history.

[0773] A "computational device" is a device that processes collected operational information and constructs the necessary mathematical learning model.

[0774] A "mathematical learning model" is a model constructed based on collected data and is used to generate optimal commands for operating heavy machinery.

[0775] "Device means" refers to a device that controls heavy machinery based on operating commands received from a computing device, enabling remote operation.

[0776] "User means" refers to a user interface that monitors the operation of heavy machinery and makes adjustments as needed.

[0777] "Psychological state" refers to the user's internal state, including emotions, stress levels, and attention span, and is evaluated in real time using emotion analysis tools.

[0778] An "emotional analysis tool" is an analytical mechanism that evaluates the user's psychological state and adjusts the operation commands based on that evaluation.

[0779] A "feedback mechanism" is a system that compares the operation history of heavy machinery with the command content and updates the mathematical learning model as needed.

[0780] The system for realizing this invention mainly consists of a server including a computing device, device means, user means, and emotion analysis means. The server collects operation information from various sensors mounted on the heavy machinery and preprocesses its relationships. Next, it constructs a mathematical learning model based on the preprocessed information and generates optimal heavy machinery operation commands. The computing device uses Python or TensorFlow to process data and build models.

[0781] The device remotely controls heavy machinery based on generated operating commands. In this system, a user interface using React Native provides the user with information on the operating status of the heavy machinery and the content of the commands.

[0782] The user's control system allows for real-time monitoring of the heavy machinery's operation and manual adjustments as needed. Furthermore, an emotion analysis system evaluates the user's psychological state and appropriately adjusts the driving commands if the user experiences significant stress or decreased attention.

[0783] As an example, consider crane operation during the construction of a high-rise building at a large-scale construction site. In this scenario, the emotion analysis system analyzes the user's facial expressions and tone of voice in real time. If the user begins to feel stressed, the server automatically adjusts the crane's operating speed based on this information. This reduces the psychological burden on the user while ensuring safety at the site.

[0784] An example of a prompt for a generated AI model might be, "Explain how to adjust the crane's operation if the operator is under high stress." This sentence provides guidance for the model to make optimal operational adjustments that respond to the user's psychological state.

[0785] This configuration allows for the optimization and unmanned operation of heavy machinery, taking into account the user's psychological state, thereby improving safety and efficiency.

[0786] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0787] Step 1:

[0788] The server collects operational information in real time from various sensors mounted on the heavy machinery. The input is data acquired from the sensors, including location information, speed, and operation history. The server preprocesses this data, removing noise and correcting outliers to generate clean and reliable data. The output is a cleaned dataset.

[0789] Step 2:

[0790] The server builds a mathematical learning model using a cleaned dataset. Here, machine learning libraries such as TensorFlow are utilized to learn the optimal operating commands for heavy machinery from the dataset. The input is pre-processed data, and the output is the optimal operating commands for the heavy machinery. This process allows, for example, the detection of specific operating patterns and the generation of efficient operating procedures based on them.

[0791] Step 3:

[0792] The calculated operating commands are transmitted to the equipment. The terminal remotely controls the heavy machinery based on these operating commands. The input is the operating commands, and the output is the actual operation of the heavy machinery. Specifically, the position of the heavy machinery's arm and crane are adjusted to perform optimal operation.

[0793] Step 4:

[0794] The user's psychological state is evaluated through an emotion analysis system. Emotion-related data, such as the user's facial expressions and voice tone, is used as input, and the system measures stress and attention in real time based on this data. The output is an evaluation result regarding the user's psychological state.

[0795] Step 5:

[0796] The server adjusts the operating commands as needed based on the evaluation results obtained by the emotion analysis system. The input is the user's psychological evaluation result, and the output is the adjusted operating command. As a result, if the user is experiencing stress, the commands are adjusted so that the heavy machinery operates more safely.

[0797] Step 6:

[0798] Based on the overall system operation history and user feedback, the server dynamically updates its mathematical learning model. The input is the operation history and user feedback, which are used to learn new heavy equipment operation patterns. The output is the updated mathematical learning model. This feedback mechanism allows the system to improve accuracy over time, enabling safer and more efficient operation.

[0799] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0800] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0801] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[0802] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0803] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[0804] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[0805] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[0806] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[0807] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[0808] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[0809] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[0810] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[0811] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0812] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0813] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0814] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[0815] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0816] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0817] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0818] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0819] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[0820] The following is further disclosed regarding the embodiments described above.

[0821] (Claim 1)

[0822] A server means for collecting and pre-processing operational data,

[0823] A server means that builds a machine learning model based on preprocessed data and generates commands for operating heavy machinery,

[0824] A terminal means for remotely operating heavy machinery based on commands received from a server,

[0825] A user means to monitor the operation of heavy machinery and enable adjustments according to the situation,

[0826] A system that includes this.

[0827] (Claim 2)

[0828] The system according to claim 1, which identifies the location and environment of heavy machinery based on information acquired from various sensors and generates optimal operation commands.

[0829] (Claim 3)

[0830] The system according to claim 1, which has a feedback mechanism that compares the operating history of heavy machinery with the command content and dynamically updates the machine learning model.

[0831] "Example 1"

[0832] (Claim 1)

[0833] Information processing device means for collecting operational information and performing preprocessing including noise reduction and normalization,

[0834] An information processing device means that extracts characteristics based on pre-processed information, constructs a learning model using a generative AI algorithm, and generates machine operation commands.

[0835] A control device means that remotely operates the machine and generates specific control signals based on commands received from the above-mentioned information processing device,

[0836] A supervisory device means that monitors the operating status of the machine and enables manual adjustment when an abnormality is detected,

[0837] A system that includes this.

[0838] (Claim 2)

[0839] The system according to claim 1, which identifies the machine's position and working environment based on information obtained from various detectors and generates optimal operation commands.

[0840] (Claim 3)

[0841] The system according to claim 1, comprising a feedback mechanism that compares the machine's operational history with the command content and dynamically optimizes and updates the generated learning model.

[0842] "Application Example 1"

[0843] (Claim 1)

[0844] Information processing means for collecting operational information and performing preprocessing,

[0845] Information processing means that constructs a mathematical model based on pre-processed information and generates instructions for operating heavy machinery,

[0846] A communication device means for remotely operating heavy machinery based on instructions received from an information processing device,

[0847] A means of human intervention that monitors the operation of heavy machinery and enables adjustments according to the situation,

[0848] A display means that visualizes the operating status of heavy machinery on a mobile device and enables correction of manual instructions,

[0849] An alarm system that immediately notifies when an anomaly is detected,

[0850] A system that includes this.

[0851] (Claim 2)

[0852] The system according to claim 1, which identifies the location and environment of heavy machinery based on information acquired from various sensors, generates optimal operation instructions, and further displays the operation status in real time on a mobile terminal.

[0853] (Claim 3)

[0854] The system according to claim 1, which has a feedback mechanism that compares the operating history of heavy machinery with the instructions given and dynamically updates a mathematical model, and monitors using an anomaly detection algorithm.

[0855] "Example 2 of combining an emotion engine"

[0856] (Claim 1)

[0857] A computing device means for collecting operation information and performing preprocessing,

[0858] A computing device means that constructs a predictive model based on preprocessed information and generates machine operation commands,

[0859] A communication device means for remotely controlling a machine based on commands received from a computing device,

[0860] A monitoring device that allows for monitoring the operating status of the machine and making adjustments according to the conditions,

[0861] An emotion analysis method that analyzes the user's psychological state and reflects it in the operation commands,

[0862] Information processing means that provides feedback on the machine's operating status and the user's condition to improve accuracy,

[0863] A system that includes this.

[0864] (Claim 2)

[0865] The system according to claim 1, which identifies the position and surrounding conditions of a machine based on information obtained from various detection devices and generates an optimal operation command.

[0866] (Claim 3)

[0867] The system according to claim 1, comprising an information processing mechanism that compares the machine's operational history with the command content and dynamically updates a predictive model.

[0868] "Application example 2 when combining with an emotional engine"

[0869] (Claim 1)

[0870] A computing device means for collecting operation information and performing preprocessing,

[0871] A computing device means that constructs a mathematical learning model based on preprocessed information and generates commands for operating heavy machinery,

[0872] A device for remotely controlling heavy machinery based on commands received from a computing device,

[0873] A user means to monitor the operation of heavy machinery and enable adjustments according to the situation,

[0874] An emotion analysis means that evaluates the user's psychological state and adjusts the operation commands based on that evaluation,

[0875] A system that includes this.

[0876] (Claim 2)

[0877] The system according to claim 1, which identifies the location and environment of heavy machinery based on information acquired from various detectors and generates an optimal operation command.

[0878] (Claim 3)

[0879] The system according to claim 1, comprising a feedback mechanism that compares the operation history of heavy machinery with the command content and dynamically updates a mathematical learning model. [Explanation of symbols]

[0880] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. A server means for collecting and pre-processing operational data, A server means that builds a machine learning model based on preprocessed data and generates commands for operating heavy machinery, A terminal means for remotely operating heavy machinery based on commands received from a server, A user means to monitor the operation of heavy machinery and enable adjustments according to the situation, A system that includes this.

2. The system according to claim 1, which identifies the location and environment of heavy machinery based on information acquired from various sensors and generates an optimal operation command.

3. The system according to claim 1, which has a feedback mechanism that compares the operating history of heavy machinery with the command content and dynamically updates the machine learning model.

Images