Unmanned loader control method and system, loader and medium

Abstract

The invention provides an unmanned loader control method and system, a loader and a medium, and relates to the technical field of engineering machinery. The method comprises the following steps: acquiring current state data of the unmanned loader, wherein the current state data comprises operation image data acquired by a camera of the unmanned loader and loader body data acquired by a sensor; determining a corresponding first state vector according to the current state data, and inputting the first state vector into a pre-trained human in-the-loop reinforcement learning model to obtain an execution action of the unmanned loader at the next moment; and outputting a corresponding control signal according to the execution action to drive the unmanned loader to execute, and performing parameter optimization on the human in-the-loop reinforcement learning model according to the execution action and the intervention action input by the human. According to the method, the automation, the working efficiency and the operation safety of loading operation are improved.

Description

Technical Field

[0001] The present application relates to the technical field of engineering machinery, and in particular to an unmanned loader control method, system, loader and medium. Background Art

[0002] Traditional loader operations are highly dependent on manual control by skilled operators. However, in dangerous, repetitive, or inaccessible environments (such as deep mines, highly polluted areas, and post-disaster sites), manual operations face efficiency bottlenecks, high safety risks, and increased costs.

[0003] With advances in artificial intelligence and robotics, unmanned (autonomous) loader control systems have become a key technology for addressing these challenges. Related technologies often employ control methods based on pre-programmed trajectories or fixed rules. These methods exhibit poor adaptability to dynamic changes in the construction environment (such as changes in the shape of the material pile, the presence of new obstacles, and changes in ground conditions). This makes it difficult to ensure robustness and efficiency, resulting in poor operational safety and requiring frequent operator intervention.

[0004] Therefore, there is a need for an unmanned loader control solution that can improve the automation, work efficiency and operational safety of loading operations. Summary of the Invention

[0005] The embodiments of the present application provide an unmanned loader control method, system, loader and medium, which can improve the automation, work efficiency and operational safety of loading operations.

[0006] In a first aspect, an embodiment of the present application provides an unmanned loader control method based on human-in-the-loop visual reinforcement learning, comprising:

[0007] Acquiring current state data of the unmanned loader, the current state data including operation image data collected by a camera of the unmanned loader and loader body data collected by a sensor;

[0008] Determining a corresponding first state vector according to the current state data, and inputting the first state vector into a pre-trained human-in-the-loop reinforcement learning model to obtain an execution action of the unmanned loader at a next moment;

[0009] A corresponding control signal is output according to the execution action to drive the unmanned loader to execute, and parameters of the human-in-the-loop reinforcement learning model are optimized according to the execution action and the intervention action input by the human.

[0010] In one possible implementation, the optimizing parameters of the human-in-the-loop reinforcement learning model according to the execution action and the intervention action input by the human includes:

[0011] Determine a second state vector corresponding to the state data of the unmanned loader at a next moment;

[0012] determining a reward value output by a reward discriminator of the human-in-the-loop reinforcement learning model based on the executed action;

[0013] generating an experience tuple according to the first state vector, the executed action, the reward value, and the second state vector, and placing the experience tuple into an online experience pool;

[0014] Parameters of the human-in-the-loop reinforcement learning model are optimized based on an online experience pool, a demonstration experience pool, and intervention actions input by humans, wherein the demonstration experience pool includes human expert demonstration data.

[0015] In one possible implementation, the performing parameter optimization on the human-in-the-loop reinforcement learning model based on the online experience pool, the demonstration experience pool, and the intervention action input by the human includes:

[0016] receiving an intervention action input by a human based on the execution action, and outputting a corresponding control signal according to the intervention action to drive the unmanned loader to execute;

[0017] Putting the intervention action into the demonstration experience pool, and putting the loader actions before and after the intervention action into the online experience pool;

[0018] The online experience pool and the demonstration experience pool are sampled in equal proportion according to a preset sampling ratio to obtain training data;

[0019] Optimizing parameters of the human-in-the-loop reinforcement learning model according to the training data, the loss function of the human-in-the-loop reinforcement learning model, and a preset optimizer, and soft-updating parameters of a target network corresponding to the human-in-the-loop reinforcement learning model;

[0020] Among them, the human-in-the-loop reinforcement learning model includes a policy network model and a value network model. The loss function corresponding to the policy network model combines the maximum entropy reinforcement learning strategy and the goal is to maximize the expected return of the strategy. The loss function corresponding to the value network model adopts the clipped double Q learning strategy and the goal is to minimize the error of the Bellman equation.

[0021] In a possible implementation, the method further includes:

[0022] According to a preset sampling ratio and sampling priority, G batches of equal-proportion sampling are performed on the online experience pool and the demonstration experience pool to obtain G batches of training data;

[0023] Performing G gradient updates on network parameters of the human-in-the-loop reinforcement learning model according to the G batches of training data;

[0024] Wherein, each data in the online experience pool and the demonstration experience pool includes a corresponding sampling priority.

[0025] In one possible implementation, the reward discriminator is a binary reward discriminator. When the execution action is successful, the reward value output by the reward discriminator is a positive value. When the execution action fails, the reward value output by the reward discriminator is 0.

[0026] In a possible implementation, outputting a corresponding control signal according to the execution action to drive the unmanned loader to execute includes:

[0027] Determining an actuator and a target controller corresponding to the execution action, wherein the target controller includes one of an impedance controller, a force controller, or a torque controller;

[0028] The actual control amount of the actuator is determined according to the execution action, and a corresponding control signal is output to the target controller according to the actual control amount of the actuator, so that the target controller drives the actuator to operate according to the actual control amount.

[0029] In a possible implementation, determining the corresponding first state vector according to the current state data includes:

[0030] Preprocessing the job image data to obtain target image data, wherein the preprocessing includes one or more of denoising, enhancing, cropping, and scaling;

[0031] Using a preset visual backbone network to extract features from the target image data to obtain corresponding visual feature vectors;

[0032] Performing coordinate transformation on the coordinate data in the loader body data to obtain relative coordinates relative to the initial posture;

[0033] Determining proprioceptive perception data according to the relative coordinates and data in the loader body data other than the coordinate data, and determining a proprioceptive perception vector corresponding to the proprioceptive perception data;

[0034] Perform vector fusion on the visual feature vector and the proprioception vector to obtain a first state vector.

[0035] In a second aspect, an embodiment of the present application provides an unmanned loader control system, comprising:

[0036] a processor, and a memory communicatively coupled to the processor;

[0037] Memory is used to store computer-executable instructions;

[0038] The processor is used to execute the computer-executable instructions stored in the memory, so that the processor executes the above-mentioned first aspect and / or various possible implementations of the first aspect.

[0039] In a third aspect, an embodiment of the present application provides an unmanned loader, comprising: one or more cameras, a plurality of sensors, and the unmanned loader control system according to claim 8, wherein the unmanned loader control system is communicatively connected to the cameras and the sensors respectively;

[0040] The camera is used to collect operating image data of the unmanned loader during operation;

[0041] The sensor is used to collect the loader body data when the unmanned loader is operating;

[0042] The unmanned loader control system is used to obtain the current state data of the unmanned loader, wherein the current state data includes the operation image data collected by the camera of the unmanned loader and the loader body data collected by the sensor; determine the corresponding first state vector based on the current state data, and input the first state vector into the pre-trained human-in-the-loop reinforcement learning model to obtain the execution action of the unmanned loader at the next moment; output the corresponding control signal according to the execution action to drive the unmanned loader to execute, and optimize the parameters of the human-in-the-loop reinforcement learning model according to the execution action and the intervention action input by humans.

[0043] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, they are used to implement the above-mentioned first aspect and / or various possible implementation methods of the first aspect.

[0044] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program, which, when executed by a processor, is used to implement the above-mentioned first aspect and / or various possible implementation methods of the first aspect.

[0045] The embodiment of the present application provides an unmanned loader control method, system, loader and medium, which can first use the camera set on the unmanned loader to obtain operation image data, use the sensor to obtain the loader body data, and obtain the first state vector that can fully characterize the working state of the loader itself and the surrounding operation environment based on the operation image data and the loader body data. The first state vector is then input into the pre-trained human-in-the-loop reinforcement learning model to obtain the execution action of the unmanned loader at the next moment. The human can input the corresponding intervention action according to the execution action, and the parameters of the human-in-the-loop reinforcement learning model can be optimized according to the execution action and the intervention action input by the human. Through such a setting, the unmanned loader can autonomously learn efficient and safe loading operation strategies in a complex and dynamic real physical environment through interaction with the environment and timely guidance from humans, thereby significantly improving the automation level, work efficiency and operational safety of the loading operation. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0047] Figure 1 This is a system architecture diagram of an embodiment of the present application;

[0048] Figure 2 This is a flow chart of an unmanned loader control method based on human-in-the-loop visual reinforcement learning according to an embodiment of the present application;

[0049] Figure 3 This is a structural diagram of an unmanned loader control system according to an embodiment of the present application;

[0050] Figure 4 This is a structural diagram of an unmanned loader control system according to another embodiment of the present application.

[0051] The above drawings illustrate specific embodiments of the present application, which will be described in more detail below. These drawings and the textual description are not intended to limit the scope of the present application in any way, but rather to illustrate the concepts of the present application to those skilled in the art by reference to specific embodiments. DETAILED DESCRIPTION

[0052] Exemplary embodiments will be described in detail herein, with examples illustrated in the accompanying drawings. In the following description, when referring to the drawings, identical numerals in different figures represent identical or similar elements, unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all embodiments consistent with the present application. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present application, as detailed in the appended claims.

[0053] The terms "first," "second," "third," "fourth," and the like (if any) in the specification and claims of this application and in the accompanying drawings are used to distinguish similar objects and are not necessarily used to describe a particular order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate so that the embodiments of the application described herein can be implemented in orders other than those illustrated or described herein. In addition, the terms "including" and "having," and any variations thereof, are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or apparatus comprising a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units that are not explicitly listed or that are inherent to such processes, methods, products, or apparatus.

[0054] In the technical solution of this application, the collection, storage, use, processing, transmission, provision and disclosure of information such as financial data or user data involved comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0055] It should be noted that in the embodiments of the present application, certain software, components, models and other existing solutions in the industry may be mentioned. They should be regarded as exemplary. Their purpose is only to illustrate the feasibility of implementing the technical solution of the present application, but it does not mean that the applicant has or will necessarily use the solution.

[0056] It should also be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0057] The unmanned loader control method, system, loader and medium of the present application can be used in the field of engineering machinery technology, and can also be used in any field other than the field of engineering machinery technology, such as the field of artificial intelligence, the field of reinforcement learning, the field of robotics, etc. The application field of the unmanned loader control method, system, loader and medium of the present application is not limited.

[0058] The unmanned loader control method, system, loader and medium of the present application can be applied to complex, dynamic and potentially dangerous working environments such as mines, construction sites, and disaster rescue. As long as the human-in-the-loop mechanism is used and reinforcement learning is combined with visual information to control the unmanned loader, the unmanned loader control method, system, loader and medium of the present application can be applied.

[0059] First, let’s explain the terms involved in this application:

[0060] Reinforcement Learning (RL), as an important branch of machine learning, allows intelligent agents to learn optimal behavior strategies through trial and error interactions with the environment. It has strong adaptive capabilities and the potential to handle complex decision-making problems, providing a new approach for the development of highly intelligent unmanned loader systems.

[0061] Human-in-the-Loop Reinforcement Learning (HIL-RL) is an important reinforcement learning paradigm that allows an intelligent agent to learn using human guidance (such as demonstration, correction, and evaluation) while interacting with its environment in real time. Compared to traditional RL that relies entirely on offline data or purely autonomous online RL, HIL-RL can:

[0062] (1) Improving sample efficiency: Human guidance can help the agent explore the effective state-action space more quickly, avoid redundant or dangerous exploration, and effectively utilize valuable real-world interaction data.

[0063] (2) Enhanced safety: In the early stages of learning or when facing unknown situations, humans can intervene to prevent the agent from performing actions that may cause equipment damage or safety accidents.

[0064] (3) Solving the sparse reward problem: In tasks where reward signals are infrequent (e.g., completing a loading cycle), human feedback (e.g., demonstrating successful paths, providing shaping rewards) can provide a denser learning signal for the agent.

[0065] (4) Utilizing prior knowledge: Ability to incorporate the experience and intuition of human operators into the learning process to accelerate the convergence of strategies.

[0066] Traditional loader operations are highly dependent on manual control by skilled operators. However, in dangerous, repetitive, or inaccessible environments (such as deep mines, highly polluted areas, and post-disaster sites), manual operations face efficiency bottlenecks, high safety risks, and increased costs.

[0067] With advances in artificial intelligence and robotics, unmanned (autonomous) loader control systems have become a key technology for addressing these challenges, aiming to improve the automation, efficiency, and safety of operations. Related technologies often use control methods based on pre-programmed trajectories or fixed rules. These methods have poor adaptability to dynamic changes in the construction environment (such as changes in the shape of the material pile, new obstacles, and changing ground conditions), making it difficult to ensure robustness and efficiency. They also suffer from poor operational safety and require frequent operator intervention.

[0068] Based on the above technical problems, the inventive concept of this application is: how to provide an unmanned loader control solution that can improve the automation, work efficiency and operational safety of loading operations.

[0069] The embodiments of the present application provide an unmanned loader control method, system, loader and medium, which can use the camera set on the unmanned loader to obtain operation image data, use the sensor to obtain the loader body data, and obtain the first state vector that can fully characterize the working state of the loader itself and the surrounding operation environment based on the operation image data and the loader body data, and input the first state vector into the pre-trained human-in-the-loop reinforcement learning model to obtain the execution action of the unmanned loader at the next moment. Humans can input corresponding intervention actions according to the execution action, and the parameters of the human-in-the-loop reinforcement learning model can be optimized based on the execution action and the intervention action input by humans. Through such a setting, loading strategies can be autonomously learned and optimized in a real, complex, and dynamically changing construction environment without the need for tedious pre-programming or rule design for specific scenarios, and it has strong adaptability. The real-time monitoring and intervention capabilities of human experts provide an important safety barrier for the system, which can effectively prevent the loader from performing dangerous actions during the learning process.

[0070] The following specific embodiments describe in detail the technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0071] Figure 1 This is a system architecture diagram of an embodiment of the present application, such as Figure 1 As shown, the unmanned loader is equipped with multiple cameras and multiple sensors, which are respectively communicated with the unmanned loader control system; the unmanned loader control system includes a human-computer interaction device and a computing platform, which are communicated with the human-computer interaction device. A human-in-the-loop reinforcement learning model is deployed in the computing platform, and humans remotely monitor and intervene in the operation of the unmanned loader through the human-computer interaction device.

[0072] The camera can collect the operating image data of the unmanned loader during operation, the sensor can collect the loader body data during operation, and the unmanned loader control system can obtain the operating image data collected by the camera of the unmanned loader and the loader body data collected by the sensor; determine the corresponding first state vector based on the current state data, and input the first state vector into the pre-trained human-in-the-loop reinforcement learning model to obtain the execution action of the unmanned loader at the next moment; output the corresponding control signal based on the execution action to drive the unmanned loader to execute, and optimize the parameters of the human-in-the-loop reinforcement learning model based on the execution action and the intervention action input by humans.

[0073] Figure 2 This is a flow chart of an unmanned loader control method based on human-in-the-loop visual reinforcement learning according to an embodiment of the present application. This embodiment uses the unmanned loader control system as the execution subject to illustrate the unmanned loader control method based on human-in-the-loop visual reinforcement learning. Figure 2 As shown, the unmanned loader control method based on human-in-the-loop visual reinforcement learning may include the following steps:

[0074] S201: Acquire current status data of the unmanned loader.

[0075] In this embodiment, the unmanned loader control system may include a computing platform and a human-computer interaction device that is communicatively connected to the computing platform. The computing platform may be a high-performance embedded computer for running perception algorithms and human-in-the-loop reinforcement learning models. The human-computer interaction device may be a display, remote control / handle / SpaceMouse, etc.

[0076] In this embodiment, the current state data may include operation image data collected by a camera of the unmanned loader and loader body data collected by sensors.

[0077] In this embodiment, one or more cameras on the loader (e.g., four cameras on the front, back, left, and right sides of the vehicle) can be used to capture real-time images of the working environment and use them as working image data. The cameras can be communicatively connected to the unmanned loader control system to transmit the working image data to the unmanned loader control system.

[0078] In this embodiment, the loader's built-in sensors (such as IMU, GPS, encoders, hydraulic sensors, etc.) can be used to obtain key loader data. The sensors can be connected to the unmanned loader control system to transmit the loader data to the unmanned loader control system.

[0079] In this embodiment, the loader body data may include: the position and posture of the loader (for example, relative to the world coordinate system or the starting position), the status of each major component, such as the boom angle, bucket angle, articulation angle, etc., the operating parameters of the hydraulic system, such as cylinder pressure, flow, etc. (reflecting the load conditions), and the speed, angular velocity and other motion information of the end effector (such as the bucket).

[0080] S202: Determine a corresponding first state vector according to current state data.

[0081] In this embodiment, a preset visual backbone network can be used to extract features from the operation image data to obtain the corresponding visual feature vector, and determine the proprioception vector corresponding to the loader body data. The visual feature vector and the proprioception vector are then fused or spliced to obtain a first state vector. The first state vector is a state representation that can be understood and utilized by the human-in-the-loop reinforcement learning model.

[0082] In this embodiment, the first state vector that integrates the working environment data and the loader data can provide the human-in-the-loop reinforcement learning model with comprehensive information about the environment layout, material location, its own posture and load, so that the human-in-the-loop reinforcement learning model can output corresponding execution actions accordingly.

[0083] S203: Input the first state vector into a pre-trained human-in-the-loop reinforcement learning model to obtain the execution action of the unmanned loader at the next moment.

[0084] In this embodiment, the human-in-the-loop reinforcement learning model can include a policy network (actor network) and a value network (critic network). The actor network's input is the state vector s, and its output is the probability distribution of the loader's control action a (for continuous action spaces, the mean and standard deviation of a Gaussian distribution are typically output). Action a can be a multidimensional vector containing control instructions such as boom lift speed, bucket flip speed, travel speed, and steering angle.

[0085] The input of the critic network is the state vector s and the action a, and the output is the value estimate of the state-action pair (s, a), that is, the expected cumulative reward that can be obtained by following the current strategy after executing action a starting from the state vector s.

[0086] The goal of the Actor network is to learn a strategy so that the action selected in a given state can maximize the expected cumulative reward. The parameters of the Actor network are .

[0087] The goal of the Critic network is to accurately evaluate the value of the strategy generated by the Actor network. The parameters are .

[0088] In this embodiment, after the first state vector is input into the pre-trained human-in-the-loop reinforcement learning model, the Actor network in the human-in-the-loop reinforcement learning model will output the execution action of the unmanned loader at the next moment based on the input first state vector. The first state vector and the execution action will be input into the Critic network as feedback at the current moment to optimize the parameters of the human-in-the-loop reinforcement learning model.

[0089] S204: Outputting a corresponding control signal according to the execution action to drive the unmanned loader to execute.

[0090] In this embodiment, the unmanned loader control system can map the execution actions output by the human-in-the-loop reinforcement learning model to the actual control quantity of the corresponding actuator of the loader, and output the corresponding control signal to the actuator to drive the unmanned loader to execute.

[0091] S205: Optimize parameters of the human-in-the-loop reinforcement learning model based on the execution action and the intervention action input by the human.

[0092] In this embodiment, the unmanned loader control system converts the execution action a into a control signal to drive the loader's execution. The human operator observes the loader's real-time status and the agent's decisions through a monitoring interface. If the agent behaves poorly (e.g., an ineffective action, getting stuck, or approaching a dangerous area), the operator can take over control of the loader using a control device (e.g., a remote controller or SpaceMouse) and input the correct intervention action a_human to ensure operational safety. After the intervention is complete, the human returns control to the agent. The unmanned loader control system optimizes the parameters of the human-in-the-loop reinforcement learning model based on the execution action and the human-input intervention action. Repeating steps S201 to S205, the agent continuously interacts with the environment, gathers experience, and updates its strategy under human supervision and guidance until the strategy performance reaches a preset target (e.g., success rate, operational efficiency) or the maximum training time is reached.

[0093] In this embodiment, if the operator does not input an intervention action after the loader performs the corresponding action, the parameters of the human-in-the-loop reinforcement learning model can be optimized only based on the execution action output at the current moment. If the operator inputs an intervention action after the loader performs the corresponding action, the parameters of the human-in-the-loop reinforcement learning model can be optimized based on the execution action and the intervention action input by the human.

[0094] In this example, during the initial stages of the loader's operation, human intervention may be required. However, as the agent's learning time increases, human intervention becomes less frequent, except under unusual operating conditions. The agent's learned strategies typically match or even exceed the average performance of human operators, enabling faster cycle times. The real-time monitoring and intervention capabilities of human experts provide a crucial safety barrier for the system, effectively preventing the agent from executing dangerous actions during the learning process. This effectively incorporates the valuable experience and intuition of human operators, overcoming the difficulties of learning from scratch, particularly when handling tasks requiring delicate manipulation or lengthy decision sequences.

[0095] In this embodiment, the camera installed on the unmanned loader can first be used to obtain operation image data, and the loader body data can be obtained using sensors. Then, based on the operation image data and the loader body data, a first state vector that can fully characterize the working state of the loader itself and the surrounding operating environment can be obtained. The first state vector is then input into the pre-trained human-in-the-loop reinforcement learning model to obtain the execution action of the unmanned loader at the next moment. The human can input the corresponding intervention action based on the execution action, and the parameters of the human-in-the-loop reinforcement learning model can be optimized based on the execution action and the intervention action input by the human. Through such a setting, the unmanned loader can autonomously learn efficient and safe loading operation strategies in a complex and dynamic real physical environment through interaction with the environment and timely guidance from humans, significantly improving the automation level, work efficiency and operational safety of the loading operation.

[0096] In a possible implementation, the above step S202 of determining the corresponding first state vector according to the current state data may include:

[0097] S11: Preprocessing the working image data to obtain target image data, where the preprocessing includes one or more of denoising, enhancement, cropping, and scaling.

[0098] S12: Use the preset visual backbone network to extract features of the target image data to obtain the corresponding visual feature vector.

[0099] S13: performing coordinate transformation on the coordinate data in the loader body data to obtain relative coordinates relative to the initial posture.

[0100] S14: determining the proprioceptive perception data according to the relative coordinates and the data other than the coordinate data in the loader proprioceptive data, and determining the proprioceptive perception vector corresponding to the proprioceptive perception data.

[0101] S15: Perform vector fusion on the visual feature vector and the proprioception vector to obtain a first state vector.

[0102] In this embodiment, before performing feature extraction on the image data, in order to improve the accuracy of feature extraction, the job image data can first be denoised and / or enhanced, and the region of interest can be cropped and resized (for example, 128×128) to accommodate subsequent network input.

[0103] In this implementation, a deep convolutional neural network (e.g., ResNet) pre-trained on a large image dataset (e.g., ImageNet) can be used as the visual backbone network. This pre-trained network effectively extracts deep semantic features from images and compresses high-dimensional raw pixel information into low-dimensional feature vectors, improving learning stability and efficiency.

[0104] In this embodiment, in order to improve the generalization ability of the strategy to the change of the working position, some ontological information (such as the end effector pose) can be expressed as relative coordinates relative to the initial pose or reference frame. That is, the pose of the end effector at the current time t in the world coordinate system {s} is expressed as Converted to the end effector pose relative to the starting time t=0 of the i-th round The relative posture of This approach makes the control law learned by the strategy less susceptible to changes in the global position.

[0105] In this embodiment, the extracted visual feature vector and the processed proprioception information vector can be concatenated or combined through other fusion mechanisms (such as attention mechanism) to form the final state vector s as the input of the subsequent model.

[0106] In this implementation, image data is preprocessed to enable the learned policy to handle sensor noise, execution errors, and environmental uncertainties. A pretrained visual model is then used to process high-dimensional image inputs. Combined with ontological information, this enables end-to-end visual servo control policy learning, enabling the loader to adjust its movements in real time based on visual feedback. Feature extraction from image data enables visual perception to adapt to changes in environmental geometry, while relative coordinate transformation improves the policy's generalization to changes in work position. The extracted visual feature vectors are concatenated or fused with the processed ontological perception information vectors, resulting in a fused first state vector that provides the agent with comprehensive information about the environment layout, material location, its own posture, and its load.

[0107] In one possible implementation, step S205 above performs parameter optimization on the human-in-the-loop reinforcement learning model based on the execution action and the intervention action input by the human, which may include:

[0108] S21: Determine a second state vector corresponding to the state data of the unmanned loader at the next moment.

[0109] S22: Determine the reward value output by the reward discriminator of the human-in-the-loop reinforcement learning model based on the executed action.

[0110] S23: Generate an experience tuple according to the first state vector, the executed action, the reward value, and the second state vector, and put the experience tuple into the online experience pool.

[0111] S24: Optimize parameters of the human-in-the-loop reinforcement learning model based on the online experience pool, the demonstration experience pool, and the intervention actions input by humans. The demonstration experience pool includes human expert demonstration data.

[0112] In this embodiment, the method for determining the second state vector may refer to the method for determining the first state vector shown in the above steps S11 to S15, and will not be described in detail here.

[0113] In this embodiment, the human-in-the-loop reinforcement learning model may also include a reward discriminator. To effectively handle sparse rewards, the reward discriminator can be a binary classifier D(s) based on the state vector s (typically the visual state). It outputs a reward value corresponding to the action output by the actor network. When the action is successfully executed, the reward value is positive, and when the action fails, the reward value is zero.

[0114] In this embodiment, the online experience pool (RL Buffer) can store data generated by the agent's autonomous exploration and interaction with the environment, that is, the experience tuple (s, a, r, s') generated by each interaction.

[0115] The actor network can output an action a based on the current state vector s. The control execution module of the unmanned loader control system converts the action a into a control signal to drive the loader. The unmanned loader control system then obtains the next state vector s' from the environment feedback and the reward r calculated by the reward discriminator D(s'), thereby obtaining the experience tuple (s, a, r, s') generated by this interaction.

[0116] In this embodiment, the demonstration experience pool (Demo Buffer) can store offline collected human expert demonstration data, that is, a human expert operates a loader to complete several (for example, 20-30) successful loading tasks, and records the corresponding (s, a_human, r, s') sequences.

[0117] In this implementation, online learning is the primary approach, enabling direct training and deployment on the target device. This simplifies the migration from simulation to reality and lowers the barrier to entry for application. Leveraging the experience tuples in the online experience pool, the system is able to autonomously learn and optimize loading strategies in real, complex, and dynamically changing construction environments, eliminating the need for tedious pre-programming or rule design for specific scenarios and demonstrating strong adaptability. By combining a demonstration experience pool comprised of human expert demonstration data and intervention actions input by humans, human demonstration and online correction can be combined, significantly reducing the number of real-world interactions required to achieve high-performance strategies and shortening training time. It also effectively incorporates the valuable experience and intuition of human operators, overcoming the difficulties of learning from scratch, particularly when tackling tasks requiring delicate manipulation or long sequences of decision-making.

[0118] In one possible implementation, step S24 above performs parameter optimization on the human-in-the-loop reinforcement learning model based on the online experience pool, the demonstration experience pool, and the intervention action input by the human, and may include:

[0119] S31: receiving an intervention action input by a human based on the execution action, and outputting a corresponding control signal according to the intervention action to drive the unmanned loader to execute.

[0120] S32: placing the intervention action into the demonstration experience pool, and placing the loader actions before and after the intervention action into the online experience pool.

[0121] S33: sampling the online experience pool and the demonstration experience pool in equal proportion according to a preset sampling ratio to obtain training data.

[0122] S34: Optimize the parameters of the human-in-the-loop reinforcement learning model based on the training data, the loss function of the human-in-the-loop reinforcement learning model, and a preset optimizer, and soft-update the parameters of the target network corresponding to the human-in-the-loop reinforcement learning model.

[0123] Among them, the human-in-the-loop reinforcement learning model includes a policy network model and a value network model. The loss function corresponding to the policy network model combines the maximum entropy reinforcement learning strategy and the goal is to maximize the expected return of the strategy. The loss function corresponding to the value network model adopts the clipped double Q learning strategy and the goal is to minimize the error of the Bellman equation.

[0124] In this implementation, the unmanned loader control system converts the execution action a into a control signal to drive the loader's execution. A human operator can observe the loader's real-time status and the agent's decisions through a monitoring interface. If the agent behaves poorly (e.g., ineffective actions, getting stuck, or approaching a dangerous area), the operator can take over control of the loader using a control device (e.g., a remote controller or SpaceMouse) and input the correct intervention action a_human to ensure operational safety. After the intervention is complete, the human operator returns control to the agent.

[0125] In this implementation, the Demo Buffer can store offline collected human expert demonstration data and online human intervention data. The Online RL Buffer can store data generated by the agent's autonomous exploration and interaction with the environment, as well as strategy transition data before and after human intervention.

[0126] In this implementation, the interaction data generated during human intervention (s, a_human, r, s') is treated as high-quality expert experience and stored in a dedicated demonstration experience pool, as well as the online experience pool. The agent's own policy transitions (s, a_policy, r, s') before and after the intervention are stored only in the online experience pool. This distinction helps the algorithm effectively learn the human's corrective policy while maintaining its ability to utilize its own policy exploration data.

[0127] In this embodiment, each time the network parameters are updated, data can be sampled equally from the two experience pools to form a training batch, thereby balancing the use of expert knowledge and independent exploration experience. Those skilled in the art can flexibly set the preset sampling ratio based on actual needs and are not limited here.

[0128] In this embodiment, the human-in-the-loop reinforcement learning model may include a policy network (Actor network ) and the value network (Critic network ).

[0129] The input to the actor network is the state vector s, and the output is the probability distribution of the loader's control action a (for continuous action spaces, the output is usually the mean and standard deviation of a Gaussian distribution). Action a can be a multidimensional vector containing control instructions such as boom lift speed, bucket flip speed, travel speed, and steering angle.

[0130] The goal of the Actor network is to learn a strategy so that the action selected in a given state can maximize the expected cumulative reward. The parameters of the Actor network are .

[0131] The input of the critic network is the state vector s and the action a, and the output is the value estimate of the state-action pair (s, a), that is, the expected cumulative reward that can be obtained by following the current strategy after executing action a starting from the state vector s.

[0132] The goal of the Critic network is to accurately evaluate the value of the strategy generated by the Actor network. The parameters are .

[0133] Human-in-the-loop reinforcement learning models can also include target networks. In order to stabilize the learning process, corresponding target networks are usually maintained for the Actor and Critic networks ( ). Parameters of the target network , Instead of updating directly through gradient descent, it is obtained by soft updating (Polyak Averaging) of the main network parameters: , , where τ is a small update rate (e.g. 0.005).

[0134] The loss function of the Critic network ( ): The goal is to minimize the error (TDerror) of the Bellman equation. Clipped Double Q-Learning is used to alleviate the problem of overestimation of value. Two independent critic networks ( , ) and its corresponding target network ( , ).

[0135] (Target Q value, combined with the maximum entropy term)

[0136] in, (next action is sampled from current policy)

[0137]

[0138] Actor loss function ( ): The goal is to maximize the expected return of the strategy and combine it with the idea of Maximum Entropy Reinforcement Learning (RL) to encourage strategy exploration.

[0139]

[0140] in, is the entropy regularization coefficient, which is used to balance reward maximization and policy entropy maximization (exploration). Can be fixed or adaptive.

[0141] In this embodiment, the discriminator loss function (L_discriminator) is trained using the standard binary cross entropy loss function.

[0142]

[0143] in, is the true success label (0 or 1) of the state vector s, and the training data From manual annotation.

[0144] In this embodiment, the optimizer usually uses a gradient descent optimizer such as Adam to update the and Parameters of the network.

[0145] In this implementation, layer normalization can also be performed on certain layers of the Actor network and Critic network to stabilize the training process, accelerate convergence, and reduce the problem of internal covariate shift. The formula is: .

[0146] In this implementation, the real-time monitoring and intervention capabilities of human experts provide a critical safety barrier for the system, effectively preventing the agent from executing dangerous actions during the learning process. Maximum entropy RL is used to encourage policy exploration, improving the policy's robustness to state perturbations and environmental changes. Combined with human demonstration and online correction, along with advanced off-policy RL algorithms and experience replay mechanisms, the learned policy often matches or even exceeds the average performance of human operators, achieving faster cycle times. Furthermore, a comprehensive approach employs target networks, soft updates, clipped double-Q learning, maximum entropy RL, and experience replay (demonstration + online, proportional sampling) to ensure the stability and efficiency of the learning process and the robustness of the final policy. Furthermore, data batches are proportionally sampled from the demonstration experience pool and the online experience pool. An optimizer is used to update the parameters of the actor and critic networks based on their loss functions. Simultaneously, soft updates are performed on the target network parameters, improving model performance and robustness.

[0147] In one possible implementation, the method may further include:

[0148] S41: Perform G batches of equal-proportion sampling on the online experience pool and the demonstration experience pool according to the preset sampling ratio and sampling priority, to obtain G batches of training data.

[0149] S42: Perform G network parameter gradient updates on the human-in-the-loop reinforcement learning model based on G batches of training data.

[0150] Wherein, each data in the online experience pool and the demonstration experience pool includes a corresponding sampling priority.

[0151] In this embodiment, the sampling priority corresponding to each data can be flexibly set by those skilled in the art according to actual conditions. For example, the sampling priority can be set based on the size of the TD error. to set it up.

[0152] In this embodiment, the specific number of G can be flexibly set by those skilled in the art according to actual conditions and is not limited here.

[0153] In this embodiment, each data in the two experience pools may have a corresponding sampling priority. The higher the value of the data, the higher the priority. When sampling in equal proportion, the data with a higher sampling priority may be collected first.

[0154] In this embodiment, multiple batches of training data can be collected, and each batch of training data performs a gradient update of the network parameters, thereby using multiple batches of training data to perform multiple gradient updates of the network parameters.

[0155] In this implementation, multiple gradient updates of network parameters can be performed after each environment interaction step. Specifically, G batches of data are sampled from the experience pool and G optimization steps are performed. This helps to more fully utilize the collected data and accelerate learning. Furthermore, samples in the experience pool can be prioritized, prioritizing those that are more valuable for learning, further improving learning efficiency.

[0156] In one possible implementation, the reward discriminator may be a binary reward discriminator. When the action is successfully executed, the reward value output by the reward discriminator is a positive value. When the action fails, the reward value output by the reward discriminator is 0.

[0157] In this embodiment, to effectively handle the sparse reward problem, the reward discriminator can be a binary classifier D(s) based on the state vector s (usually a visual state). This classifier is trained through supervised learning, and the training data comes from human demonstrations or annotations. The input is the state (such as an image), and the output is the probability of whether the task is successful (or a 0 / 1 label). For example, 200 successful state images and 1000 failed state images are collected for training. During the reinforcement learning process, the classifier is used to automatically determine whether the current state s has achieved the task goal. The main reward function r: r = D(s). That is, the agent only receives a positive reward (e.g., +1) if the discriminator believes that the task is successful, otherwise the reward is 0.

[0158] In this implementation, within the HIL framework, the human expert can also provide additional, more intensive shaping rewards F(s, a) as needed to guide the agent to learn specific sub-skills or behavioral preferences. The total reward becomes r' = r + F(s, a).

[0159] In this implementation, a learning-based binary reward discriminator is used to address the sparse reward problem in the loading task, which simplifies the reward function design and can effectively guide policy learning.

[0160] In one possible implementation, the above step S204 outputs a corresponding control signal according to the execution action to drive the unmanned loader to execute, which may include:

[0161] S51: Determine an actuator and a target controller corresponding to the execution action, where the target controller includes an impedance controller, a force controller, or a torque controller.

[0162] S52: Determine the actual control amount of the actuator according to the execution action, and output a corresponding control signal to the target controller according to the actual control amount of the actuator, so that the target controller drives the actuator to operate according to the actual control amount.

[0163] In this implementation, after the human-in-the-loop reinforcement learning model outputs the unmanned loader's next action, the unmanned loader control system can convert the abstract action vector a into a specific control signal that the underlying controller can understand and execute. Specifically, the action vector a output by the actor network (e.g., normalized velocity or angular velocity instructions) is mapped to the actual control variable (e.g., valve opening, motor speed) of the loader's corresponding actuator (e.g., hydraulic valve, motor).

[0164] In this embodiment, after determining the execution action, it is necessary to determine a stable and secure underlying controller that matches the loader hardware to execute the converted instruction. Those skilled in the art can pre-set the corresponding execution mechanism and target controller according to different actions, for example:

[0165] (1) For common contact interaction scenarios in loading tasks (such as shoveling and pushing), an impedance controller can be used. Impedance control allows operation in a force-position hybrid control mode, so that the loader exhibits a certain degree of compliance when in contact with the environment, improving safety and facilitating precision operations. The force generated by the controller can be constrained by limiting the range of the error e (|e|≤Δ) to prevent excessive contact force. The basic form of the impedance controller can be: .

[0166] (2) For actions that require fast, dynamic motion (e.g., rapid movement), other types of controllers may be required, such as direct feedforward wrench control, in which the RL policy directly outputs the desired applied force or torque.

[0167] In this embodiment, the target controller includes but is not limited to: an impedance controller, a force controller, a torque controller, etc., and may also be a controller in other loaders, without any limitation here.

[0168] It should be noted that the loader also needs to have necessary safety logic built in, such as monitoring joint limits, hydraulic pressure, and rollover stability indicators. When potential danger is detected, it can override the instructions of the RL strategy and perform risk avoidance or stopping actions to further improve the safety of operation.

[0169] In this embodiment, after determining the execution action, the unmanned loader can be driven to execute it in combination with a secure underlying controller (such as impedance control) and possible built-in safety logic, thereby enhancing safety during autonomous operation.

[0170] The following is an explanation of the unmanned loader control method based on human-in-the-loop visual reinforcement learning of the present application using a specific embodiment.

[0171] In a specific embodiment, an unmanned loader is about to operate in a dangerous, highly polluted area. The unmanned loader control system controls it based on human-in-the-loop visual reinforcement learning. The specific process is as follows:

[0172] The first step is hardware preparation: the operator integrates the necessary sensors into the unmanned loader, including one or more cameras (for visual input) and proprioception sensors (such as IMUs, GPS, joint encoders, and hydraulic sensors). The unmanned loader control system includes a computing platform (a high-performance embedded computer for running perception algorithms and reinforcement learning models), human-computer interaction devices (displays, remote controls / handles / SpaceMouse), and network communication equipment (for data transmission and remote monitoring). This network communication equipment enables communication between the unmanned loader control system and the cameras and sensors.

[0173] Step 2: Offline Preparation: (1) Collect initial demonstration data: Have a human expert operate the loader and complete several (e.g., 20-30) successful loading tasks. Record the corresponding (s, a_human, r, s') sequences and store them in the Demo Buffer. (2) Train the reward discriminator: Use the collected demonstration data (or additional labeled data) to annotate success and failure states and train the binary reward discriminator D(s). Ensure that the discriminator achieves a high accuracy (e.g., >95%) on the validation set.

[0174] (3) Model pre-training: The collected demonstration data can be used to perform preliminary pre-training on the Actor network and the Critic network (such as behavior cloning BC or offline RL), providing a better initial strategy for online learning.

[0175] The third step is data preprocessing and state construction: the unmanned loader control system collects the working image data from the camera and the loader body data from the sensor in real time; performs denoising, enhancement, cropping, scaling and other preprocessing on the working image data to obtain the target image data; uses a pre-trained visual backbone network (such as ResNet) to extract the visual feature vector of the target image data; processes the loader body data and performs coordinate transformation (such as realizing egocentric representation) to obtain the body perception vector; and performs vector fusion on the visual feature vector and the body perception vector to obtain the current state vector s.

[0176] Step 4: Online HIL reinforcement learning:

[0177] (S1) The pre-trained (or initialized) human-in-the-loop reinforcement learning model is deployed to the loader. Autonomous operation begins, and the loader interacts with the environment. The state perception module of the unmanned loader control system uses the second step above to obtain the current state vector s in real time and transmit it to the human-in-the-loop reinforcement learning model and the human monitoring interface.

[0178] (S2) The Actor Network of the human-in-the-loop reinforcement learning model outputs an execution action a based on the current state vector s. The control execution module of the unmanned loader control system converts the execution action a into a control signal to drive the loader. The unmanned loader control system uses the second step above to obtain the next state vector s' from the environment feedback and obtains the reward r calculated by the reward discriminator D(s').

[0179] (S3) The unmanned loader control system stores the experience tuple (s, a, r, s') generated by the interaction into the online experience pool (RL Buffer).

[0180] (S4) The unmanned loader control system samples data batches proportionally from the demonstration experience pool and the online experience pool, and uses the optimizer to update the network parameters of the actor network and the critic network respectively according to the loss function of the actor network and the critic network. At the same time, the network parameters of the corresponding target network are soft-updated.

[0181] (S5) The human operator monitors the loader behavior in real time. If the strategy is found to be performing poorly or there are risks, the control right conversion unit of the unmanned loader control system switches to the manual intervention mode. The operator provides the intervention action a_human, records the intervention data (s, a_human, r, s') and stores it in the corresponding experience pool. After the intervention is completed, the control right is returned to the intelligent agent.

[0182] (S6) Steps S1 to S5 are repeated. The unmanned loader control system continuously outputs corresponding control signals according to the execution action to drive the unmanned loader to execute. The intelligent agent (human-in-the-loop reinforcement learning model) also continuously interacts with the environment, collects experience, and updates strategies under human supervision and guidance until the strategy performance reaches the preset target (such as success rate, operation efficiency) or the maximum training time is reached.

[0183] Figure 3 This is a structural diagram of an unmanned loader control system according to an embodiment of the present application. Figure 3 As shown, the unmanned loader control system includes: an acquisition module 31, which is used to obtain the current state data of the unmanned loader, and the current state data includes the operation image data collected by the camera of the unmanned loader and the loader body data collected by the sensor; a processing module 32, which is used to determine the corresponding first state vector according to the current state data, and input the first state vector into the pre-trained human-in-the-loop reinforcement learning model to obtain the execution action of the unmanned loader at the next moment; output the corresponding control signal according to the execution action to drive the unmanned loader to execute, and optimize the parameters of the human-in-the-loop reinforcement learning model according to the execution action and the intervention action input by humans.

[0184] The unmanned loader control system provided in the embodiment of the present application can execute the technical solution shown in the above method embodiment. Its implementation principle and beneficial effects are similar and will not be repeated here.

[0185] Figure 4 This is a structural diagram of an unmanned loader control system according to an embodiment of the present application. Figure 4As shown, the unmanned loader control system includes: a processor 401, and a memory 402 communicatively connected to the processor 401; the memory 402 stores computer execution instructions; the processor 401 executes the computer execution instructions stored in the memory 402 to implement the steps of the unmanned loader control method based on human-in-the-loop visual reinforcement learning in the above-mentioned method embodiments.

[0186] In the aforementioned unmanned loader control system, memory 402 and processor 401 are directly or indirectly electrically connected to enable data transmission or interaction. For example, these components may be electrically connected via one or more communication buses or signal lines, such as a bus connection. Memory 402 stores computer-executable instructions for implementing the data access control method, including at least one software functional module that may be stored in memory 402 in the form of software or firmware. Processor 401 executes various functional applications and data processing by running the software programs and modules stored in memory 402.

[0187] The memory 402 may be, but is not limited to, a random access memory (RAM), a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), etc. The memory 402 is used to store programs, and the processor 401 executes the programs after receiving execution instructions. Furthermore, the software programs and modules in the memory 402 may also include an operating system, which may include various software components and / or drivers for managing system tasks (e.g., memory management, storage device control, power management, etc.), and may communicate with various hardware or software components to provide an operating environment for other software components.

[0188] Processor 401 can be an integrated circuit chip with signal processing capabilities. The processor 401 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc. It can implement or execute the various methods, steps, and logic diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor.

[0189] An embodiment of the present application further provides an unmanned loader, comprising: one or more cameras, multiple sensors, and Figure 4 The unmanned loader control system shown is communicatively connected to the camera and the sensor respectively.

[0190] The camera is used to collect operating image data of the unmanned loader during operation.

[0191] Sensors are used to collect data from the loader itself during unmanned loader operation.

[0192] An unmanned loader control system is used to obtain the current state data of the unmanned loader, which includes operation image data collected by the unmanned loader's camera and loader body data collected by sensors; determine the corresponding first state vector based on the current state data, and input the first state vector into a pre-trained human-in-the-loop reinforcement learning model to obtain the execution action of the unmanned loader at the next moment; output a corresponding control signal based on the execution action to drive the unmanned loader to execute, and optimize the parameters of the human-in-the-loop reinforcement learning model based on the execution action and the intervention action input by humans.

[0193] An embodiment of the present application further provides a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, they are used to implement the steps of each method embodiment of the present application.

[0194] An embodiment of the present application further provides a computer program product, including a computer program, which implements the steps of each method embodiment of the present application when executed by a processor.

[0195] It should be noted that for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that this application is not limited by the order of the actions described, because according to this application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in this specification are all optional embodiments, and the actions and modules involved are not necessarily required by this application.

[0196] It should be further noted that, although the various steps in the flowchart are shown in sequence as indicated by the arrows, these steps are not necessarily performed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps may be performed in other orders. Moreover, at least a portion of the steps in the flowchart may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily performed at the same time, but may be performed at different times. The execution order of these sub-steps or stages is not necessarily to be performed in sequence, but may be performed in turn or alternately with other steps or at least a portion of the sub-steps or stages of other steps.

[0197] It should be understood that the above-described device embodiments are merely illustrative, and the device of the present application may also be implemented in other ways. For example, the division of units / modules in the above-described embodiments is merely a logical functional division, and actual implementations may employ other division methods. For example, multiple units, modules, or components may be combined or integrated into another system, or some features may be omitted or not implemented.

[0198] In addition, unless otherwise specified, the functional units / modules in the various embodiments of the present application may be integrated into a single unit / module, each unit / module may exist physically separately, or two or more units / modules may be integrated together. The aforementioned integrated units / modules may be implemented in the form of hardware or software program modules.

[0199] In the above embodiments, the description of each embodiment has its own emphasis. For parts not described in detail in a particular embodiment, please refer to the relevant description of other embodiments. The technical features of the above embodiments can be combined in any way. To keep the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0200] Those skilled in the art will readily appreciate other embodiments of the present invention after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present invention that follow the general principles of the present invention and include common knowledge or customary techniques in the art not disclosed herein. The description and examples are to be considered merely as exemplary, and the true scope and spirit of the present application are indicated by the appended claims.

[0201] It should be understood that the present application is not limited to the exact structure described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. A control method for an unmanned loader based on human-in-the-loop visual reinforcement learning, characterized in that: include: Acquiring current state data of the unmanned loader, the current state data including operation image data collected by a camera of the unmanned loader and loader body data collected by a sensor; Determining a corresponding first state vector according to the current state data, and inputting the first state vector into a pre-trained human-in-the-loop reinforcement learning model to obtain an execution action of the unmanned loader at a next moment; A corresponding control signal is output according to the execution action to drive the unmanned loader to execute, and parameters of the human-in-the-loop reinforcement learning model are optimized according to the execution action and the intervention action input by the human.

2. The unmanned loader control method based on human-in-the-loop visual reinforcement learning according to claim 1 is characterized in that: Optimizing parameters of the human-in-the-loop reinforcement learning model according to the execution action and the intervention action input by the human includes: Determine a second state vector corresponding to the state data of the unmanned loader at a next moment; determining a reward value output by a reward discriminator of the human-in-the-loop reinforcement learning model based on the executed action; generating an experience tuple according to the first state vector, the executed action, the reward value, and the second state vector, and placing the experience tuple into an online experience pool; Parameters of the human-in-the-loop reinforcement learning model are optimized based on an online experience pool, a demonstration experience pool, and intervention actions input by humans, wherein the demonstration experience pool includes human expert demonstration data.

3. The unmanned loader control method based on human-in-the-loop visual reinforcement learning according to claim 2 is characterized in that: Optimizing parameters of the human-in-the-loop reinforcement learning model based on the online experience pool, the demonstration experience pool, and the intervention action input by humans includes: receiving an intervention action input by a human based on the execution action, and outputting a corresponding control signal according to the intervention action to drive the unmanned loader to execute; Putting the intervention action into the demonstration experience pool, and putting the loader actions before and after the intervention action into the online experience pool; The online experience pool and the demonstration experience pool are sampled in equal proportion according to a preset sampling ratio to obtain training data; Optimizing parameters of the human-in-the-loop reinforcement learning model according to the training data, the loss function of the human-in-the-loop reinforcement learning model, and a preset optimizer, and soft-updating parameters of a target network corresponding to the human-in-the-loop reinforcement learning model; Among them, the human-in-the-loop reinforcement learning model includes a policy network model and a value network model. The loss function corresponding to the policy network model combines the maximum entropy reinforcement learning strategy and the goal is to maximize the expected return of the strategy. The loss function corresponding to the value network model adopts the clipped double Q learning strategy and the goal is to minimize the error of the Bellman equation.

4. The unmanned loader control method based on human-in-the-loop visual reinforcement learning according to claim 3 is characterized in that: Also includes: According to a preset sampling ratio and sampling priority, G batches of equal-proportion sampling are performed on the online experience pool and the demonstration experience pool to obtain G batches of training data; Performing G gradient updates on network parameters of the human-in-the-loop reinforcement learning model according to the G batches of training data; Wherein, each data in the online experience pool and the demonstration experience pool includes a corresponding sampling priority.

5. The unmanned loader control method based on human-in-the-loop visual reinforcement learning according to claim 2 is characterized in that: The reward discriminator is a binary reward discriminator. When the execution action is successful, the reward value output by the reward discriminator is a positive value. When the execution action fails, the reward value output by the reward discriminator is 0.

6. The unmanned loader control method based on human-in-the-loop visual reinforcement learning according to any one of claims 1 to 5, characterized in that: Outputting a corresponding control signal according to the execution action to drive the unmanned loader to execute includes: Determining an actuator and a target controller corresponding to the execution action, wherein the target controller includes one of an impedance controller, a force controller, or a torque controller; The actual control amount of the actuator is determined according to the execution action, and a corresponding control signal is output to the target controller according to the actual control amount of the actuator, so that the target controller drives the actuator to operate according to the actual control amount.

7. The unmanned loader control method based on human-in-the-loop visual reinforcement learning according to any one of claims 1 to 5, characterized in that: The determining a corresponding first state vector according to the current state data includes: Preprocessing the job image data to obtain target image data, wherein the preprocessing includes one or more of denoising, enhancing, cropping, and scaling; Using a preset visual backbone network to extract features from the target image data to obtain corresponding visual feature vectors; Performing coordinate transformation on the coordinate data in the loader body data to obtain relative coordinates relative to the initial posture; Determining proprioceptive perception data according to the relative coordinates and data in the loader body data other than the coordinate data, and determining a proprioceptive perception vector corresponding to the proprioceptive perception data; Perform vector fusion on the visual feature vector and the proprioception vector to obtain a first state vector.

8. An unmanned loader control system, characterized in that: include: a processor, and a memory communicatively connected to the processor; The memory is used to store computer-executable instructions; The processor is used to execute the computer-executable instructions stored in the memory, so that the processor executes the unmanned loader control method based on human-in-the-loop visual reinforcement learning as described in any one of claims 1 to 7.

9. An unmanned loader, characterized in that: include: One or more cameras, a plurality of sensors, and the unmanned loader control system according to claim 8, wherein the unmanned loader control system is communicatively connected to the cameras and the sensors respectively; The camera is used to collect operating image data of the unmanned loader during operation; The sensor is used to collect the loader body data when the unmanned loader is operating; The unmanned loader control system is used to obtain the current state data of the unmanned loader, wherein the current state data includes the operation image data collected by the camera of the unmanned loader and the loader body data collected by the sensor; determine the corresponding first state vector based on the current state data, and input the first state vector into the pre-trained human-in-the-loop reinforcement learning model to obtain the execution action of the unmanned loader at the next moment; output the corresponding control signal according to the execution action to drive the unmanned loader to execute, and optimize the parameters of the human-in-the-loop reinforcement learning model according to the execution action and the intervention action input by humans.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the unmanned loader control method based on human-in-the-loop visual reinforcement learning according to any one of claims 1 to 7.

Images