Automatic shovel loading control method, training method, device, equipment and medium

Abstract

The invention provides an automatic shovel loading control method, a training method, a training device, equipment and a medium, an intelligent agent is trained by utilizing collected data formed by state data of a carry-scraper, a control instruction and a reward value so as to obtain the intelligent agent for realizing automatic shovel loading, and particularly, the intelligent agent can be used for realizing automatic shovel loading. The state data is selected from the expansion and contraction amount of an oil cylinder, a steering hinge angle, the coordinate position of the carry-scraper and the acceleration and angular velocity of the front vehicle body edge, and the control instruction is selected from the expansion and contraction speed of the oil cylinder, the angular velocity of a steering hinge and the angular velocity of wheels. And the to-be-shoveled ore heap which dynamically changes in the shoveling process is effectively shoveled with high quality and high efficiency.

Description

Technical Field

[0001] The present application relates to the field of mining technology, and in particular to an automatic shoveling control method, training method, device, equipment and medium. Background Art

[0002] Scrapers are a type of multifunctional engineering machinery that integrates shoveling, transporting and unloading functions. They play an important role in ore mining and tunneling operations in underground mines. As mining activities continue to extend deep underground, the operating conditions of scrapers have become more severe, facing challenges such as high temperature, dust and noise. These unfavorable factors not only threaten the performance and life of the equipment, but also seriously endanger the physical and mental health of the operators. In view of this, the automatic control technology of scraper operation has become a hot research area. In this field, the automatic loading control technology of scrapers is particularly critical.

[0003] Among the existing automatic loading control methods for shovel loaders, the force feedback-based method relies on on-site parameter adjustment and is difficult to adapt to the dynamic changes of the ore pile to be shoveled. Summary of the invention

[0004] The present application proposes an automatic shoveling control method, training method, device, equipment and medium, which can solve one of the problems existing in the background technology.

[0005] In order to achieve the above objectives, this application adopts the following technical solutions: In a first aspect, a training method for an automatic shovel loading agent is provided, wherein the agent acts on a scraper, and the method comprises: Initializing the agent; and Using the collected data, the intelligent agent is trained, The collected data includes: the state data of the scraper, the control instructions of the intelligent agent to the scraper, and the reward value, wherein: The state data includes: cylinder extension amount, steering articulation angle, scraper coordinate position, front vehicle body edge acceleration and angular velocity, The control instructions include: oil cylinder extension speed, steering hinge angular velocity and wheel angular velocity.

[0006] Based on the above technical scheme, the intelligent agent is trained using the collected data consisting of the state data, control instructions and reward values ​​of the scraper to obtain an intelligent agent that realizes automatic shoveling. In particular, the state data includes the cylinder extension amount, steering articulation angle, scraper coordinate position, acceleration and angular velocity of the front vehicle body edge, and the control instructions include the cylinder extension speed, steering hinge angular velocity and wheel angular velocity. It is found in simulation experiments and practical applications that the control instructions output by the trained intelligent agent can control the scraper to effectively load the dynamically changing ore pile to be shoveled during the loading process with high quality and high efficiency.

[0007] In a possible design of the first aspect, the reward function r in the agent is defined as: ,in, is the coordinate of the bucket in the direction of the loader moving toward the ore pile; a is the row vector composed of the current action control quantity corresponding to the control instruction; F is the amount of ore in the bucket; is the extension of the bucket cylinder; k and f are coefficients; f1 is the amount of ore when the bucket is full, and f1 is a decimal less than f2 and greater than 0.

[0008] In a possible design of the first aspect, the intelligent agent includes: a strategy network and a value network, wherein the input of the strategy network is the current state, and the output is the probability of performing an action under the current state; the input of the value network is the current state, and the output is the value of the current state.

[0009] In a possible design manner of the first aspect, the objective function of the agent includes: a strategy loss and a value loss defined by a reward function, and an entropy regularization term.

[0010] In a possible design manner of the first aspect, the objective function is optimized by mini-batch gradient descent to update parameters of the policy network and the value network.

[0011] In a second aspect, an automatic shoveling control method is provided, wherein the control method is based on the intelligent agent trained as described in any one of the first aspects, and the intelligent agent acts on the shovel loader to complete automatic shoveling control.

[0012] In a third aspect, a training device for an automatic shovel loading agent is provided, wherein the agent acts on a shovel loading machine, and the training device comprises: an initialization unit, used to initialize the agent; and A training unit, used to train the intelligent agent using the collected data, The collected data includes: the state data of the scraper, the control instructions of the intelligent agent to the scraper, and the reward value, wherein: The state data includes: cylinder extension amount, steering articulation angle, scraper coordinate position, acceleration and angular velocity of the front vehicle body, The control instructions include: oil cylinder extension speed, steering hinge angular velocity and wheel angular velocity.

[0013] In a fourth aspect, a shovel loader is provided, wherein the shovel loader is provided with the intelligent agent trained as in any one of the first aspects, and the intelligent agent acts on the shovel loader to complete automatic shoveling control.

[0014] In a fifth aspect, an electronic device is provided, comprising: a processor, and a memory coupled to the processor, the memory being used to store a computer program; the processor being used to execute the computer program stored in the memory, so that the electronic device executes the training method as any possible implementation in the first aspect, or executes the automatic shoveling control method as described in the second aspect.

[0015] In a sixth aspect, a computer-readable storage medium is provided, comprising a computer program or instructions, which, when executed on a computer, enables the computer to execute the training method as any possible implementation in the first aspect, or to execute the automatic shoveling control method as described in the second aspect.

[0016] In the seventh aspect, a computer program product is provided, comprising: a computer program or instructions, which, when executed on a computer, enables the computer to execute the training method as any possible implementation in the first aspect, or to execute the control method as described in the second aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0018] Figure 1 is an implementation flow chart of the automatic shoveling method for a shovel loader based on reinforcement learning provided in an embodiment of the present application; Figure 2 is a structural schematic diagram of a scraper provided in an embodiment of the present application; Figure 3 is a comparison chart of the amount of ore shoveled provided in the embodiment of the present application; Figure 4 It is a comparison chart of the shoveling time provided in the embodiments of the present application. DETAILED DESCRIPTION

[0019] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0020] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification, claims and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0021] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0022] like Figure 1 As shown, the embodiment of the present application proposes an automatic shoveling method for a shovel loader based on reinforcement learning. The method is based on Figure 2 The system structure shown is carried out.

[0023] First, in the controller module, follow the steps below to train and obtain a reinforcement learning agent.

[0024] Step 1: Initialize two neural networks, one for representing strategy and the other for representing value. The input of the strategy network is the current state s, and the output is the probability of executing action a under the current state s, denoted as ; The input of the value network is the current state s, and the output is the value of the current state s, denoted as . Where θ and are the parameters of the two neural networks respectively.

[0025] Among them, the state s is composed of the observation data obtained by the sensor and is defined as: , where W b , W c are the extension and retraction of the arm cylinder and the bucket cylinder of the scraper respectively; z is the steering articulation angle of the scraper; is the wheel speed; [d x ,d y ,d z ] is the position coordinate of the loader in the local coordinate system with the center of the ore pile to be shoveled as the origin; [a x ,a y ,a z ]、[gx ,g y ,g z ] are the acceleration and angular acceleration of the front body of the scraper along the x, y, and z coordinate axes respectively.

[0026] Action a is the control instruction output by the reinforcement learning agent and is defined as: . Where v b ,v c are the extension and retraction speeds of the boom cylinder and bucket cylinder, respectively, with extension being positive; is the angular velocity of the steering hinge of the scraper; is the angular velocity of the wheel.

[0027] Step 2: Collect data. Based on the current strategy Interact in the environment, collect and save data generated during the interaction: , where s t is the current state, a t is the current action, r t is the current reward, s t+1 For the next state, done t Indicates whether the current sequence has ended, 1 for end, 0 for not end. The criteria for determining the end of the interaction episode are: the bucket reaches the set rotation limit position or the interaction episode length reaches the limit value. Continue to interact with the environment and save data until the set maximum cache length B is reached. m .

[0028] The reward r t is defined as: , where is the coordinate of the bucket in the direction of the loader moving toward the ore pile; a is the row vector composed of the current action control quantity; F is the amount of ore in the bucket; is the extension of the bucket cylinder; k and f are coefficients. f1 is the amount of ore when the bucket is full, and f1 is a decimal less than f2 and greater than 0.

[0029] Step 3: Calculate the objective function. The objective function used in this method consists of three parts: strategy loss, value loss, and entropy regularization term: , where c1, c2 are hyperparameters used to control the weights of value loss and entropy regularization.

[0030] The strategy loss is: , where ,in, is the discount factor, is a parameter that controls the degree of advantage smoothing, represents the old policy, Indicates the new strategy, It is the clipping threshold parameter. clip() is the clipping function, and min() is the minimum value. Their mathematical definitions are as follows: , where a and b are preset values.

[0031] Value loss: ,in, .

[0032] Entropy regularization term: ,in, It’s a strategy In status t The entropy under , β is the regularization coefficient.

[0033] Step 4: Optimize the objective function. Optimize the objective function through mini-batch gradient descent and update the parameters of the strategy and value network: , where α is the learning rate.

[0034] Step 5: Iterative optimization. Repeat steps 2 to 4 until the model converges or reaches the set maximum number of training steps.

[0035] After training is completed, reload the agent in the control module. Encode and pass it into the agent's policy network. Then the action output by the policy network The information is transmitted to the actuator module, and the scraper executes the corresponding action instructions until the loading action is completed.

[0036] The above embodiment is exemplarily described below through a specific application example.

[0037] The purpose of this example is to provide an automatic shoveling method for a shovel loader based on deep reinforcement learning, which can adapt to dynamically changing ore piles; and to provide a modeling method for the reinforcement learning action space, state space, and reward function for automatic shoveling of a shovel loader, which greatly improves the convergence success rate of training and the application effect.

[0038] To achieve the above purpose, the specific implementation method is as follows: Step 1: Build an agent. Initialize two neural networks, one for representing strategy and the other for representing value. The input of the strategy network is the current state s, and the output is the probability of executing action a under the current state s, denoted as ; The input of the value network is the current state s, and the output is the value of the current state s, denoted as . Where θ and are the parameters of the two neural networks. Both neural networks have two hidden layers with 64 nodes and the activation function is tanh.

[0039] Among them, the state s is composed of the observation data obtained by the sensor and is defined as: , where W b , W c are the extension and retraction of the arm cylinder and the bucket cylinder of the scraper respectively; z is the steering articulation angle of the scraper; is the wheel speed; [d x ,d y ,d z ] is the position coordinate of the loader in the local coordinate system with the center of the ore pile to be shoveled as the origin; [a x ,a y ,a z ]、[g x ,g y ,g z ] are the acceleration and angular acceleration of the front body of the scraper along the x, y, and z coordinate axes respectively.

[0040] Action a is the control instruction output by the reinforcement learning agent and is defined as: . Where v b ,v c are the extension and retraction speeds of the boom cylinder and bucket cylinder, respectively, with extension being positive; is the angular velocity of the steering hinge of the scraper; is the angular velocity of the wheel.

[0041] Step 2: Agent training.

[0042] Step 2.1: Collect data. Based on the current strategy Interact in the environment, collect and save data generated during the interaction: , where s t is the current state, a t is the current action, r t is the current reward, s t+1 For the next state, done t Indicates whether the current sequence has ended, 1 for end, 0 for not end. The end of the interaction episode is determined when the bucket reaches the set rotation limit position or the interaction episode length reaches the limit value. Continue to interact with the environment and save data until the set maximum cache length of 2048 is reached.

[0043] The reward r t is defined as: , where is the coordinate of the bucket in the direction of the loader moving toward the ore pile; a is the row vector composed of the current action control quantity; F is the amount of ore in the bucket; is the extension of the bucket cylinder.

[0044] Step 2.2: Calculate the objective function. The objective function used in this method consists of three parts: policy loss, value loss, and entropy regularization term: .

[0045] The strategy loss is: .

[0046] In the formula, ,in, represents the old policy (the old policy given state s t When selecting action a t probability), represents a new strategy (the current strategy given state s t When selecting action a t The probability of ). clip() is the clipping function, min() is the minimum value, and their mathematical definitions are as follows: , where a and b are preset values.

[0047] Value loss: ,in, .

[0048] Entropy regularization term: ,in, It’s a strategy In status t The entropy under .

[0049] Step 2.3: Optimize the objective function. Optimize the objective function through mini-batch gradient descent and update the parameters of the policy and value networks as shown in the following formula: ; Step 2.4: Iterative optimization. Repeat steps 2.1 to 2.3 until the model reaches the set maximum number of training steps 20,000,000.

[0050] Step 3: Get the status information of the scraper and convert the status data into Encode and pass it into the agent's policy network.

[0051] Step 5: The agent outputs action instructions based on state input.

[0052] Step 6: The scraper performs corresponding actions according to the action instructions.

[0053] Step 7: Repeat steps 3 to 6 until the shovel installation is complete.

[0054] In order to show the superiority of the method proposed in this aspect, Figure 3 As shown in the literature, this method is compared with [1] To the literature [4]40 groups of shoveling experiments were conducted to compare the existing methods and human drivers in the paper. Compared with the existing methods, this method performs best in terms of shoveling success rate and loading volume, close to human drivers; and is better than human drivers in shoveling time and shoveling consistency.

[0055] literature [1] To the literature [4] List: literature [1] Eriksson D, Ghabcheloo R, Geimer M. Automatic Loading ofUnknown Material with a Wheel Loader Using Reinforcement Learning[C] / / 2024IEEE International Conference on Robotics and Automation (ICRA). 2024: 3646-3652. literature [2] Eriksson D, Ghabcheloo R, Geimer M. Optimizing a BucketFilling Strategy for Wheel Loaders Inside a Dream Environment[A]. 2024. literature [3] Shen C, Sloth C. Generalized Framework for Wheel LoaderAutomatic Shoveling Task with Expert Initialized Reinforcement Learning[C] / / 2024 IEEE / SICE International Symposium on System Integration (SII). 2024:382-389. literature [4]Dadhich S, Sandin F, Bodin U, et al. Adaptation of a wheelloader automatic bucket filling neural network using reinforcement learning[C] / / 2020 International Joint Conference on Neural Networks (ijcnn). NewYork: Ieee, 2020. The above-mentioned automatic shoveling method of the loader based on deep reinforcement learning can also be based on generating a model of the underground ore pile to be shoveled in a physical simulator for simulation training and simulation experiments. By generating a model of the ore pile to be shoveled, a batch of ore block models with different shapes and block sizes of a specified distribution can be automatically generated to form an ore pile. This model can be used for research in the fields of shoveling automatic control and bulk migration law.

[0056] The present application also provides a training device for an automatic shovel loading agent, wherein the agent acts on a shovel loader, and the training device comprises: An initialization unit, used to initialize the agent; and A training unit, used to train the intelligent agent using the collected data, The collected data includes: the state data of the scraper, the control instructions of the intelligent agent to the scraper, and the reward value, wherein: The state data includes: cylinder extension amount, steering articulation angle, scraper coordinate position, front vehicle body edge acceleration and angular velocity, The control instructions include: oil cylinder extension speed, steering hinge angular velocity and wheel angular velocity.

[0057] An embodiment of the present application also provides a scraper loader, which is provided with the intelligent agent trained as described in any one of the first aspects, and the intelligent agent acts on the scraper loader to complete automatic loading control.

[0058] An embodiment of the present application also provides an electronic device, comprising: a processor, and a memory coupled to the processor, wherein the memory is used to store a computer program; the processor is used to execute the computer program stored in the memory, so that the electronic device executes a method as described in any one of the above embodiments.

[0059] The electronic device may be a computing device such as a desktop computer, a notebook, a palmtop computer, a cloud server, etc. The electronic device may include, but is not limited to, a processor and a memory.

[0060] The processor may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc. The processor is the control center of the electronic device, and various interfaces and lines are used to connect various parts of the entire device.

[0061] The memory may be used to store the computer program, and the processor implements various functions of the electronic device by running or executing the computer program stored in the memory and calling the data stored in the memory.

[0062] The memory may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, at least one application required for a function, etc.; the data storage area may store data created according to the use of the mobile phone, etc. In addition, the memory may include a high-speed random access memory, and may also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0063] The embodiment of the present application also provides a storage medium, which is a computer-readable storage medium, and the computer program is stored in the computer-readable storage medium. When the computer program is executed by the processor, the steps of each of the above-mentioned method embodiments can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electrical carrier signal, telecommunication signal and software distribution medium, etc.

[0064] An embodiment of the present application further provides a computer program product, including: a computer program or instructions, which, when executed on a computer, enables the computer to execute any of the above-mentioned possible implementation methods.

[0065] The above is a preferred embodiment of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications are also considered to be within the scope of protection of the present application.

Claims

1. A training method for an automatic shovel loading agent, characterized in that: The intelligent agent acts on a scraper, and the method comprises: Initializing the agent; and Using the collected data, the intelligent agent is trained, The collected data includes: the state data of the scraper, the control instructions of the intelligent agent to the scraper, and the reward value, wherein: The state data includes: cylinder extension amount, steering articulation angle, scraper coordinate position, front vehicle body edge acceleration and angular velocity, The control instructions include: oil cylinder extension speed, steering hinge angular velocity and wheel angular velocity.

2. The training method according to claim 1, characterized in that: The reward function r in the agent is defined as: ,in, is the coordinate of the bucket in the direction of the loader moving toward the ore pile; a is the row vector composed of the current action control quantity corresponding to the control instruction; F is the amount of ore in the bucket; is the extension of the bucket cylinder; k and f are coefficients; f1 is the amount of ore when the bucket is full, and f1 is a decimal less than f2 and greater than 0.

3. The training method according to claim 2, characterized in that: The intelligent agent includes: a strategy network and a value network. The input of the strategy network is the current state, and the output is the probability of executing an action in the current state; the input of the value network is the current state, and the output is the value of the current state.

4. The training method according to claim 3, characterized in that: The objective function of the agent includes: strategy loss and value loss defined by a reward function, and an entropy regularization term.

5. The training method according to claim 4, characterized in that: The objective function is optimized by mini-batch gradient descent to update the parameters of the policy network and the value network.

6. An automatic shovel loading control method, characterized in that: The control method is based on the intelligent agent trained by the training method as described in any one of claims 1 to 5, and the intelligent agent acts on the shovel loader to complete automatic shoveling control.

7. A training device for an automatic shovel loading agent, characterized in that: The intelligent agent acts on a scraper, and the training device comprises: An initialization unit, used to initialize the agent; and A training unit, used to train the intelligent agent using the collected data, The collected data includes: the state data of the scraper, the control instructions of the intelligent agent to the scraper, and the reward value, wherein: The state data includes: cylinder extension amount, steering articulation angle, scraper coordinate position, front vehicle body edge acceleration and angular velocity, The control instructions include: oil cylinder extension speed, steering hinge angular velocity and wheel angular velocity.

8. A scraper, characterized in that: The scraper loader is provided with the intelligent agent trained by the training method as described in any one of claims 1 to 5, and the intelligent agent acts on the scraper loader to complete automatic loading control.

9. An electronic device, characterized in that: The electronic device comprises: a processor, and a memory coupled to the processor, The memory is used to store a computer program; and The processor is used to execute the computer program stored in the memory so that the electronic device executes the training method as described in any one of claims 1 to 5, or executes the automatic shoveling control method as described in claim 6.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a computer program or instructions, and when the computer program or instructions are executed on a computer, the computer executes the training method as described in any one of claims 1 to 5, or executes the automatic shoveling control method as described in claim 6.

Images