Scraper autonomous unloading control system based on reinforcement learning
Through the independent unloading control system of the shovel machine based on reinforcement learning, the problems of low unloading control efficiency and insufficient environmental adaptability in the existing technology are solved, and the independent unloading operation in the underground mine environment is realized, and the operation efficiency and environmental adaptability are improved.
Patent Information
- Application Number
- CN202411576613.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-06
- Publication Date
- 2025-03-07
AI Technical Summary
The existing shovel unloading control method is inefficient and has insufficient environmental adaptability in underground mine environments, so it is impossible to achieve coordinated control of the drive system and the operating mechanism system.
The autonomous unloading control system of the shovel machine based on reinforcement learning is adopted to obtain the state and environmental information of the shovel machine through the perception system, and the unloading control strategy network trained by the reinforcement learning framework is used to realize trajectory planning and target action output, and the drive system and operating system are coordinated to complete the autonomous unloading operation.
It realizes the independent unloading operation of the shovel machine under the constrained environment, improves the operation efficiency and environmental adaptability, reduces the driver's staying time underground, and improves the efficiency of large-scale operations of multi-shovel machines.
Smart Images

Figure CN119571880A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of scraper control, and in particular to an autonomous unloading control system for a scraper based on reinforcement learning. Background Art
[0002] Underground shovel loaders are widely used in underground mine operations and construction material handling operations due to their high maneuverability. However, due to the harsh environment of underground mines and construction sites, the autonomous operation function of underground shovel loaders needs to be realized urgently for safety, efficiency and cost considerations. The functions of shovel loaders are similar to those of wheel loaders, including core functions such as loading, transportation and unloading. Among them, the material transportation function has initially realized autonomous transportation operations with the help of research related to autonomous driving of articulated chassis; loading operations involve complex dynamic interactions between shovel loaders and target material piles, and autonomous operations are still not possible; the autonomy of unloading functions has become the key to improving the autonomous operation performance of shovel loaders. At present, shovel loaders are in the transition stage from manual assisted operations to semi-autonomous operations in mining operations in underground mines.
[0003] Conventional wheel loaders operate in open areas, and their unloading operations usually require unloading the materials in the bucket to a transport truck or a fixed dumping point. Their operating environment can be regarded as a motion control problem without obvious external environmental constraints. This motion control usually involves a drive system and a working mechanism system, where the drive system controls the vehicle to travel to the unloading point, and the working mechanism system dumps the materials in the bucket into the truck or unloading point. Basic autonomous unloading operations can be achieved without involving the coordinated control of the two systems. However, underground shovel loaders usually work in narrow underground tunnels, and their ore dumping points change with the location of the working tunnel. The unloading task must be completed through the coordinated control of the drive system and the working mechanism system without collision. Therefore, conventional motion control in an unconstrained environment cannot be applied to underground shovel loaders to achieve autonomous unloading.
[0004] Summarizing the existing shovel loader unloading control methods, the main problems are as follows: 1. The underground mine environment is complex, and the existing shovel loader unloading control is mostly manually operated by the driver. The driver needs to stay in the underground mine for a long time, which poses a high risk factor; 2. Some underground mines use remote-controlled unmanned shovel loaders for unloading operations. Each shovel loader requires at least one person to remotely control it. To achieve large-scale operations of multiple shovel loaders, more manpower is required, and the operation efficiency is low. The remote control signal is easily interfered by the outside world due to the limitation of the transmission method; 3. The existing autonomous unloading control method cannot achieve coordinated control of the drive system and the operating mechanism system, and the unloading operation efficiency is low and the environmental adaptability is insufficient. Summary of the invention
[0005] The present invention provides an autonomous unloading control system for a scraper based on reinforcement learning, so as to solve the technical problems of low unloading operation efficiency and insufficient environmental adaptability existing in the prior art.
[0006] In order to solve the above technical problems, the present invention provides the following technical solutions:
[0007] On the one hand, the present invention provides an autonomous unloading control system for a scraper based on reinforcement learning, wherein the scraper includes a walking system and an operating system, and the control system includes a controller and a perception system; wherein:
[0008] The perception system is used to obtain the scraper state information and the environment state information;
[0009] The controller includes an upper-layer controller and a lower-layer controller. The upper-layer controller is used to perform trajectory planning according to the shovel loader state information and the environmental state information, obtain the target position of the shovel tip, and use the unloading control strategy network trained based on the reinforcement learning framework to obtain the target action information according to the shovel loader state information and the shovel tip target position; the lower-layer controller is used to output control signals of the walking system and the operating system according to the current shovel loader state information and the target action information, control the shovel loader to perform the target action, and realize the autonomous unloading operation of the shovel loader in a constrained mining environment through the coordinated movement of the drive system and the operating system.
[0010] Furthermore, the travel system includes a drive system and a brake system, the drive system receives an accelerator pedal signal, and the brake system receives a brake pedal signal to achieve acceleration, braking or constant speed travel of the scraper;
[0011] The operating system includes a bucket, a lifting cylinder, a dump bucket cylinder, a lifting cylinder control electric proportional valve and a dump bucket cylinder electric proportional valve; the operating system receives control signals of the lifting cylinder control electric proportional valve and the dump bucket cylinder electric proportional valve, and realizes the lifting and unloading of the bucket through the combined action of the lifting cylinder and the dump bucket cylinder.
[0012] Furthermore, the perception system includes a speed encoder, a dump cylinder displacement sensor, a lifting cylinder displacement sensor, a laser radar, an inertial measurement unit and a camera; wherein,
[0013] The laser radar, inertial measurement unit, speed encoder and camera cooperate with each other to realize the modeling of the working environment of the scraper, and analyze and obtain the scraper position, scraper speed, chute information, lane information and obstacle information according to the built working environment model; wherein the chute information includes the chute position, chute height and chute width, the lane information includes the lane wall position and the lane top height, and the obstacle information includes the obstacle position and obstacle height;
[0014] The lifting cylinder displacement sensor is used to measure the current lifting cylinder displacement;
[0015] The bucket cylinder displacement sensor is used to measure the current bucket cylinder displacement.
[0016] Further, the scraper status information includes scraper position and scraper speed;
[0017] The environmental status information includes chute information, lane information and obstacle information.
[0018] Furthermore, the shovel tip target position and posture includes shovel tip horizontal and vertical coordinates and a shovel tip posture angle.
[0019] Furthermore, the target action information includes a target vehicle speed of the scraper, and a target telescopic displacement of the lifting cylinder and the dump cylinder.
[0020] Furthermore, the control signals of the travel system and the working system include an accelerator pedal signal, a brake pedal signal, a control signal of a lifting cylinder control electric proportional valve, and a control signal of a dump cylinder electric proportional valve.
[0021] Furthermore, the lower-layer controller is used to output control signals of the walking system and the operating system according to the current state information and target action information of the scraper, control the scraper to perform the target action, and realize the autonomous unloading operation of the scraper in the constrained mining environment through the coordinated movement of the driving system and the operating system, including:
[0022] According to the current state information of the scraper loader and the target action information, combined with the PID control algorithm, the accelerator pedal signal, the brake pedal signal, the control signal of the lifting cylinder control electric proportional valve and the control signal of the dump cylinder electric proportional valve are output to control the drive system and the operating system, so that the scraper loader can travel at the target speed, and the lifting cylinder and the dump cylinder can be extended and retracted to the target position, thereby realizing the autonomous unloading operation of the scraper loader.
[0023] Furthermore, the acquisition process of the unloading control strategy network includes:
[0024] A simulation environment for scraper unloading is built, and a reinforcement learning framework is constructed. The reinforcement learning framework describes the state through seven parameters: scraper position, scraper speed, lifting cylinder displacement, bucket cylinder displacement, shovel tip lateral coordinate, shovel tip longitudinal coordinate, and shovel tip posture angle. The action is described through three parameters: scraper longitudinal acceleration, lifting cylinder speed, and bucket cylinder speed. The reward function is set by considering tracking error and action penalty, expressed as r t =r e +r a , where r e represents the tracking error reward considering the displacement error and attitude angle error of the x-axis and y-axis, ra represents the action penalty considering the longitudinal acceleration of the scraper, the change in the speed of the lifting cylinder and the change in the speed of the dump cylinder; r e and r a The specific expression is:
[0025]
[0026] Among them, c1, c2, c3, c4, c5, c6 are coefficients; e x is the error in the x direction; e y is the error in the y direction; e γ is the bucket angle error; a t is the longitudinal acceleration of the scraper; v1 is the speed of the lifting cylinder; v2 is the speed of the dump cylinder;
[0027] The reinforcement learning framework is trained based on the constructed scraper unloading simulation environment to obtain the trained unloading control strategy network, and the trained unloading control strategy network is deployed in the upper controller.
[0028] Furthermore, the reinforcement learning framework is any one of DDPG, SAC, DSAC and PPO.
[0029] The beneficial effects brought about by the technical solution provided by the present invention include at least:
[0030] The present invention realizes the autonomous unloading operation of the shovel loader under a constrained environment. The control system of the present invention can avoid the tedious calculation of the kinematic model of the shovel loader and the inverse kinematics solution of the working mechanism, and can realize the coordinated control of the drive system and the operating system in the unloading task without the kinematic model and the dynamic model. In the control process, the collision safety of the tunnel, chute and obstacle, the constraints of the operating system and the drive system actuator are considered, and the error rates of the longitudinal, lateral and attitude angles of the shovel tip are 3.1%, 3.7% and 1.1% respectively. It can realize the control of the shovel loader to complete the unloading task according to the plan, effectively reduce the driver's stay time in the underground mine, and improve the efficiency of large-scale autonomous operation of multiple shovel loaders. It has the advantages of simplicity, safety, efficiency, and good real-time performance. BRIEF DESCRIPTION OF THE DRAWINGS
[0031] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.
[0032] Figure 1 It is a structural schematic diagram of an autonomous unloading control system for a scraper based on reinforcement learning provided by an embodiment of the present invention;
[0033] Figure 2a It is a schematic diagram of the structure of the scraper;
[0034] Figure 2b It is a schematic diagram of the structure of the scraper operation system;
[0035] Figure 3 This is a schematic diagram of a typical unloading operation environment for a scraper;
[0036] Figure 4 is the vehicle speed and cylinder control result provided by the embodiment of the present invention;
[0037] Figure 5 is the bucket posture result provided by the embodiment of the present invention;
[0038] Figure 6 This is the bucket tip trajectory result provided by an embodiment of the present invention.
[0039] Description of reference numerals:
[0040] 1. Travel system; 2. Front frame; 3. Rear frame; 4. Operating system; 5. Bucket;
[0041] 6. Upper controller; 7. Lower controller; 8. Target scraper; 9. Safety baffle;
[0042] 10. chute; 11. tunnel; 41. lifting cylinder; 42. rocker arm; 43. connecting rod;
[0043] 44. Tipping cylinder; 45. Boom. DETAILED DESCRIPTION
[0044] In order to make the objectives, technical solutions and advantages of the present invention more clear, the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings.
[0045] First of all, it should be noted that in the embodiments of the present invention, words such as "exemplarily" and "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "exemplary" in the present invention should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of the word "exemplarily" is intended to present the concept in a concrete way. In addition, in the embodiments of the present invention, the meaning expressed by "and / or" can be both, or it can be either of the two.
[0046] This embodiment provides an autonomous unloading control system for a shovel loader based on reinforcement learning, with the drive system and operating system of the shovel loader as the control objects, and the hardware system mainly includes an upper-level controller, a lower-level controller, a drive system state sensor, an operating system state sensor, and an environmental perception sensor. During the unloading operation, the perception system in the control system is used to obtain information about the drive system, the operating system, and the external environment. The policy network trained based on the reinforcement learning framework is deployed in the upper-level controller, and the control targets of the drive system and the operating system are output in combination with the current state and environmental information. The lower-level controller receives the target and outputs the control signals of the accelerator pedal, the brake pedal, the lifting cylinder, and the bucket cylinder solenoid valve in combination with the lower-level controller algorithm, controls the shovel loader to travel at the target speed, and controls the lifting cylinder and the bucket cylinder to extend and retract to the target displacement. Through the coordinated movement of the drive system and the operating system, the autonomous unloading operation of the shovel loader in a constrained mining environment is realized.
[0047] Specifically, Figure 2a As shown, the scraper loader includes a travel system 1, a front frame 2, a rear frame 3, an operating system 4, and a bucket 5; wherein the travel system 1 includes a drive system and a brake system, which receive acceleration pedal signals and brake pedal signals respectively to realize acceleration, braking, or constant speed driving of the scraper loader; wherein the drive system can be a hydraulic mechanical transmission or a motor drive. The operating system 4 is installed on the front frame 2, and a bucket 5 is installed in front of the operating system 4; as shown in FIG. Figure 2b As shown, the operating system 4 includes a lifting cylinder 41, a rocker arm 42, a connecting rod 43, a bucket cylinder 44, a lifting cylinder electromagnetic proportional valve, a bucket cylinder electromagnetic proportional valve, and a large arm 45. It is a reverse Z-type six-link operating mechanism. The operating system 4 receives electromagnetic proportional valve control signals of the lifting cylinder 41 and the bucket cylinder 44, and controls the lifting cylinder 41 and the bucket cylinder 44 to extend and retract at different speeds by controlling the flow direction and flow of the pressure oil. The lifting and unloading of the bucket 5 is achieved through the combined action of the two cylinders.
[0048] like Figure 1As shown, the control system includes a controller and a perception system; wherein the perception system is used to obtain the state information of the loader and the environmental state information; the controller includes an upper controller 6 and a lower controller 7, and the upper controller 6 and the lower controller 7 communicate in real time through the CAN bus; wherein the upper controller 6 is generally an industrial computer, in which a discharge control strategy network trained based on a reinforcement learning framework is deployed, and the upper controller 6 is used to perform trajectory planning according to the state information of the loader and the environmental state information (which can be achieved with the help of the existing A* algorithm), plan the target posture of the shovel tip of the loader, input the target posture and the current state information of the loader into the discharge control strategy network, and use the discharge control strategy network to obtain the target action information according to the state information of the loader and the target posture of the shovel tip, including the target driving speed of the loader, the target displacement of the lifting cylinder and the dump cylinder, and input it to the lower controller 7; the lower controller 7 is generally a PLC controller, which receives the walking system 1 The control targets of the travel system 1 and the operating system 4 include the target vehicle speed of the scraper loader, the target telescopic displacement of the lifting cylinder and the bucket cylinder, and the real speed of the scraper loader is obtained through the speed encoder. The displacement sensor obtains the real displacement of the lifting cylinder and the bucket cylinder in real time. According to the current vehicle speed and cylinder displacement fed back by the vehicle sensor, combined with the target vehicle speed and cylinder displacement, the deployed closed-loop control algorithm (PID control algorithm) is used to obtain the control signals of the travel system 1 and the operating system 4 (wherein the control quantity of the travel system 1 is the accelerator pedal and brake pedal signals, and the control quantity of the operating system 4 is the control signal of the electromagnetic proportional valve of the lifting cylinder 41 and the bucket cylinder 44), and input them to the target scraper 8. At the same time, the travel system 1 and the operating system 4 are controlled to make the target scraper 8 travel to the target chute at the target speed, and make the lifting cylinder and the bucket cylinder telescope to the target position. Through the coordinated movement of the travel system 1 and the operating system 4, the autonomous unloading operation of the scraper loader in the constrained mining environment is realized.
[0049] Furthermore, if Figure 3 As shown, the typical unloading environment of the shovel loader includes: a target shovel loader 8, a safety baffle 9, a chute 10 and a tunnel 11; the autonomous unloading control system needs to control the target shovel loader 8 to travel to the vicinity of the chute 10, quickly lift the ore material in the bucket of the target shovel loader 8 and dump it into the chute 10. During the driving and lifting unloading process, no part of the shovel loader can collide with the safety baffle 9 and the tunnel 11.
[0050] A perception system is set on the target scraper 8; the perception system obtains the state and external environment information of the target scraper 8 with the help of state sensors and perception sensors. Specifically, the perception system includes a laser radar, an inertial measurement unit, a speed encoder, and two displacement sensors; wherein the laser radar is used to scan the environment in the tunnel 11 where the scraper loader works, for working environment modeling, and to obtain the vehicle position, chute information, tunnel information and safety baffle information according to the established working environment model analysis, the inertial measurement unit is used to assist the laser radar in data processing, the speed encoder is used to obtain the vehicle form speed, and at the same time assist the laser radar in data processing; two displacement sensors are used to measure the displacement of the working cylinder, one of the displacement sensors is used to measure the current lifting cylinder displacement, and the other displacement sensor is used to measure the current dump bucket cylinder displacement, the chute information includes the chute position, chute height and chute width, the tunnel information includes the tunnel wall position and the tunnel top height, and the safety baffle information includes the safety baffle position and the safety baffle height.
[0051] Furthermore, the realization of the unloading strategy network includes the training of the autonomous unloading decision network and the deployment and implementation of the autonomous unloading strategy network; wherein, the training of the unloading strategy network is carried out by establishing a coupled kinematic model of the working mechanism and the drive system. In the training process of the unloading strategy network, a simulation environment for the unloading of the scraper is firstly built, which includes the size, position and orientation information of the chute, tunnel and obstacles, the shovel tip posture of the unloading operation, i.e., the lateral, longitudinal and bucket inclination reference values of the shovel tip, the scraper drive system model, the kinematic model of the working mechanism, the drive system speed, the working mechanism, the working mechanism, the driving ... The state quantities of the lifting cylinder displacement and the dump cylinder displacement of the working mechanism system; specifically, this embodiment is based on the actual structure and size parameters of the working mechanism of the scraper loader, and constructs a kinematic model of the working mechanism system through trigonometric functions. Combined with the vehicle state parameters of the scraper loader, the conversion between the vehicle position, the working cylinder displacement and the bucket posture is realized, that is, the shovel tip coordinates and the bucket posture angle are obtained through the vehicle reference point coordinates, the lifting cylinder displacement and the dump cylinder displacement. The autonomous unloading simulation environment of the scraper loader is constructed through the kinematic model, and the safety baffle, chute and tunnel information are set in the simulation environment.
[0052] The interaction between the shovel loader unloading simulation environment and the control strategy can be regarded as a Markov decision process. During the training process of the unloading control strategy network, a reasonable reward function needs to be designed. In this embodiment, the lateral, longitudinal and inclination errors of the shovel tip unloading trajectory, the cost of the shovel loader speed, the lifting cylinder and the dump cylinder control quantity, and the collision risk between the shovel loader and the tunnel, obstacles and chutes are considered; the maximum value of the control quantity of the strategy network, including the maximum speed, acceleration, deceleration, and the maximum extension and retraction speed of the cylinder are subject to the execution capability constraints of the drive system and the operating mechanism system; the shovel tip unloading trajectory tracking error is obtained by subtracting the actual value from the reference value, and the reference value is obtained by the current chute position, obstacle position, and vehicle state planning, and the actual value is obtained by the vehicle body position, the actual displacement of the lifting cylinder and the dump cylinder; the collision risk is obtained by calculating the minimum distance between the shovel loader and the tunnel, obstacles and chutes. Specifically, the reinforcement learning framework in this embodiment is obtained by the vehicle position x f , vehicle speed v f , Lifting cylinder displacement l j , Tipping cylinder displacement l f , lateral coordinate of target trajectory r x , target trajectory longitudinal coordinate r y and the target trajectory attitude angle r γ Seven parameters describe the state t , denoted as s t =[x f ,v f ,l j ,l f ,r x ,r y ,r γ ], given by the vehicle longitudinal acceleration a v , lifting cylinder speed v j and the tipping cylinder speed v f Three parameters describe the action a t , represented by a t =[a v ,v j ,v f ], considering the tracking error and action penalty, the reward function r is set, expressed as r t =r e +r a , where r e represents the tracking error reward considering the displacement error of the shovel tip and the attitude angle error of the x-axis and y-axis, r a It represents the action penalty considering the longitudinal acceleration and the change in the speed of the two cylinders; the specific expression is:
[0053]
[0054] In addition, environmental constraint penalties are added to the reward function, such as giving a larger penalty when collisions occur at locations such as tunnel side walls, chutes, obstacles, etc., to avoid safety accidents in constrained environments; relying on the reinforcement learning algorithm to complete the policy network training, the unloading control strategy that maximizes the reward can be obtained. Among them, the training algorithm can use different algorithms such as DDPG, SAC, DSAC, PPO, etc. In the policy network deployment and implementation stage, it is necessary to receive the state parameters of the perception system and input them into the trained policy network. The network outputs the target vehicle speed, lifting cylinder and dump cylinder target displacement at this moment. The lower-level controller receives the target value and the actual vehicle speed, lifting cylinder and dump cylinder actual displacement at the current moment through the communication protocol; combined with the closed-loop control algorithm that has been deployed by the lower-level controller, by outputting the acceleration and brake pedal signals, and the lifting cylinder and dump cylinder control solenoid valve control signals, the vehicle and working mechanism are controlled to reach the target speed or position, and finally the autonomous unloading task is realized.
[0055] In summary, this embodiment provides an autonomous unloading control system for shovel loaders based on reinforcement learning. It utilizes the environmental perception system information of underground shovel loaders, including environmental information such as chutes, tunnels, obstacles, etc., as well as the status of the shovel loader drive system and operating mechanism system. Based on the reinforcement learning control framework, it uses the extension and retraction amount of the operating mechanism system's lifting cylinder and dump cylinder, and the speed of the drive system as control quantities. By setting the drive system and operating system control constraints and environmental collision constraints during the control strategy learning process, a reward function for the unloading task is designed and implemented. Combined with the vehicle's environmental perception data, the coordinated control of the drive system and the operating mechanism system in a constrained environment is achieved through continuous interactive learning between the controlled shovel loader and the unloading environment, thereby achieving autonomous unloading operations in a constrained environment. Figure 3 Based on the unloading operation scenario shown in the figure, the autonomous unloading control method was verified in the simulation environment. The results are as follows Figures 4 to 6 As shown by Figures 4 to 6 It can be seen that the control system can make the bucket of the scraper move according to the planned reference posture through the coordinated control of the drive system and the operation system, and finally complete the autonomous unloading task, wherein the error rates of the longitudinal, lateral and attitude angles of the shovel tip are 3.1%, 3.7% and 1.1% respectively. The control system can make adaptive adjustments according to the location of the unloading point, the underground tunnel environment and the differences of the scraper, so as to control the scraper to complete the unloading task according to the plan, effectively reduce the driver's stay time in the underground mine, and improve the efficiency of large-scale operations of multiple scrapers. It has the advantages of simplicity, safety, efficiency and good real-time performance.
[0056] In addition, it should be noted that the present invention can be provided as a method, an apparatus or a computer program product. Therefore, the embodiment of the present invention can be in the form of a full or partial hardware embodiment, a full or partial software embodiment or an embodiment combining software and hardware. Moreover, when implemented using software, the embodiment of the present invention can be in the form of a computer program product implemented on one or more computer-usable storage media containing computer-usable program codes. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions can be transmitted from one website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center containing one or more available media sets. The available medium may be a magnetic medium (eg, a floppy disk, a hard disk, a magnetic tape), an optical medium (eg, a DVD), or a semiconductor medium. The semiconductor medium may be a solid state hard disk.
[0057] The embodiments of the present invention are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0058] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device, so that a series of operation steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable terminal device provide for implementing the process in the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0059] It should also be noted that, in this article, the terms "include", "comprise" or any other variant thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "include one..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements. In addition, the term "and / or" is only a description of the association relationship of the associated objects, indicating that there may be three relationships, for example, A and / or B, which can represent: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship, but it may also indicate an "and / or" relationship, which can be understood by referring to the context before and after. "At least one" means one or more, and "more" means two or more. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, ab, ac, bc or abc, where a, b and c can be single or plural.
[0060] In addition, it can be understood that in various embodiments of the present invention, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.
[0061] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.
[0062] In several embodiments provided by the present invention, it should be understood that the disclosed equipment, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of functional modules / units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point, the coupling or direct coupling or communication connection between each other shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms. The unit described as a separate component may or may not be physically separated, and the component displayed as a unit may or may not be a physical unit, that is, it may be located in one place, or it may be distributed on multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, each functional unit in each embodiment of the present invention can be integrated in a processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit.
[0063] If the method is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.
[0064] Finally, it should be noted that the above is only a preferred embodiment of the present invention. It should be pointed out that although the preferred embodiment of the present invention has been described, for ordinary technicians in this technical field, once the basic creative concept of the present invention is known, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as the protection scope of the present invention. Therefore, the attached claims are intended to be interpreted as including the preferred embodiment and all changes and modifications that fall within the scope of the embodiments of the present invention.
Claims
1. A self-discharging control system for a scraper based on reinforcement learning, wherein the scraper includes a walking system and an operating system, and is characterized in that: The control system includes a controller and a sensing system; wherein, The perception system is used to obtain the scraper state information and the environment state information; The controller includes an upper-layer controller and a lower-layer controller. The upper-layer controller is used to perform trajectory planning according to the shovel loader state information and the environmental state information, obtain the target position of the shovel tip, and use the unloading control strategy network trained based on the reinforcement learning framework to obtain the target action information according to the shovel loader state information and the shovel tip target position; the lower-layer controller is used to output control signals of the walking system and the operating system according to the current shovel loader state information and the target action information, control the shovel loader to perform the target action, and realize the autonomous unloading operation of the shovel loader in a constrained mining environment through the coordinated movement of the drive system and the operating system.
2. The autonomous unloading control system for a scraper based on reinforcement learning as claimed in claim 1, characterized in that: The travel system includes a drive system and a brake system. The drive system receives an accelerator pedal signal, and the brake system receives a brake pedal signal to achieve acceleration, braking or constant speed travel of the scraper. The operating system includes a bucket, a lifting cylinder, a dump bucket cylinder, a lifting cylinder control electric proportional valve and a dump bucket cylinder electric proportional valve; the operating system receives control signals of the lifting cylinder control electric proportional valve and the dump bucket cylinder electric proportional valve, and realizes the lifting and unloading of the bucket through the combined action of the lifting cylinder and the dump bucket cylinder.
3. The autonomous unloading control system for a scraper based on reinforcement learning as claimed in claim 2, characterized in that: The sensing system includes a speed encoder, a dump cylinder displacement sensor, a lifting cylinder displacement sensor, a laser radar, an inertial measurement unit and a camera; wherein, The laser radar, inertial measurement unit, speed encoder and camera cooperate with each other to realize the modeling of the working environment of the scraper, and analyze and obtain the scraper position, scraper speed, chute information, lane information and obstacle information according to the built working environment model; wherein the chute information includes the chute position, chute height and chute width, the lane information includes the lane wall position and the lane top height, and the obstacle information includes the obstacle position and obstacle height; The lifting cylinder displacement sensor is used to measure the current lifting cylinder displacement; The bucket cylinder displacement sensor is used to measure the current bucket cylinder displacement.
4. The autonomous unloading control system for a scraper based on reinforcement learning as claimed in claim 3, characterized in that: The scraper status information includes scraper position and scraper speed; The environmental status information includes chute information, lane information and obstacle information.
5. The autonomous unloading control system for a scraper based on reinforcement learning as claimed in claim 4, characterized in that: The shovel tip target position and posture include the shovel tip horizontal and vertical coordinates and the shovel tip posture angle.
6. The autonomous unloading control system for a scraper based on reinforcement learning as claimed in claim 5, characterized in that: The target motion information includes the target vehicle speed of the scraper, and the target telescopic displacements of the lifting cylinder and the dump cylinder.
7. The autonomous unloading control system for a scraper based on reinforcement learning as claimed in claim 6, characterized in that: The control signals of the traveling system and the working system include an accelerator pedal signal, a brake pedal signal, a control signal of a lifting cylinder control electric proportional valve, and a control signal of a dump cylinder electric proportional valve.
8. The autonomous unloading control system for a scraper based on reinforcement learning as claimed in claim 7, characterized in that: The lower-layer controller is used to output control signals of the walking system and the operating system according to the current scraper state information and the target action information, control the scraper to perform the target action, and realize the autonomous unloading operation of the scraper in the constrained mining environment through the coordinated movement of the driving system and the operating system, including: According to the current state information of the scraper loader and the target action information, combined with the PID control algorithm, the accelerator pedal signal, the brake pedal signal, the control signal of the lifting cylinder control electric proportional valve and the control signal of the dump cylinder electric proportional valve are output to control the drive system and the operating system, so that the scraper loader can travel at the target speed, and the lifting cylinder and the dump cylinder can be extended and retracted to the target position, thereby realizing the autonomous unloading operation of the scraper loader.
9. The autonomous unloading control system for a scraper based on reinforcement learning as claimed in claim 2, characterized in that: The acquisition process of the unloading control strategy network includes: A simulation environment for scraper unloading is built, and a reinforcement learning framework is constructed. The reinforcement learning framework describes the state through seven parameters: scraper position, scraper speed, lifting cylinder displacement, bucket cylinder displacement, shovel tip lateral coordinate, shovel tip longitudinal coordinate, and shovel tip posture angle. The action is described through three parameters: scraper longitudinal acceleration, lifting cylinder speed, and bucket cylinder speed. The reward function is set by considering tracking error and action penalty, expressed as r t =r e +r a , where r e represents the tracking error reward considering the displacement error and attitude angle error of the x-axis and y-axis, r a represents the action penalty considering the longitudinal acceleration of the scraper, the change in the speed of the lifting cylinder and the change in the speed of the dump cylinder; r e and r a The specific expression is: Among them, c1, c2, c3, c4, c5, c6 are coefficients; e x is the error in the x direction; e y is the error in the y direction; e γ is the bucket angle error; a t is the longitudinal acceleration of the scraper; v1 is the speed of the lifting cylinder; v2 is the speed of the dump cylinder; The reinforcement learning framework is trained based on the constructed scraper unloading simulation environment to obtain the trained unloading control strategy network, and the trained unloading control strategy network is deployed in the upper controller.
10. The autonomous unloading control system for a scraper based on reinforcement learning according to claim 1, characterized in that: The reinforcement learning framework is any one of DDPG, SAC, DSAC and PPO.
Citation Information
Cited By
Automatic shovel loading control method, training method, device, equipment and medium
CN119830019A
Automatic loading control method, training method, device, equipment and medium
CN119830019B
