Model training method, excavator control method, device and equipment and excavator

Abstract

The embodiment of the invention provides a model training method, an excavator control method, device and equipment and an excavator. The model training method comprises the following steps: firstly, constructing a first state space according to first environment information of a total excavation area in a simulation environment and first motion information of a sample excavator; then, a first action space is constructed according to a first control instruction of the sample excavator; then, according to a preset reward function and a security constraint condition, training a first strategy network in the first initial model to obtain an initial excavator control model; further, second environment information and second motion information of the current excavation area are used for constructing a second state space, the second state space is input into the initial excavator control model, and a second action space is obtained; and finally, according to the reward function and the action value function, training a second strategy network in the second initial model to obtain a target excavator control model. The accuracy of outputting the control instruction by the excavator control model is improved.

Description

Technical Field

[0001] The present application relates to the field of automatic control of excavators, and in particular to a model training method, an excavator control method, a device, equipment and an excavator. Background Art

[0002] As an important engineering machinery, excavators play an irreplaceable role in construction, earthquake rescue and other fields. Their versatility and high efficiency make them the core equipment widely used in various mechanized constructions. With the rapid development of artificial intelligence technology, unmanned excavators are gradually applied to the field of engineering machinery, which can realize autonomous operation in complex environments. However, how to accurately control unmanned excavators and ensure the quality of operation is crucial.

[0003] In the prior art, unmanned excavators are usually controlled by remote control, that is, the operator remotely controls the excavator's driving route, excavation actions, loading operations, etc. through a wireless communication system and image transmission equipment to ensure that the unmanned excavator completes the excavation operation in a complex environment.

[0004] However, the control of unmanned excavators in the prior art is difficult to adapt to complex and changeable actual working conditions, which affects the operating accuracy and efficiency of the excavator. Summary of the invention

[0005] The embodiments of the present application provide a model training method, an excavator control method, an apparatus, a device and an excavator, so as to solve the problem in the prior art that the control of unmanned excavators is difficult to adapt to complex and changeable actual working conditions, thus affecting the operating accuracy and efficiency of the excavators.

[0006] In a first aspect, an embodiment of the present application provides a model training method, comprising:

[0007] constructing a first state space according to first environmental information of a total excavation area in a simulated environment and first motion information of a sample excavator in the simulated environment;

[0008] Constructing a first action space according to a first control instruction of the sample excavator;

[0009] According to a preset reward function and safety constraints, a first policy network in a first initial model is trained to obtain an initial excavator control model, wherein the first policy network is used to map the first state space to the first action space;

[0010] constructing a second state space according to second environmental information of a current excavation area in the simulation environment and second motion information of the sample excavator in the simulation environment;

[0011] The second state space is input into the initial excavator control model to obtain the second action space output by the initial excavator control model, wherein the second action space includes a second control instruction; according to the reward function and the action value function, the second policy network in the second initial model is trained to obtain the target excavator control model, wherein the second policy network is used to map the second state space to the second action space.

[0012] In a possible implementation, the safety constraint condition includes at least one of the following: the joint angle of the sample excavator is within a preset angle range, the joint angular velocity is less than a preset velocity, and the distance between the bucket and the obstacle is greater than a preset distance.

[0013] In one possible implementation, the reward function is related to the efficiency, safety, and energy consumption of the sample excavator, the efficiency of the sample excavator is determined based on the action space, and the safety of the sample excavator is related to at least one of the distance between the bucket of the sample excavator and an obstacle, the joint angle, and the joint angular velocity.

[0014] In a possible implementation, the environmental information includes environmental point cloud data and soil type, and the motion information includes the joint angle, angular velocity, and bucket position of the sample excavator.

[0015] In a second aspect, an embodiment of the present application provides an excavator control method, including:

[0016] During the working process of the excavator, current environmental information of the current excavation area and current movement information of the excavator are obtained;

[0017] The current environment information and the current motion information are input into a target excavator control model to obtain a target control instruction output by the target excavator control model, wherein the target excavator control model is obtained by the model training method as described in the first aspect.

[0018] In a possible implementation, the current environment information includes current environment point cloud data and current soil type, and the current motion information includes joint angles, angular velocities, and bucket positions of the excavator.

[0019] In a third aspect, an embodiment of the present application provides a model training device, comprising:

[0020] A first construction module is used to construct a first state space according to first environmental information of a total excavation area in a simulated environment and first motion information of a sample excavator in the simulated environment;

[0021] The first construction module is further used to construct a first action space according to a first control instruction of the sample excavator;

[0022] A first training module, used for training a first policy network in a first initial model according to a preset reward function and safety constraints to obtain an initial excavator control model, wherein the first policy network is used for mapping the first state space to the first action space;

[0023] A second construction module is used to construct a second state space according to second environmental information of a current excavation area in the simulation environment and second motion information of the sample excavator in the simulation environment; and construct a second state space according to the second environmental information and the second motion information;

[0024] a processing module, configured to input the second state space into the initial excavator control model, and obtain a second action space output by the initial excavator control model, wherein the second action space includes a second control instruction;

[0025] The second training module is used to train the second strategy network in the second initial model according to the reward function and the action value function to obtain the target excavator control model, and the second strategy network is used to map the second state space to the second action space.

[0026] In a possible implementation, the safety constraint condition includes at least one of the following: the joint angle of the sample excavator is within a preset angle range, the joint angular velocity is less than a preset velocity, and the distance between the bucket and the obstacle is greater than a preset distance.

[0027] In one possible implementation, the reward function is related to the efficiency, safety, and energy consumption of the sample excavator, the efficiency of the sample excavator is determined based on the action space, and the safety of the sample excavator is related to at least one of the distance between the bucket of the sample excavator and an obstacle, the joint angle, and the joint angular velocity.

[0028] In a possible implementation, the environmental information includes environmental point cloud data and soil type, and the motion information includes the joint angle, angular velocity, and bucket position of the sample excavator.

[0029] In a fourth aspect, an embodiment of the present application provides an excavator control device, comprising:

[0030] A first acquisition module is used to acquire current environmental information of a current excavation area and current motion information of the excavator during the operation of the excavator;

[0031] The second acquisition module is used to input the current environmental information and the current motion information into a target excavator control model to obtain a target control instruction output by the target excavator control model, wherein the target excavator control model is obtained by the model training method described in the first aspect.

[0032] In a possible implementation, the current environment information includes current environment point cloud data and current soil type, and the current motion information includes joint angles, angular velocities, and bucket positions of the excavator.

[0033] In a fifth aspect, an embodiment of the present application provides an electronic device, including: a memory, a processor;

[0034] The memory stores computer-executable instructions;

[0035] The processor executes the computer-executable instructions stored in the memory, so that the processor executes the above first aspect and various possible implementations of the first aspect.

[0036] In a sixth aspect, an embodiment of the present application provides an excavator, the excavator comprising a vehicle body, an excavator boom, a memory, and a processor;

[0037] The memory stores computer-executable instructions;

[0038] The processor executes the computer-executable instructions stored in the memory, so that the processor executes the above second aspect and various possible implementations of the second aspect.

[0039] The model training method, excavator control method, device, equipment and excavator provided in the embodiments of the present application first construct a first state space based on the first environmental information of the total excavation area in the simulated environment and the first motion information of the sample excavator in the simulated environment; then, construct a first action space based on the first control instruction of the sample excavator; then, according to the preset reward function and safety constraints, train the first policy network in the first initial model to obtain the initial excavator control model. At this point, since the input first state space contains the global environmental information of the simulated environment (first environmental information), the first policy network can understand the layout of the entire environment and the obstacle distribution when generating control decisions, and according to the preset reward function and safety constraints, it can help the first policy network generate more appropriate control decisions on the basis of ensuring the safety of excavator operation. control strategy, providing the necessary data basis for the subsequent training of the second strategy network; further, according to the second environmental information of the current excavation area in the simulation environment and the second motion information of the sample excavator in the simulation environment, a second state space is constructed; then, the second state space is input into the initial excavator control model to obtain the second action space output by the initial excavator control model; finally, according to the reward function and the action value function, the second strategy network in the second initial model is trained to obtain the target excavator control model. By using the control strategy generated by the initial excavator control model with global information to train and optimize the second strategy network with local information, the generated target excavator control model can have better generalization performance and intelligent decision-making ability, and ensure the safety and efficiency of the generated control strategy, thereby improving the efficiency and accuracy of the excavator work. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0041] Figure 1 A flowchart of a model training method provided in an embodiment of the present application;

[0042] Figure 2 A schematic flow chart of an excavator control method provided in an embodiment of the present application;

[0043] Figure 3 A schematic diagram of the structure of a model training device provided in an embodiment of the present application;

[0044] Figure 4 A schematic diagram of the structure of an excavator control device provided in an embodiment of the present application;

[0045] Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application.

[0046] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0047] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0048] Before introducing the present application, the application background of the present application is first explained.

[0049] As an important engineering machinery equipment, excavators play an irreplaceable role in construction, earthquake rescue and other fields. Its versatility and high efficiency make it a core equipment widely used in various mechanized construction. Traditional excavator control mainly relies on manual operation, which requires the driver to have rich experience and skilled operating skills. However, the operating environment of excavators is usually harsh, with unfavorable factors such as dust, noise, and high temperature, which pose a threat to the driver's health and safety. In addition, manual operation is inefficient and easily affected by the driver's subjective factors, making it difficult to ensure the quality and consistency of operations. With the rapid development of artificial intelligence technology, unmanned excavators have gradually been applied to the field of engineering machinery, which can realize autonomous operation in complex environments. However, it is crucial to accurately control unmanned excavators to ensure the quality of operations.

[0050] In the prior art, unmanned excavators are usually controlled by remote control, that is, the operator remotely controls the excavator's driving route, excavation actions, loading operations, etc. through a wireless communication system and image transmission equipment to ensure that the unmanned excavator completes the excavation operation in a complex environment.

[0051] However, the remote control operation of unmanned excavators in the prior art requires continuous manual monitoring and operation, and the work efficiency is highly dependent on the experience and skill level of the operator. Under complex working conditions, the operator needs to frequently adjust the excavator's driving and working route, which is prone to reaction delays or misoperation. This affects the excavator's operating accuracy and efficiency, and the operating path cannot be adjusted in time when encountering unexpected situations, which is prone to cause work interruption or erroneous operation.

[0052] Based on this, the present application provides a model training method. Considering that the operation of unmanned excavators at this stage mainly relies on remote control by staff, if the excavator can generate appropriate control strategies according to actual conditions, it can not only greatly improve the accuracy and efficiency of the excavator's work, but also because the excavator control model needs to adapt to different working conditions and complex environments, it is difficult to directly train the control model in a complex environment, which can easily lead to unstable training or slow convergence. Therefore, a two-stage model training method can be adopted to gradually optimize the policy network. Specifically, an initial policy network (first policy network) can be generated in a simplified state space (first state space) containing global environmental information, so that the model has global planning capabilities and safety control capabilities; further, the first policy network with global environmental information is used to train the second policy network with local environmental information to optimize the control strategy of the target excavator. Through this step-by-step optimization training method, the target excavator control model can quickly adapt to complex dynamic environments on the basis of initial training, and finally generate a better control strategy.

[0053] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0054] Figure 1 A flow chart of the model training method provided in the embodiment of the present application is as follows: Figure 1 As shown, the method includes:

[0055] S101 : construct a first state space according to first environmental information of a total excavation area in a simulation environment and first motion information of a sample excavator in the simulation environment.

[0056] The environmental information includes environmental point cloud data and soil type, and the motion information includes the joint angle, angular velocity and bucket position of the sample excavator.

[0057] It should be noted that the first motion information can also be interpreted as the proprioceptive information of the excavator, that is, the posture and position of the excavator boom and bucket. In addition to the environmental point cloud data and soil type, the first environmental information can also include some data that cannot be obtained through actual data acquisition equipment, such as more accurate coordinates, contact points between objects, normal vectors and other geometric information, as well as implicit variables, such as environmental dynamic parameters, object mass, friction, etc., and even possible future changes in the excavation area. Optionally, the first environmental information can also be interpreted as privileged information, which is generated by predefining the simulation environment. The purpose is to provide it as prior knowledge to the first policy network in the subsequent first initial model, to assist the first initial model to learn more accurate scene understanding and task strategies, that is, to enable the model to have a global perspective, thereby generating more accurate, safe and efficient control strategies. Correspondingly, the first policy network can also be interpreted as a privileged policy network, and the control strategy generated by the first policy network can also be interpreted as a privileged strategy.

[0058] Optionally, the simulation environment is a virtual simulation environment built to imitate the actual working environment of the excavator, and the model environment can be changed using domain randomization technology, that is, the terrain, lighting, object appearance, etc. in the simulation environment can be randomly adjusted. Through this domain randomization technology, the generalization ability of the initial excavator control model can be improved; the environmental point cloud data includes the terrain information and obstacle information of the model environment; the motion information is the joint angle, angular velocity and bucket position of the excavator obtained by using sensors through the excavator's own state estimation module.

[0059] It can be understood that by constructing the first state space, the motion information of the excavator itself can be reflected, and the global layout of the total excavation area in the surrounding simulation environment can also be reflected, providing a data basis for the subsequent training of the first strategy network.

[0060] S102: Construct a first action space according to a first control instruction of the sample excavator.

[0061] Among them, the control instructions can be further converted into actual hydraulic control signals to drive the various joints of the excavator to move.

[0062] It can be understood that the first action space is formed according to the first control instruction of the sample mining machine in the first state space, which is a set of executable actions of the sample mining machine.

[0063] S103: According to a preset reward function and safety constraints, a first policy network in the first initial model is trained to obtain an initial excavator control model.

[0064] The first strategy network is used to map the first state space to the first action space, that is, the actions corresponding to different sample excavator states.

[0065] It should be noted that the first strategy network is executed by the reinforcement learning module inside the sample excavator, and the reinforcement learning module includes a strategy network for outputting control instructions according to the current state, namely, joint angles and angular velocities; in addition, the first strategy network is a deep neural network, which can be a multilayer perceptron or a convolutional neural network. The specific network model used can be selected according to actual conditions, and the embodiments of the present application do not make specific restrictions here.

[0066] For example, suppose the first action space is a t =(q t , q' t ), the first state space s t =(o t , p t ), where q t is the joint angle; q' t is the joint angular velocity; o t is the first motion information; t is the first environment information. Then use the first strategy network to generate a probability distribution and then sample it:

[0067] a~π θ (a t |s t )

[0068] Among them, a is the state s t Mapped to action a t The probability distribution of θ is the network parameter, π θ For strategy.

[0069] In one achievable manner, the safety constraint condition includes at least one of the following: the joint angle of the sample excavator is within a preset angle range, the joint angular velocity is less than a preset velocity, and the distance between the bucket and the obstacle is greater than a preset distance.

[0070] It should be understood that the unmanned excavator has a complex operating environment and needs to avoid obstacles, maintain stable operation, and avoid dangerous actions or states. Therefore, it is necessary to set safety constraints and limit the control strategy output by the model to ensure that the excavator does not make illegal or dangerous operations.

[0071] For example, suppose the safety constraint c i =(s t , a t )≤0, i=1,2,…,N, where i is the number of safety constraints, s t is the first state space, at is the first action space.

[0072] Specifically, assuming the joint angle is q t , then the joint angle of the sample excavator is within the preset angle range and can be expressed as:

[0073] q min ≤q t ≤q max

[0074] where q min is the minimum value of the joint angle, q max is the maximum value of the joint angle.

[0075] Assume the joint angular velocity is q' t In order to avoid danger or high energy consumption caused by excessively fast movement of the sample excavator, the joint angular velocity needs to be less than the preset speed:

[0076] q' t ≤q' max

[0077] Among them, q' max is the maximum value of the joint angular velocity.

[0078] Assuming that the distance between the bucket and the obstacle is d, in order to avoid collision, the distance between the bucket and the obstacle is greater than the preset distance:

[0079] d=|bucket position-obstacle position|, d≥d min

[0080] Among them, d min is the minimum distance between the bucket and the obstacle.

[0081] It can be understood that by adding safety constraints, the stable operation of the sample excavator can be ensured, and the first strategy network can be subsequently guided to generate a control strategy that is more in line with actual operation requirements, thereby improving the accuracy of the excavator control model in generating control instructions.

[0082] Optionally, the reward function is related to the efficiency, safety, and energy consumption of the sample excavator, wherein the efficiency of the sample excavator is determined according to the action space, and the safety of the sample excavator is related to at least one of the distance between the bucket of the sample excavator and the obstacle, the joint angle, and the joint angular velocity.

[0083] For example, suppose the reward function is r t , then the specific reward function can be:

[0084] r t =w1×V t +w2×C t+w3×E t

[0085] Among them, r t is the reward function; w1, w2, w3 are weight coefficients; V t is the volume of earth excavated; C t For collision penalty; E t Energy consumption of the sample excavator.

[0086] It should be understood that by setting the weight coefficients of different objectives in the reward function, different requirements are balanced to ensure that the first policy network learns the optimal balance strategy during training, thereby guiding the model to learn a safe and efficient control strategy.

[0087] Furthermore, after setting the reward function and the safety constraint conditions, the first policy network in the first initial model is trained by establishing an objective function to obtain an initial excavator control model. The specific objective function may be:

[0088] J θ =E[min(r t (θ)A t ,clip(r t (θ),1-ε,1+ε)A t )-βc t ]

[0089] Among them, θ is the network parameter; E is the expectation; r t (θ) is the strategy π θ The probability ratio after taking an action at time step t is used to measure the difference between the new and old strategies; clip(r t (θ),1-ε,1+ε) is the value of r t The value of (θ) is limited to between 1-ε and 1+ε, which is used to limit the amplitude of strategy update and ensure the stability of training; ε is a hyperparameter, usually set to 0.1 or 0.2; β is the penalty coefficient; c t is the penalty term for violating the constraint at time step t.

[0090] It can be understood that by utilizing the first environmental information of the total excavation area in the simulated environment, the global state of the current environment can be accurately described, so that the initial excavator control model can converge faster during the training phase, and by setting safety constraints and reward functions, the initial excavator control model can accurately generate the optimal action in each state, thereby greatly improving the efficiency of strategy training.

[0091] S104: construct a second state space according to second environment information of the current excavation area in the simulation environment and second motion information of the sample excavator in the simulation environment.

[0092] It should be noted that the second environmental information of the current excavation area in the simulated environment obtained in this step is obtained through a data acquisition device, and the environmental point cloud data and soil type can be obtained through a visual perception module in the sample excavator. The visual perception module includes a depth (RGBD) camera and an image processing unit, etc., which are used to collect and process information about the environment around the excavator, generate color images and depth images, and generate a local height map to represent the terrain undulations and obstacle information.

[0093] It should be understood that the second environmental information can also be interpreted as visual information. Different from the privileged information, the privileged information is global information used to describe the total mining area, while the visual information is environmental information within the collection range collected by the data collection device, which is local information.

[0094] It should be understood that in order to further train the second strategy network in the second initial model, it is necessary to construct a second state space based on the second environmental information and the second motion information as input information of the initial excavator control model and the second strategy network.

[0095] S105, inputting the second state space into the initial excavator control model to obtain a second action space output by the initial excavator control model, wherein the second action space includes a second control instruction.

[0096] It should be understood that by inputting the second environmental information and the second motion information into the initial excavator control model, the initial excavator control model can generate the corresponding optimal control instructions under the current state, as well as the corresponding state space and action space at the next sample moment.

[0097] S106. According to the reward function and the action value function, train the second strategy network in the second initial model to obtain the target excavator control model.

[0098] The second strategy network is used to map the second state space to the second action space. Accordingly, the second strategy network can also be interpreted as a visual strategy network, and the control instructions generated by the visual strategy network according to the second state space can also be interpreted as a visual strategy.

[0099] It should be understood that by inputting the second state space into the second strategy network, the second strategy network outputs the action probability distribution for each state, that is, the probability distribution of the second state space mapped to the second action space, and calculates the first objective function value of the control instruction generated by the second strategy network according to the reward function and the action value function; further, the second objective function value of the control instruction generated by the initial excavator control model is calculated according to the second action space output by the initial excavator control model, the reward function and the action value function, and finally, according to the difference between the first objective function value and the second objective function value, the network parameters of the second strategy network are adjusted to finally obtain the target excavator control model.

[0100] For example, suppose the second action space output by the initial excavator control model is a t =(q t , q' t ), the second environment information is s t =(o t , v t ), the reward function corresponding to the second action space output by the initial excavator control model is r t , the state space at the next sample time is s t+1 =(o t+1 , v t+1 ), the strategy generated by the second strategy network is Then the objective function can be:

[0101]

[0102] in, is the action value function; α is the conservative regularization coefficient; is the strategy entropy; λ is the entropy regularization coefficient; o t is the second motion information; t is the second environment information; a t is the second action space; r t is the reward value corresponding to the second action space; t+1 is the second motion information corresponding to the next sample moment; t+1 It is the second environment information corresponding to the next sample time.

[0103] It can be understood that by utilizing the second state space and the second action space, and using the parameters generated by the initial excavator control model to train the second strategy model, the invalid actions of the second strategy model in the random exploration process can be reduced, the training efficiency can be improved, and ultimately the optimal excavator control model, i.e., the target excavator control model, can be trained.

[0104] The model training method provided in the embodiment of the present application first constructs a first state space according to the first environmental information of the total excavation area in the simulated environment and the first motion information of the sample excavator in the simulated environment; then, constructs a first action space according to the first control instruction of the sample excavator; then, trains the first policy network in the first initial model according to the preset reward function and safety constraints to obtain the initial excavator control model. At this point, since the input first state space contains the global environmental information of the simulated environment (first environmental information), the first policy network can understand the layout of the entire environment and the obstacle distribution when generating control decisions, and can help the first policy network generate a more suitable control strategy on the basis of ensuring the safety of the excavator operation according to the preset reward function and safety constraints, so as to provide a reference for subsequent training. The second strategy network provides the necessary data basis; further, according to the second environmental information of the current excavation area in the simulation environment and the second motion information of the sample excavator in the simulation environment, a second state space is constructed; then, the second state space is input into the initial excavator control model to obtain the second action space output by the initial excavator control model; finally, according to the reward function and the action value function, the second strategy network in the second initial model is trained to obtain the target excavator control model. By using the control strategy generated by the initial excavator control model with global information to train and optimize the second strategy network with local information, the generated target excavator control model can have better generalization performance and intelligent decision-making ability, and ensure the safety and efficiency of the generated control strategy, thereby improving the efficiency and accuracy of the excavator work.

[0105] Figure 2 A flow chart of the excavator control method provided in the embodiment of the present application is shown in FIG. Figure 2 As shown, the method includes:

[0106] S201. During the working process of the excavator, current environmental information of the current excavation area and current movement information of the excavator are obtained.

[0107] Among them, the current environment information includes the current environment point cloud data and the current soil type, and the current motion information includes the joint angle, angular velocity and bucket position of the excavator.

[0108] It should be noted that the actual working process of the excavator is similar to the method of obtaining environmental information and motion information during the training process, and the embodiments of the present application will not be elaborated here.

[0109] S202: Input the current environment information and the current motion information into the target excavator control model, and obtain the target control instruction output by the target excavator control model.

[0110] Among them, the target excavator control model is Figure 1 The model training method in the embodiment is obtained.

[0111] Optionally, after obtaining the target control instructions output by the target excavator control model, the control execution module inside the excavator can be used to convert the target control instructions output by the target excavator control model, that is, the output joint angle and joint angular velocity instructions, into actual hydraulic control signals to drive the various joints of the excavator to move.

[0112] It can be understood that the trained target excavator control model is deployed on a real unmanned excavator platform. By obtaining the current environmental information of the current excavation area in the real environment and the current motion information of the excavator, the target excavator control model corresponds to the output response target control instructions and performs fine-tuning to adapt to various complex situations in the real environment, thereby achieving accurate control of the excavator in various complex environments and improving the working accuracy and efficiency of the excavator.

[0113] Figure 3 A schematic diagram of the structure of the model training device provided in the embodiment of the present application; Figure 3 As shown, the model training device 30 includes:

[0114] A first construction module 301 is used to construct a first state space according to first environmental information of a total excavation area in a simulation environment and first motion information of a sample excavator in the simulation environment;

[0115] The first construction module 301 is further used to construct a first action space according to a first control instruction of the sample excavator;

[0116] A first training module 302 is used to train a first policy network in a first initial model according to a preset reward function and safety constraints to obtain an initial excavator control model, wherein the first policy network is used to map a first state space to a first action space;

[0117] A second construction module 303 is used to construct a second state space according to second environmental information of a current excavation area in the simulation environment and second motion information of the sample excavator in the simulation environment;

[0118] A processing module 304 is used to input the second state space into the initial excavator control model to obtain a second action space output by the initial excavator control model, where the second action space includes a second control instruction;

[0119] The second training module 305 is used to train the second strategy network in the second initial model according to the reward function and the action value function to obtain the target excavator control model. The second strategy network is used to map the second state space to the second action space.

[0120] In a possible implementation, the safety constraint condition includes at least one of the following: the joint angle of the sample excavator is within a preset angle range, the joint angular velocity is less than a preset velocity, and the distance between the bucket and the obstacle is greater than a preset distance.

[0121] In one possible implementation, the reward function is related to the efficiency, safety, and energy consumption of the sample excavator. The efficiency of the sample excavator is determined based on the action space, and the safety of the sample excavator is related to at least one of the distance between the bucket of the sample excavator and the obstacle, the joint angle, and the joint angular velocity.

[0122] In a possible implementation, the environmental information includes environmental point cloud data and soil type, and the motion information includes joint angles, angular velocities, and bucket positions of the sample excavator.

[0123] The model training device provided in this embodiment can execute the method provided in the above-mentioned model training method embodiment. Its implementation principle and technical effects are similar, and will not be described in detail in this embodiment.

[0124] Figure 4 A schematic diagram of the structure of the excavator control device provided in the embodiment of the present application; Figure 4 As shown, the excavator control device 40 includes:

[0125] The first acquisition module 401 is used to acquire the current environment information of the current excavation area and the current movement information of the excavator during the operation of the excavator;

[0126] The second acquisition module 402 is used to input the current environment information and the current motion information into the target excavator control model, and obtain the target control instruction output by the target excavator control model. The target excavator control model is Figure 1 The model training method in the embodiment is obtained.

[0127] In a possible implementation, the current environment information includes current environment point cloud data and current soil type, and the current motion information includes joint angles, angular velocities, and bucket positions of the excavator.

[0128] The excavator control device provided in this embodiment can execute the method provided in the above-mentioned excavator control method embodiment, and its implementation principle and technical effect are similar, so this embodiment will not be described in detail here.

[0129] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Figure 5As shown, the electronic device 50 provided in this embodiment includes: at least one processor 501 and a memory 502. Optionally, the device 50 also includes a communication component 503. The processor 501, the memory 502 and the communication component 503 are connected via a bus 504.

[0130] During the specific implementation process, at least one processor 501 executes the computer execution instructions stored in the memory 502, so that at least one processor 501 executes the above-mentioned model training method.

[0131] The specific implementation process of the processor 501 can be found in the above method embodiment, and its implementation principle and technical effect are similar, so this embodiment will not be repeated here.

[0132] In the above embodiments, it should be understood that the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. A general-purpose processor may be a microprocessor or any conventional processor. The steps of the method disclosed in the invention may be directly implemented as being executed by a hardware processor, or may be executed by a combination of hardware and software modules in the processor.

[0133] The memory may include a high-speed memory (Random Access Memory, RAM), and may also include a non-volatile memory (Non-volatile Memory, NVM), such as at least one disk memory.

[0134] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into an address bus, an information bus, a control bus, etc. For ease of representation, the bus in the drawings of this application is not limited to only one bus or one type of bus.

[0135] The embodiment of the present application also provides an excavator, the excavator comprising a vehicle body, an excavator boom, a memory, and a processor;

[0136] The memory stores computer-executable instructions;

[0137] The processor executes the computer-executable instructions stored in the memory, so that the processor executes the above-mentioned excavator control method.

[0138] The division of units is only a logical function division, and there may be other divisions in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interface, device or unit, which can be electrical, mechanical or other forms.

[0139] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0140] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0141] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods of each embodiment of the present application.

[0142] Finally, it should be noted that those skilled in the art will readily conceive of other embodiments of the present application after considering the specification and practicing the invention disclosed herein. The present application is intended to cover any variations, uses or adaptations of the present application, which follow the general principles of the present application and include common knowledge or customary technical means in the art that are not disclosed in the present application, are not limited to the precise structures described above and shown in the drawings, and may be modified and changed in various ways without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. A model training method, characterized in that: Applied to electronic equipment, the method comprises: constructing a first state space according to first environmental information of a total excavation area in a simulated environment and first motion information of a sample excavator in the simulated environment; Constructing a first action space according to a first control instruction of the sample excavator; According to a preset reward function and safety constraints, a first policy network in a first initial model is trained to obtain an initial excavator control model, wherein the first policy network is used to map the first state space to the first action space; constructing a second state space according to second environmental information of a current excavation area in the simulation environment and second motion information of the sample excavator in the simulation environment; Inputting the second state space into the initial excavator control model to obtain a second action space output by the initial excavator control model, wherein the second action space includes a second control instruction; According to the reward function and the action value function, a second strategy network in the second initial model is trained to obtain a target excavator control model, wherein the second strategy network is used to map the second state space to the second action space.

2. The method according to claim 1, characterized in that The safety constraint condition includes at least one of the following: the joint angle of the sample excavator is within a preset angle range, the joint angular velocity is less than a preset velocity, and the distance between the bucket and the obstacle is greater than a preset distance.

3. The method according to claim 2, characterized in that The reward function is related to the efficiency, safety, and energy consumption of the sample excavator. The efficiency of the sample excavator is determined based on the action space, and the safety of the sample excavator is related to at least one of the distance between the bucket of the sample excavator and the obstacle, the joint angle, and the joint angular velocity.

4. The method according to any one of claims 1 to 3, characterized in that: The environmental information includes environmental point cloud data and soil type, and the motion information includes the joint angle, angular velocity and bucket position of the sample excavator.

5. A method for controlling an excavator, characterized in that: Applied to an excavator, the method comprises: During the working process of the excavator, current environmental information of the current excavation area and current movement information of the excavator are obtained; The current environment information and the current motion information are input into a target excavator control model to obtain a target control instruction output by the target excavator control model, wherein the target excavator control model is obtained by the model training method according to any one of claims 1 to 4.

6. The method according to claim 5, characterized in that The current environment information includes current environment point cloud data and current soil type, and the current motion information includes joint angles, angular velocities, and bucket positions of the excavator.

7. A model training device, characterized in that: include: A first construction module is used to construct a first state space according to first environmental information of a total excavation area in a simulated environment and first motion information of a sample excavator in the simulated environment; The first construction module is further used to construct a first action space according to a first control instruction of the sample excavator; A first training module, used for training a first policy network in a first initial model according to a preset reward function and safety constraints to obtain an initial excavator control model, wherein the first policy network is used for mapping the first state space to the first action space; A second construction module is used to construct a second state space according to second environmental information of a current excavation area in the simulation environment and second motion information of the sample excavator in the simulation environment; a processing module, configured to input the second state space into the initial excavator control model, and obtain a second action space output by the initial excavator control model, wherein the second action space includes a second control instruction; The second training module is used to train the second strategy network in the second initial model according to the reward function and the action value function to obtain the target excavator control model, and the second strategy network is used to map the second state space to the second action space.

8. An excavator control device, characterized in that: include: A first acquisition module is used to acquire current environmental information of a current excavation area and current motion information of the excavator during the operation of the excavator; The second acquisition module is used to input the current environmental information and the current motion information into a target excavator control model to obtain a target control instruction output by the target excavator control model, wherein the target excavator control model is obtained by the model training method according to any one of claims 1 to 4.

9. An electronic device, characterized in that: include: Memory, processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory, so that the processor performs the method according to any one of claims 1 to 4.

10. An excavator, characterized in that: It includes a vehicle body, an excavator arm, a memory and a processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory, so that the processor performs the method according to claim 5 or 6.

Images