Boom action model training method and boom action recommendation method based on DDPG algorithm

Abstract

The present invention discloses a boom action model training method and a boom action recommendation method based on a DDPG algorithm, the training method comprising: creating an Actor network and a Critic network; interacting with the environment through the Actor network to determine the second state reached by the boom taking a first action in the first state and the reward value obtained, the preset reward and punishment strategy used for the reward value includes a reward and punishment mechanism associated with at least one of the boom target posture and the number of boom action sections; storing the first state, the first action, the reward value and the second state in a training database; training the Actor network and the Critic network to obtain a trained boom action model. In this way, the present application trains a boom action model based on the DDPG algorithm, and uses a reward and punishment mechanism associated with motion parameters such as the boom's posture, position, and number of action sections during training, so that the model can be applied to recommend boom actions to meet the needs of reasonable boom movement and / or reaching a desired state.

Description

Technical Field

[0001] The present invention relates to the field of automatic control technology, and in particular to a boom action model training method based on a DDPG algorithm, a boom action recommendation method, an electronic device and a readable storage medium. Background Art

[0002] As an intelligent mechanical device, the main function of a mechanical boom is to assist or replace humans in completing certain work tasks. At present, engineering equipment with multi-joint booms, such as pump trucks and truck cranes, mainly rely on the collaborative work of multiple relevant personnel to complete the control of the boom movement. However, this type of equipment has a long boom, multiple postures, and most of them are high-altitude operations. In the complex environment of the construction site, it is difficult to determine the actions to be performed by the boom to meet the needs of making the boom move reasonably and / or reach the desired state. Summary of the invention

[0003] In view of this, the purpose of the present invention is to provide a boom action model training method, a boom action recommendation method, an electronic device and a readable storage medium based on the DDPG algorithm (Deep Deterministic Policy Gradient). The boom action model is trained based on the DDPG algorithm, and a reward and punishment mechanism related to motion parameters such as the boom's posture, position, number of action sections, etc. is used during training. The model can be applied to recommend boom actions to meet the needs of reasonable boom movement and / or reach a desired state.

[0004] To achieve the above object, the present invention provides a boom motion model training method based on the DDPG algorithm, the method comprising:

[0005] Creating an initial boom motion model, wherein the boom motion model includes an Actor network and a Critic network;

[0006] Determine, by interacting with the environment through the Actor network, a second state reached by the boom taking a first action in the first state and a reward value obtained, wherein the reward value is determined according to a preset reward and punishment strategy, and the preset reward and punishment strategy includes a reward and punishment mechanism associated with at least one of a target posture of the boom and the number of boom action sections;

[0007] The first state, the first action, the reward value and the second state obtained by the interaction are used as training data and stored in a training database;

[0008] Based on the training data in the training database, the Actor network and the Critic network are trained to obtain a trained boom motion model.

[0009] Optionally, in the preset reward and punishment strategy, the reward and punishment mechanism associated with the target posture of the boom includes rewarding or punishing according to the matching result between the posture of the boom and the target posture of the boom, and the reward and punishment mechanism associated with the number of action sections of the boom includes rewarding according to the number of action sections of the boom.

[0010] Optionally, the preset reward and punishment strategy includes a reward and punishment mechanism associated with the target posture of the boom and the number of boom action sections, and also includes a reward and punishment mechanism associated with the target position of the end of the boom. The interaction between the Actor network and the environment to determine the second state reached by the boom taking the first action in the first state and the reward value obtained include:

[0011] Interacting with the environment through the Actor network to determine a second state reached by the boom taking a first action in the first state;

[0012] Determine the end position of the boom according to the second state, and obtain the distance value between the end position of the boom and the target position of the end of the boom; determine the posture of the boom according to the second state; determine the number of action sections of the boom according to the first action;

[0013] The reward value is determined according to the preset reward and punishment strategy, and the preset reward and punishment strategy is:

[0014] reward=r+r1+r2+r3

[0015] Among them, reward is the reward value; r is the initial value, which is the negative value of the distance value; r1 is the reward and punishment value associated with the arm end position, if the distance value is within the preset range, r1 is a positive value, if the distance value is not within the preset range, r1 is a negative value; r2 is the reward and punishment value associated with the arm posture, if the distance value is within the preset range and the arm posture is similar to the arm target posture, r2 is a positive value, if the distance value is within the preset range and the arm posture is not similar to the arm target posture, r2 is a negative value; r3 is the reward and punishment value associated with the number of arm action sections, when the distance value is within the preset range and the arm posture is similar to the arm target posture, it is determined according to the number of arm action sections, and r3 is a positive value.

[0016] Optionally, the preset reward and punishment strategy further includes a reward and punishment mechanism associated with the target position of the end of the boom, and the interaction between the Actor network and the environment to determine the second state reached by the boom taking the first action in the first state and the reward value obtained include:

[0017] resetting the first state;

[0018] By interacting with the environment through the Actor network, determining a second state reached by the arm frame taking a first action in the first state and a reward value obtained;

[0019] When the interaction ends, determining whether the posture of the arm and / or the end position of the arm meets a preset condition;

[0020] If the preset conditions are met, the step of storing the first state, the first action, the reward value and the second state obtained by the interaction as training data in a training database is entered;

[0021] If the preset condition is not met, return to the step of resetting the first state.

[0022] Optionally, the training of the Actor network and the Critic network based on the training data in the training database to obtain a trained boom motion model includes:

[0023] Randomly extract training data from the training database to train the Actor network and the Critic network;

[0024] Interacting with the environment through the trained Actor network, and updating the training data in the training database according to the interaction results;

[0025] Continue to train the Actor network and the Critic based on the updated training data;

[0026] Return to the step of interacting with the environment through the trained Actor network until the training condition is met and stop the training to obtain the trained arm motion model.

[0027] The present application also provides a boom action recommendation method, which is applied to a boom action model trained by any of the above methods, and the boom action recommendation method comprises:

[0028] Get the current state of the arm;

[0029] The current state of the boom is input into the boom action model for environmental interaction, and a recommended action of the boom is output.

[0030] Optionally, inputting the current state of the boom into the boom action model to interact with the environment and outputting a recommended action of the boom includes:

[0031] Inputting the current state of the boom into the boom action model to interact with the environment, determining a recommended boom action of the boom and a next state to be reached by taking the recommended boom action;

[0032] Determining whether the posture of the boom and the end position of the boom meet preset conditions in the first state;

[0033] If the preset condition is met, the number of times the preset condition is met continuously is increased by one;

[0034] If the number of times the preset condition is continuously satisfied does not reach the preset number, continue searching and return to the step of inputting the current state of the boom into the boom motion model for environmental interaction;

[0035] If the preset conditions are met consecutively for a preset number of times, the interaction is terminated and the recommended arm action is output.

[0036] Optionally, after determining whether the posture of the boom and / or the end position of the boom meets a preset condition in the first state, the method further includes:

[0037] If the preset conditions are not met, the number of search steps is increased by one;

[0038] If the number of search steps reaches the preset number of search steps, the interaction ends and a recommendation failure message is output.

[0039] The present application also provides an electronic device, comprising a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the computer program implements the steps of any of the above methods when executed by the processor.

[0040] The present application also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above methods are implemented.

[0041] From the above, the arm motion model training method based on the DDPG algorithm of the present application includes: creating an Actor network and a Critic network; interacting with the environment through the Actor network to determine the second state reached by the arm taking the first action in the first state and the reward value obtained, and the preset reward and punishment strategy used for the reward value includes a reward and punishment mechanism associated with at least one of the target posture of the arm, the target position of the end of the arm, and the number of arm action sections of the arm; storing the first state, the first action, the reward value and the second state in a training database; training the Actor network and the Critic network to obtain a trained arm motion model. In this way, the present application trains the arm motion model based on the DDPG algorithm, which can enable the model to be applied to recommend arm motions, and during training, a reward and punishment mechanism associated with motion parameters such as the arm's posture, position, and number of arm action sections is used, so that the arm motion recommended by the model can meet the needs of reasonable arm motion and / or reach the desired state.

[0042] The arm action recommendation method of the present application includes: obtaining the current state of the arm; inputting the current state of the arm into the arm action model for environmental interaction, and outputting the recommended arm action. In this way, since the arm action model is trained based on the DDPG algorithm, and a reward and punishment mechanism related to motion parameters such as the arm's posture, position, and number of action segments is used during training, the arm's recommended action can meet the requirements of reasonable arm movement and / or reaching the desired state. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0044] Figure 1 A flowchart of a boom motion model training method based on a DDPG algorithm is provided in accordance with an embodiment of the present invention.

[0045] Figure 2 A schematic diagram of the network structure used in the arm motion model training method based on the DDPG algorithm provided in one embodiment of the present invention.

[0046] Figure 3 A schematic flow chart of a boom motion recommendation method provided in another embodiment of the present invention. DETAILED DESCRIPTION

[0047] The specific embodiments of the present invention will be described in detail below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments. Based on the description of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0048] In the description of the present invention, unless otherwise clearly specified and limited, the terms "set", "install", "connection" and the like should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium. For ordinary technicians in this field, the specific meanings of the above terms can be understood according to specific circumstances.

[0049] The terms "first", "second", "third", etc. are merely used to distinguish values ​​or elements of similar attributes, rather than to indicate or imply relative importance or a particular order.

[0050] The terms "comprises," "comprising," or any other variations thereof, are intended to cover a non-exclusive inclusion of the elements listed and may also include additional elements not expressly listed.

[0051] Figure 1 FIG. 1 is a flow chart of a method for training a boom motion model based on a DDPG algorithm according to an embodiment of the present invention. Figure 1 As shown, a boom motion model training method based on the DDPG algorithm includes:

[0052] S1, create the initial arm motion model, which includes the Actor network and the Critic network;

[0053] S2, interacting with the environment through the Actor network, determining the second state reached by the boom taking the first action in the first state and the reward value obtained, wherein the reward value is determined according to a preset reward and punishment strategy, and the preset reward and punishment strategy includes a reward and punishment mechanism associated with at least one of the target posture of the boom and the number of action sections of the boom;

[0054] S3, storing the first state, the first action, the reward value and the second state obtained by the interaction as training data in a training database;

[0055] S4, based on the training data in the training database, train the Actor network and the Critic network to obtain a trained boom motion model.

[0056] In this way, the present application trains the boom motion model based on the DDPG algorithm, so that the model can be applied to recommend boom motions, and a reward and punishment mechanism related to motion parameters such as the boom's posture, position, and number of action segments is used during training, so that the boom motion recommended by the model can meet the requirements of reasonable boom movement and / or reach the desired state.

[0057] When training the model, the DDPG algorithm creates copies of the neural network for the Actor network and the Critic network, that is, two independent target Actor networks and target Critic networks, thus obtaining four networks: online Actor network, target Actor network, online Critic network, and target Critic network. Using the fixed network technology, after training a batch of data, the parameters of the online network are updated by the gradient ascent / gradient descent algorithm, and then the parameters of the target network are updated by the sliding average (soft update) method. The sliding average means that when updating the target network parameters, it follows: θ′→τθ+(1-τ)θ′, and τ<<1, generally τ=0.001, which means that the target network parameters can only change slowly, greatly improving the stability of learning.

[0058] Please refer to Figure 2 , is a schematic diagram of the network structure used in the arm motion model training method based on the DDPG algorithm, and describes the main process of network update during training. In the subsequent formula description, θ Q ,θ Q′ ,θ μ ,θ μ′ They represent the network parameters of the online critic network, target critic network, online actor network, and target actor network respectively; a represents the executed action (in this application, it represents the change in the joint angle of the arm); s represents the state (in this application, it represents the joint angle of the arm); N represents the number of actions a.

[0059] The online critic network adjusts the parameter θ Q The update adopts the TD-error method in the DQN algorithm (Deep Q-network), and the loss function minimizes the mean square error:

[0060]

[0061] Among them, the target value y i =r i +Q′(S i+1 ,μ′(s i+1 |θ μ′ )|θ Q′ ), the target value is calculated using the target Critic network and the target Actor network. This is done to make the learning process of the network parameters of the online Critic network more stable and easier to converge. With the loss function L, we can find L for θ based on the standard backward bp method. Q Gradient Optimize and update it to get the new parameter θ Q .

[0062] Network parameters θ of the online Actor network μ The update follows a deterministic strategy, and the formula is:

[0063]

[0064] The target Actor network and the target Critic network are updated using a sliding average method, and the formula is:

[0065] θ Q′ ←τθ Q +(1-τ)θ Q′ ,

[0066] θ μ′ ←τθ μ +(1-τ)θ μ′ .

[0067] The following introduces the input and output of the online critic network, target critic network, online actor network, and target actor network.

[0068] Online Actor Network: Input is the current state s i , the output is the behavior value a of the current state i =μ(s i |θ μ ). Since the action output by the online Actor network gives a Q value in the online critic network, the outputs of the online Actor network and the online Critic network are used together to update the parameters of the online Actor network so that the Q value output by the online critic network is maximized, that is:

[0069]

[0070] Target Actor Network: Its input is the next state s i+1 , in calculating the target Critic's value Q'(s i+1 ,μ′(s i+1 |θ μ′ )|θ Q′ ), a i+1 =μ′(s i+1 |θ μ′ ) is used to predict the behavior value of the next state;

[0071] Current Critic network: input is the current state S i and actually perform action a i, its output is first used to calculate the loss function, the formula is: At the same time, it is also used to update some parameters in the online Actor network

[0072] Target Critic Network: Input is the next state s i+1 And the strategy μ′(s) output by the target Actor network i+1 |θ μ′ ), output the target value used to calculate TD-error, that is: y i =r i +Q′(s i+1 ,μ′(s i+1 |θ μ′ )|θ Q′ ).

[0073] Based on the application of the above network structure, the training method of the present application creates a boom motion model, so that the boom motion recommended by the model can meet the requirements of reasonable boom movement and / or reaching a desired state.

[0074] In step S1, an initial arm motion model is created, which includes an Actor network and a Critic network. The Actor network includes an online Actor network and a target Actor network, and the Critic network includes an online Critic network and a target Critic network.

[0075] In step S2, the Actor network interacts with the environment to determine the second state reached by the boom taking the first action in the first state and the reward value obtained, wherein the reward value is determined according to a preset reward and punishment strategy, and the preset reward and punishment strategy includes a reward and punishment mechanism associated with at least one of the target posture of the boom and the number of boom action sections.

[0076] Taking into account the characteristics of the DDPG algorithm: when the strategy is a deterministic strategy, the action-state value function Q(s, a) of the algorithm is only related to the environment. Therefore, a strategy associated with the recommended posture is designed in combination with the characteristics of the arm, and the corresponding penalty / incentive function is designed in combination with the environment to achieve the expected purpose of the algorithm.

[0077] The reward value is used to calculate the target value of the online Critic network. Optionally, in the preset reward and punishment strategy, the reward and punishment mechanism associated with the target posture of the boom includes rewarding or punishing according to the matching result between the posture of the boom and the target posture of the boom, and the reward and punishment mechanism associated with the number of action sections of the boom includes rewarding according to the number of action sections of the boom. In this way, the trained boom action model can better optimize the posture of the boom and the number of action sections, which conforms to the characteristics of the boom movement. In addition, the preset reward and punishment strategy may also include a reward and punishment mechanism associated with the target position of the end of the boom, so that the trained boom action model can better optimize the end position of the boom and make more accurate action recommendations.

[0078] Optionally, the preset reward and punishment strategy includes a reward and punishment mechanism associated with the target posture of the boom and the number of boom action sections, and also includes a reward and punishment mechanism associated with the target position of the end of the boom, and interacts with the environment through the Actor network to determine the second state reached by the boom taking the first action in the first state and the reward value obtained, including:

[0079] Interact with the environment through the Actor network to determine the second state reached by the arm taking the first action in the first state;

[0080] Determine the end position of the boom according to the second state, and obtain the distance value between the end position of the boom and the target position of the end of the boom; determine the posture of the boom according to the second state; determine the number of boom action sections according to the first action;

[0081] The reward value is determined according to the preset reward and punishment strategy, which is:

[0082] Roward=r+r1+r2+r3,

[0083] Among them, reward is the reward value; r is the initial value, which is the negative value of the distance value; r1 is the reward and punishment value associated with the arm end position. If the distance value is within the preset range, r1 is a positive value; if the distance value is not within the preset range, r1 is a negative value; r2 is the reward and punishment value associated with the arm posture. If the distance value is within the preset range and the arm posture is similar to the arm target posture, r2 is a positive value; if the distance value is within the preset range and the arm posture is not similar to the arm target posture, r2 is a negative value; r3 is the reward and punishment value associated with the number of arm action sections. When the distance value is within the preset range and the arm posture is similar to the arm target posture, r3 is determined according to the number of arm action sections and is a positive value.

[0084] The above interaction process is the process of interaction between the online Actor network and the environment. Optionally, when r2 is a positive value, r2>>r1; when r2 is a negative value, r1<r2<0. r3>>r2, so that when the distance value is within the preset range and the arm's posture is similar to the arm's target posture, no penalty is imposed, and r3 is used as an algorithm optimization item.

[0085] Optionally, the preset reward and punishment strategy also includes a reward and punishment mechanism associated with the target position of the end of the boom, interacting with the environment through the Actor network to determine the second state reached by the boom taking the first action in the first state and the reward value obtained, and also includes:

[0086] Reset the first state;

[0087] By interacting with the environment through the Actor network, the second state reached by the arm taking the first action in the first state and the reward value obtained are determined;

[0088] When the interaction ends, it is determined whether the posture of the arm and / or the end position of the arm meets the preset conditions;

[0089] If the preset conditions are met, the step of storing the first state, the first action, the reward value and the second state obtained by the interaction as training data in a training database is entered;

[0090] If the preset condition is not met, return to the step of resetting the first state.

[0091] Among them, the above-mentioned interaction process is the process of interaction between the online Actor network and the environment. At the beginning of an EPISODES, the first state is reset, and the online Actor network is used to interact with the environment to determine the second state reached by the boom taking the first action in the first state and the reward value obtained. Among them, the above-mentioned three conditions are used to obtain the reward value: ① The distance value between the end position of the boom and the target position of the end of the boom is within the preset range; ② The posture of the boom is similar to the target posture of the boom; ③ The number of boom action sections. Rewards or penalties are performed based on the above three conditions to obtain the reward value.

[0092] When the arm's posture and the arm's end position continuously meet conditions ① and ② for a preset number of times, the interaction ends and a set of state transition sequences (s i , a i , r i ,s i+1), and since the arm's posture and the arm's end position meet conditions ① and ②, the state transition sequence is stored in the training database. When the arm's posture and the arm's end position meet conditions ① and ② continuously for less than the preset number of times, and the number of search steps reaches the preset number of search steps, the interaction ends. The arm's posture and the arm's end position do not meet conditions ① and ② in this interaction, and the state transition sequence is not obtained. Therefore, the first state is reset to start a new round of search.

[0093] In the above manner, sufficient training data can be stored in the training database.

[0094] Optionally, in step S3, the first state, the first action, the reward value and the second state obtained by the interaction are used as training data and stored in a training database, including:

[0095] The first state, the first action, the reward value and the second state corresponding to one interaction are stored in a training database represented by a matrix in the form of a state transition sequence;

[0096] Determine whether the number of state transition sequences in the training database reaches a preset number;

[0097] If the preset number is reached, the step of training the Actor network and the Critic network based on the training data in the training database is entered;

[0098] If the preset number is not reached, return to the step of interacting with the environment through the Actor network.

[0099] When generating sample data, DDPG will use the data obtained from the environment as a state transition sequence (s i , a i , r i ,s i+1 ) is stored in a database represented by a matrix. The database capacity is a preset value (such as 50,000). When the database is full of data, the oldest data is deleted to ensure that the database always stores the latest 50,000 conversion sequence data. Each time the algorithm training is updated, the Actor network and the Critic network will randomly extract a part of the samples for optimization to increase the stability of the algorithm.

[0100] The data sampling process includes: randomly extracting BATCH_SIZE row numbers from the database; extracting the data; cutting the extracted data according to the saved data and restoring it into four groups of bs, ba, br, and bs for subsequent training.

[0101] Optionally, in step S4, based on the training data in the training database, the Actor network and the Critic network are trained to obtain a trained boom motion model, including:

[0102] Randomly extract training data from the training database to train the Actor network and Critic network;

[0103] Interact with the environment through the trained Actor network, and update the training data in the training database based on the interaction results;

[0104] Continue to train the Actor network and Critic based on the updated training data;

[0105] Return to the step of interacting with the environment through the trained Actor network until the training conditions are met and stop training to obtain the trained arm motion model.

[0106] The training methods of Actor network and Critic network refer to the previous article about Figure 2 During the training process, the trained Actor network interacts with the environment, and the training data in the training database is updated based on the interaction results. The Actor network and Critic are then trained based on the updated training data. This breaks the correlation of the training data and improves the stability of the neural network training process by drawing on the experience replay mechanism in the DQN algorithm. In addition, the sampling strategy of environmental interaction is continuously optimized to increase the accuracy of the model.

[0107] From the above, the boom motion model training method based on the DDPG algorithm of the present application creates an Actor network and a Critic network; interacts with the environment through the Actor network to determine the second state reached by the boom taking the first action in the first state and the reward value obtained, and the preset reward and punishment strategy used for the reward value includes a reward and punishment mechanism associated with at least one of the boom target posture, the boom end target position, and the number of boom action segments; the first state, the first action, the reward value, and the second state are stored in a training database; the Actor network and the Critic network are trained to obtain the trained boom motion model. In this way, the present application trains the boom motion model based on the DDPG algorithm, which can enable the model to be applied to recommend boom motions, and during training, a reward and punishment mechanism associated with motion parameters such as the boom posture, position, and number of action segments is used, so that the boom motion recommended by the model can meet the requirements of reasonable boom motion and / or reach the desired state.

[0108] Figure 3 FIG. 1 is a flow chart of a boom action recommendation method provided by another embodiment of the present invention. Figure 3As shown, the boom action recommendation method of the present application is applied to the boom action model trained by the method of the above embodiment, and the boom action recommendation method includes:

[0109] S10, obtaining the current state of the boom;

[0110] S20, inputting the current state of the boom into the boom action model for environmental interaction, and outputting a recommended action for the boom.

[0111] In this way, since the boom motion model is trained based on the DDPG algorithm, and a reward and punishment mechanism related to motion parameters such as the boom's posture, position, and number of action segments is used during training, the recommended boom motion can meet the requirements of reasonable boom movement and / or reach the desired state.

[0112] Optionally, the current state of the boom is input into the boom action model for environmental interaction, and the recommended action of the boom is output, including:

[0113] Input the current state of the boom into the boom action model to interact with the environment, determine the recommended boom action of the boom and the next state to be reached by taking the recommended boom action;

[0114] Determine whether the posture of the boom and the end position of the boom meet preset conditions in the first state;

[0115] If the preset condition is met, the number of times the preset condition is met continuously is increased by one;

[0116] If the number of times the preset condition is met continuously does not reach the preset number, the search continues and returns to the step of inputting the current state of the boom into the boom motion model for environmental interaction;

[0117] If the preset conditions are met consecutively for a preset number of times, the interaction ends and the recommended arm action is output.

[0118] Optionally, after determining whether the posture of the boom and the end position of the boom meet preset conditions in the first state, the method further includes:

[0119] If the preset conditions are not met, the number of search steps is increased by one;

[0120] If the number of search steps reaches the preset number of search steps, the interaction ends and a recommendation failure message is output.

[0121] In the process of recommending actions, the current state of the boom is input into the boom action model to determine the boom recommended action of the boom and the next state to be reached by taking the recommended action of the boom. If in the first state, the posture of the boom and the end position of the boom meet the preset conditions, for example, the preset conditions include that the distance value between the end position of the boom and the target position of the end of the boom is within a preset range, and the posture of the boom is similar to the target posture of the boom, then when the number of times the preset conditions are met continuously reaches the preset number, it indicates that the output of the model will be able to meet the needs of boom control, and the interaction can be ended at this time to output the recommended action of the boom. On the contrary, if the preset conditions are not met, the search step number is increased by one. If the search step number reaches the preset search step number, the interaction is ended and the recommendation failure information is output, indicating that there is no recommended action of the boom applicable to the current state.

[0122] The arm action recommendation method of the present application includes: obtaining the current state of the arm; inputting the current state of the arm into the arm action model for environmental interaction, and outputting the recommended arm action. In this way, since the arm action model is trained based on the DDPG algorithm, and a reward and punishment mechanism related to motion parameters such as the arm's posture, position, and number of action segments is used during training, the arm's recommended action can meet the requirements of reasonable arm movement and / or reaching the desired state.

[0123] The present application also provides an electronic device, including a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the computer program implements the steps of the method described in any of the above embodiments when executed by the processor.

[0124] The present application also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the method described in any of the above embodiments are implemented.

[0125] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed by the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the attached claims.

Claims

1. A boom motion model training method based on DDPG algorithm, characterized in that: The method comprises: Creating an initial boom motion model, wherein the boom motion model includes an Actor network and a Critic network; Determine, by interacting with the environment through the Actor network, a second state reached by the boom taking a first action in the first state and a reward value obtained, wherein the reward value is determined according to a preset reward and punishment strategy, and the preset reward and punishment strategy includes a reward and punishment mechanism associated with at least one of a target posture of the boom and the number of boom action sections; The first state, the first action, the reward value and the second state obtained by the interaction are used as training data and stored in a training database; Based on the training data in the training database, the Actor network and the Critic network are trained to obtain a trained boom motion model; The preset reward and punishment strategy includes a reward and punishment mechanism associated with the target posture of the boom and the number of boom action sections, and also includes a reward and punishment mechanism associated with the target position of the end of the boom. The interaction between the Actor network and the environment determines the second state reached by the boom taking the first action in the first state and the reward value obtained, including: Interacting with the environment through the Actor network to determine a second state reached by the boom taking a first action in the first state; Determine the end position of the boom according to the second state, and obtain the distance value between the end position of the boom and the target position of the end of the boom; determine the posture of the boom according to the second state; determine the number of action sections of the boom according to the first action; The reward value is determined according to the preset reward and punishment strategy, and the preset reward and punishment strategy is: in, reward is the reward value; r is the initial value; r 1 is the reward or penalty value associated with the boom end position, which is determined based on whether the distance between the boom end position and the boom end target position is within a preset range; r 2 is the reward and penalty value associated with the boom posture, which is determined based on whether the boom posture is similar to the boom target posture; r 3 is a reward or penalty value associated with the number of boom action sections, which is determined according to the number of boom action sections when the distance value is within a preset range and the posture of the boom is similar to the boom target posture.

2. The method according to claim 1, characterized in that r The value of is the negative value of the distance value; if the distance value is within the preset range, then r 1 is a positive value. If the distance value is not within the preset range, then r 1 is a negative value; if the distance value is within the preset range and the arm's posture is similar to the arm's target posture, then r 2 is a positive value. If the distance value is within the preset range and the arm's posture is not similar to the arm's target posture, then r 2 is a negative value; r 3 is a positive value.

3. The method according to any one of claims 1 to 2, characterized in that The preset reward and punishment strategy also includes a reward and punishment mechanism associated with the target position of the end of the boom, and the interaction between the Actor network and the environment to determine the second state reached by the boom taking the first action in the first state and the reward value obtained include: resetting the first state; By interacting with the environment through the Actor network, determining a second state reached by the arm frame taking a first action in the first state and a reward value obtained; When the interaction ends, it is determined whether the posture of the arm and the end position of the arm meet preset conditions; If the preset conditions are met, the step of storing the first state, the first action, the reward value and the second state obtained by the interaction as training data in a training database is entered; If the preset condition is not met, return to the step of resetting the first state.

4. The method according to any one of claims 1 to 2, characterized in that The step of training the Actor network and the Critic network based on the training data in the training database to obtain a trained boom motion model includes: Randomly extract training data from the training database to train the Actor network and the Critic network; Interacting with the environment through the trained Actor network, and updating the training data in the training database according to the interaction results; Continue to train the Actor network and the Critic based on the updated training data; Return to the step of interacting with the environment through the trained Actor network until the training condition is met and stop the training to obtain the trained arm motion model.

5. A boom action recommendation method, characterized in that: Applied to a boom action model trained by the method according to any one of claims 1 to 3, the boom action recommendation method comprises: Get the current state of the arm; The current state of the boom is input into the boom action model for environmental interaction, and a recommended action of the boom is output.

6. The method according to claim 5, characterized in that The step of inputting the current state of the boom into the boom action model to interact with the environment and outputting a recommended action of the boom includes: Inputting the current state of the boom into the boom action model to interact with the environment, determining a recommended boom action of the boom and a next state to be reached by taking the recommended boom action; Determining whether the posture of the boom and the end position of the boom meet preset conditions in the first state; If the preset condition is met, the number of times the preset condition is met continuously is increased by one; If the number of times the preset condition is continuously satisfied does not reach the preset number, continue searching and return to the step of inputting the current state of the boom into the boom motion model for environmental interaction; If the preset conditions are met consecutively for a preset number of times, the interaction is terminated and the recommended arm action is output.

7. The method according to claim 6, characterized in that After determining whether the posture of the boom and / or the end position of the boom meets a preset condition in the first state, the method further includes: If the preset conditions are not met, the number of search steps is increased by one; If the number of search steps reaches the preset number of search steps, the interaction ends and a recommendation failure message is output.

8. An electronic device, characterized in that: The method comprises a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the computer program implements the steps of the method according to any one of claims 1 to 7 when executed by the processor.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images