Mechanical arm throwing method, throwing system, electronic device and computer storage medium

Abstract

This invention discloses a robotic arm throwing method, a throwing system, an electronic device, and a computer storage medium. The robotic arm throwing method includes: constructing a neural network and a priority experience replay pool through a virtual simulation environment; the neural network outputs an action based on the current state space of the robotic arm, and the robotic arm obtains the next state space after executing the action; a parabolic motion prediction model predicts the predicted landing position of the object based on the next state space; calculating the predicted total reward function based on the error between the predicted landing position and the future position of the dynamic target; calculating the landing total reward function after the object is thrown and lands; storing the state transition samples generated during the interaction process into the priority experience replay pool, and training the neural network for multiple rounds based on the predicted total reward function or the landing total reward function to obtain a throwing control strategy with faster convergence speed.

Description

Technical Field

[0001] The invention relates to the field of industrial robotic arm control technology, specifically a robotic arm throwing method, throwing system, electronic device, and computer storage medium. Background Technology

[0002] With the rapid development of industrial automation technology, robotic arms have been widely used in industrial assembly, logistics sorting, and assembly line handling, playing a vital role in reducing labor costs and improving production efficiency. Traditional robotic arm control methods require precise modeling of the robotic arm's own dynamics, target state, and environmental parameters, as well as planning of control strategies. These methods are mainly suitable for applications with stable environments and relatively fixed target positions, where high operational efficiency can be achieved.

[0003] However, in actual industrial production, industrial assembly line operations such as logistics sorting often exhibit significant dynamic characteristics, with target objects constantly moving along with the conveyor. When the target's position or motion state changes, traditional control methods based on precise modeling have limited adaptability, easily leading to a decrease in control accuracy and failing to meet the demands of high-efficiency, flexible, and dynamic operations. Simultaneously, existing robotic arms mostly employ a follow-and-place method, which, limited by the robotic arm's movement speed and workspace, struggles to balance operational efficiency and control accuracy under dynamic target conditions. To overcome these limitations, throwing-type operations, as a rapid object transfer method that eliminates the need for a robotic arm to reach the target location, are gradually gaining attention.

[0004] In recent years, with the development of artificial intelligence, deep reinforcement learning has emerged and been applied to the field of robot control, such as enabling robots to learn skills like grasping, obstacle avoidance, shaft and hole assembly, and throwing. Through trial and error learning between the agent and the environment, reward values ​​serve as feedback information from the environment, guiding the robot to learn an optimal strategy. However, reinforcement learning algorithms suffer from problems such as slow convergence speed, low learning efficiency, and the need for a large amount of sample data for robot-environment interaction. Summary of the Invention

[0005] This invention aims to address at least one of the technical problems existing in the prior art. To this end, this invention proposes a robotic arm throwing method. Under the constraints of a parabolic motion prediction model and a two-stage total reward function, and combined with a priority experience replay mechanism, the robotic arm is controlled to perform multi-round interactive training in a virtual simulation environment to execute dynamic target throwing tasks, thereby quickly learning a stable control strategy that satisfies the dynamic target throwing constraints.

[0006] The present invention also proposes a throwing system.

[0007] The present invention further proposes an electronic device.

[0008] The present invention further proposes a computer-storable medium.

[0009] According to a first aspect of the present invention, a robotic arm throwing method is provided, wherein the robotic arm includes: a robotic arm body and an end effector, the method comprising: S1: constructing a neural network based on the TD3 algorithm and a priority experience replay pool through a virtual simulation environment; S2: obtaining the current state space of the robotic arm. The policy network of the neural network uses the current state space. Input, output action The robotic arm performs the action The next state space is then obtained. S3: Establish a parabolic motion prediction model, which is based on the next state space. S4: Predict the predicted landing point of the object at a specified height; S5: Obtain the error between the predicted landing point of the thrown object and the future position of the dynamic target based on the predicted landing point, and calculate the total prediction reward function based on the error. When the error is less than a preset threshold, the end effector is triggered to release the object and a positive reward is given; S5: After the object falls to a specified height, the error between the actual landing position and the future position of the dynamic target is calculated; the total landing reward function is calculated based on this error. S6: Store the state transition samples generated during the interaction into the priority experience replay experience pool, and train the neural network based on the predicted total reward function or the landing total reward function; repeat steps S1 to S5 for multiple rounds of interactive training until a converged throwing control strategy is obtained; S7: Deploy the trained converged strategy network parameters to the physical robotic arm and control the robotic arm to perform dynamic target throwing tasks.

[0010] According to some embodiments of the present invention, in step S1, the neural network includes an online network and a target network, the online network including an online policy network and an online value network, the online policy network being based on the current state space of the robotic arm. The output is used to control the movements of the robotic arm. The action includes joint angle increments and joint angular velocity increments, and the online value network combines the current state space. Action An evaluation is performed; the target network is used to update the online network parameters with delay or smoothness, and to generate target actions and target value references.

[0011] According to some embodiments of the present invention, in step S2, the state space includes the angles and angular velocities of each joint of the robotic arm, the pose and velocity of the end effector, and the current position and velocity of the dynamic target.

[0012] According to some embodiments of the present invention, in step S3, the projectile motion prediction model is as follows: based on the initial position and initial velocity of the object at the moment of throwing, and ignoring interference such as air resistance, a unique landing point can be determined through projectile motion. According to the formula for projectile motion:

[0013] , in, , These are the initial position coordinates and initial velocity of the object in the x, y, and z directions at the instant of its throw; And, when the object falls to a specified height hour, The time t for the object to fall to the specified height is determined by the equation. We obtain the predicted landing positions (x_pre, y_pre) of the thrown object in the x and y axes based on time t.

[0014] According to some embodiments of the present invention, in step S4, the predicted total reward function The formula is: , in, The maximum error threshold between the predicted landing point of a thrown object and the future position of a dynamic target. The error between the predicted landing point of a thrown object and the future position of the target. It is the first reward adjustment constant and It is a real number greater than 1.

[0015] According to some embodiments of the present invention, in step S5, the total landing reward function The formula is: , in, Let n be the maximum error threshold between the actual landing point of the thrown object and the future position of the dynamic target, where n is equal to a. It is a weighting coefficient and is set to a real number greater than 1. This represents the error between the actual landing point of the thrown object and the future position of the target. It is the second reward adjustment constant and Greater than .

[0016] Beneficial effects: To address the dynamic target throwing requirements in industrial assembly line operations such as logistics sorting, this paper proposes a robotic arm throwing method that combines a parabolic motion model with TD3 deep reinforcement learning, incorporating a priority experience replay mechanism. This method boasts faster convergence and learning efficiency. Through interactive trial and error between the agent and the environment, the robot autonomously learns throwing timing and control strategies, achieving high-precision throwing of dynamic targets even when the target position is constantly changing. This allows for throwing operations to be completed without real-time human intervention, significantly improving the automation level and overall operational efficiency of assembly line operations. Compared to traditional control methods that rely on precise modeling and manual parameter tuning, this invention eliminates the need for complex dynamic models, exhibiting stronger adaptability and robustness. Even with minor perturbations in target velocity, it maintains stable throwing accuracy. Furthermore, deep reinforcement learning, through large-scale training, can autonomously learn more energy-efficient throwing strategies, helping to reduce the overall energy consumption of the robotic arm performing throwing tasks, demonstrating significant engineering application value.

[0017] According to a second aspect of the present invention, a throwing system includes: a robotic arm, comprising: a robotic arm body and an end effector, the robotic arm body being used to perform a throwing action, and the end effector being mounted at the end of the robotic arm body for gripping and releasing a thrown object; a control unit being used to execute the dynamic target throwing method based on deep reinforcement learning and to generate corresponding robotic arm control commands according to environmental state information; a power supply unit being electrically connected to the robotic arm for providing power support to the robotic arm; and a conveying device being used to carry a target container and drive its movement to form a dynamic target.

[0018] An electronic device according to a third aspect of the present invention is characterized in that it includes at least one processor coupled to a memory storing a computer program configured to be executed by the processor to perform the method.

[0019] According to a fourth aspect of the present invention, a computer storage medium thereon stores a computer program for performing the method.

[0020] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0021] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the description of the embodiments taken in conjunction with the following drawings, in which: Figure 1 This is a flowchart illustrating a throwing round of a robotic arm throwing method according to an embodiment of the present invention. Figure 2This is a schematic diagram of a simulation environment established by the robotic arm throwing method according to an embodiment of the present invention; Figure 3 This is a schematic diagram of the parabolic motion at the moment of throwing using a robotic arm throwing method according to an embodiment of the present invention. Figure 4 This is an algorithm block diagram of the robotic arm throwing method based on the TD3 algorithm according to an embodiment of the present invention. Detailed Implementation

[0022] The embodiments of the present invention are described in detail below. The embodiments described with reference to the accompanying drawings are exemplary. The embodiments of the present invention are described in detail below.

[0023] The following is for reference. Figures 1-4 A robotic arm throwing method according to embodiments of the present invention is described, as well as a throwing system including the above method, an electronic device including the above method, and a computer storage medium including the above method.

[0024] The robotic arm throwing method includes the following steps: S1: Construct a neural network based on the TD3 algorithm and a priority experience replay pool through a virtual simulation environment.

[0025] For example, a simulation environment can be built using CoppeliaSim simulation software, which facilitates the subsequent acquisition of the state space for a six-axis robotic arm's dynamic target throwing task.

[0026] In some embodiments, the simulation environment is as follows: Figure 2 As shown. Among them, For the main body of the robotic arm, For end effector, For throwing objects, For control unit, For dynamic objectives, It is a transmission device.

[0027] Furthermore, the neural network is built using the TD3 algorithm. The neural network comprises six networks in total, including online networks and target networks. The online networks include an online policy network and two online value networks: the online Q1 network and the online Q2 network. The target networks include a target policy network and two target value networks: the target Q1 network and the target Q2 network.

[0028] Online policy network, represented as ,in, It is the current state space. These are the network parameters of an online policy network; an online Q1 network, denoted as... Online Q2 network, represented as ;in, It is the current state space. This represents the current motion of the robotic arm. ω1 and ω2 are the network parameters of the online Q1 network and the online Q2 network, respectively.

[0029] The target policy network is represented as The target Q1 network is represented as: The target Q2 network is represented as: . For the network parameters of the target policy network, , These are the network parameters for target Q1 network and target Q2 network, respectively.

[0030] Among them, network parameters are a set of scientific coefficient values ​​used to describe the "input-output" mapping relationship within each network, including the weights and biases of each layer, and record the behavioral patterns learned by the agent through real-time interaction with the environment.

[0031] Both the online Q1 network and the online Q2 network are evaluation networks used to evaluate the throwing action output by the online policy network, taking into account the current state space of the robotic arm, while the target is in continuous motion. Conduct joint assessments, assessment actions The value of the throwing effect in the current state is used to reduce the bias in the throwing effect evaluation caused by dynamic changes in the target position and improve the reliability of throwing decisions.

[0032] The target network is used to delay or smooth the updates of online network parameters and generate target actions and target value references.

[0033] The priority experience replay pool stores state transition samples of the robotic arm during the throwing preparation, object release, and throwing result feedback processes. These state transition samples are stored in quintuple form, representing the relationship between the robotic arm's control actions and the throwing effect. The throwing success flag indicates whether the thrown object actually lands on the dynamic target. When the thrown object actually lands on the dynamic target, the interaction is marked as a successful completion of the throwing task, denoted as . .

[0034] State transition samples are denoted as .in, This represents the state of the robotic arm at time step t. This indicates the action taken by the robotic arm in the given state. This represents the reward value obtained after performing an action. This indicates the next state obtained after performing an action. This indicates whether the thrown object actually lands on the dynamic target after the action is performed at time step t.

[0035] When the thrown object actually lands on or is predicted to land on the dynamic target based on its parabolic motion at the moment of throwing, the corresponding state transition sample is given an additional positive reward, which makes it exhibit a larger temporal difference error in value assessment and obtain a higher sampling priority under the priority experience replay mechanism. This alleviates the problem of low success rate in dynamic target throwing tasks, accelerates policy convergence, and improves the throwing success rate.

[0036] S2: Obtain the current state space of the robotic arm. The policy network of the neural network uses the current state space. Input, output action The robotic arm performs the action The next state space is then obtained. .

[0037] The state space of a robotic arm when performing a dynamic target throwing task This includes the joint angles of the six joints of the robotic arm, the joint angular velocities of the six joints, the current pose of the end effector in the x, y, and z directions, the current velocity of the end effector in the x, y, and z directions, and the current position and velocity of the dynamic target in the x, y, and z directions.

[0038] Each state space Represented as:

[0039] in, , These are the angle and angular velocity values ​​for the six joints, respectively. , , These represent the current position, attitude, and current velocity of the end effector in the x, y, and z directions, respectively. , These represent the current position and current velocity of the dynamic target object in the x, y, and z directions, respectively.

[0040] Since this invention focuses solely on throwing tasks, the pre-throw pose is linked to the end effector. That is, the pose and velocity of the end effector at the time of throwing are the initial pose and velocity of the thrown object during its flight through the air.

[0041] A state space of the robotic arm As input to the online policy network, the output is the joint angle increment used to control the robotic arm's throwing motion as the action. The robotic arm performs the action The next state space is then obtained. Through the next state space The pose and velocity of the end effector of the robotic arm are used to indirectly determine the initial state of the thrown object at the moment of release, and to achieve adaptive optimization of the throwing trajectory and the effect of hitting dynamic targets.

[0042] S3: Establish a parabolic motion prediction model, which is based on the next state space. Predict the predicted landing point of an object at a specified height.

[0043] Each time the neural network outputs an action A projectile motion prediction model is established in real time. Specifically, the projectile motion prediction model is described as follows: given the initial position and initial velocity of the object to be thrown by the robotic arm at the moment of release, and ignoring interference such as air resistance, a unique landing point can be determined through projectile motion. The formula for projectile motion is shown below: .

[0044] in, , These are the initial position coordinates and initial velocity of the object in the x, y, and z directions at the instant of throwing. Since the object's position and velocity are bound to the end effector, the initial position and velocity at the instant of throwing... = ; = g is the acceleration due to gravity, taken as 9.81 m / s²; When the thrown object falls to the designated height hour, The time t taken for the thrown object to fall to the designated height is calculated using the following quadratic equation: .

[0045] When the discriminant of the quadratic equation If it is determined that the thrown object cannot reach the specified height under the current initial conditions, the prediction process is terminated. When the discriminant of the quadratic equation Calculate the time t it takes for the object to fall to a specified height, and calculate the predicted landing position (x_pre, y_pre) of the thrown object in the x and y axis directions based on the time t.

[0046] Predicting the landing point as an intermediate evaluation metric in the pre-throw decision-making stage allows for a forward-looking judgment of the potential throwing result corresponding to the current joint motion state of the robotic arm before the object is released. This enables the reinforcement learning strategy to obtain physical constraint information related to the throwing result before the release action occurs. By introducing the prediction of the landing point based on parabolic motion, the throwing task is transformed from a delayed feedback problem that traditionally relies on the final landing result into a continuous decision-making problem with the ability to evaluate and guide in advance.

[0047] S4: Obtain the error between the predicted landing position of the thrown object and the future position of the dynamic target based on the predicted landing position; calculate the total reward function based on the error. When the error is less than a preset threshold, the end effector is triggered to release the object and give a positive reward.

[0048] A dynamic target is a container whose position is constantly changing. The robotic arm needs to throw objects onto the moving dynamic target.

[0049] The predicted landing point of a projectile is calculated based on projectile motion, using the following formula: .

[0050] in: This represents a mapping relationship for predicting the trajectory of a thrown object based on the physical constraints of projectile motion. , representing the initial position and initial velocity of the object at the moment of throwing, respectively; and t, representing the time taken for the thrown object to reach the specified height. This indicates the predicted landing point of the thrown object along the x and y axes.

[0051] Formula for the future position of a dynamic target: .

[0052] in, For the future position of the dynamic target, The current position of the dynamic target. The current velocity of the dynamic target. The time it takes for a thrown object to fall to a designated height; Predicting the total reward function This method is used to evaluate the robotic arm's movements before throwing, guiding the current movement trend of the robotic arm using parabolic motion prediction results before the thrown object enters the free flight phase. The landing position can be predicted through parabolic motion, and the distance difference between the predicted landing position and the future position of the target affects the predicted total reward function, which in turn affects the learning of the neural network. This adjusts the joint angles, joint angular velocities, and the current pose and speed of the gripper of the robotic arm, avoiding the problems of reward sparsity and slow policy convergence caused by relying solely on the termination reward after the throwing is completed. The predicted total reward function includes the predicted reward function and the predicted hit reward function.

[0053] Predicting the total reward function Including the predictive reward function and prediction hit reward function .

[0054] Among them, the prediction reward function This is related to the error between the predicted landing point of the projectile and the future position of the dynamic target. The future position of the target is calculated based on its current position and velocity, as well as the time t it takes for the object to fall to a specified height during parabolic motion; simultaneously, the predicted landing point is calculated based on the current position and velocity of the projectile.

[0055] Predicting reward function The formula is: .

[0056] in, The error between the predicted landing point of a thrown object and the future position of the target.

[0057] Predicted hit reward function The description is as follows: when the error between the predicted landing point and the future target position is less than a preset threshold, it is determined that the target has been predicted to be hit. The end effector is then triggered to release the object and give a positive reward, so as to guide the strategy to find the future target position faster before the object is released. The release action is not a preset throwing time, but is triggered autonomously by the reinforcement learning strategy when the prediction hit condition is met, so that the release timing is coupled with the future position of the dynamic target and the current motion state of the robotic arm.

[0058] Predicted hit reward function The formula is: .

[0059] in, It is the first reward adjustment constant and Set to a positive real number greater than 1. The maximum error threshold between the predicted landing point of the projectile and the future position of the dynamic target is set to ensure the accuracy of the prediction reward function. It is a positive reward function.

[0060] Predicting the total reward function for: .

[0061] S5: After the object falls to the specified height, calculate the error between the actual landing position and the future position of the dynamic target; calculate the total landing reward function based on the error. .

[0062] Total reward function upon landing Including landing reward function and landing hit reward function .

[0063] Total reward function upon landing Used to terminate the evaluation of the landing result after the throw during the landing phase, serving as the final constraint for the throwing task, and used to correct and verify the guidance results during the prediction phase, thereby preventing the strategy from relying solely on prediction errors and ignoring the actual landing deviation. This includes the landing reward function and the landing hit reward function.

[0064] Among them, the landing reward function It is related to the error between the actual landing point of the projectile and the future position of the dynamic target.

[0065] Landing reward function The formula is: .

[0066] in, This represents the error between the actual landing point of the thrown object and the future position of the target.

[0067] Landing hit reward function The description is as follows: when the error between the actual landing point and the future target position is less than a preset threshold, it is determined as a successful landing hit on the target. The landing hit reward is set to a positive number that is greater than the predicted hit reward, thereby increasing the constraint and guidance on the actual hit result.

[0068] Landing hit reward function The formula is: .

[0069] in, It is the second reward adjustment constant and Greater than This ensures that the robotic arm is more focused on actually hitting the target. This is the maximum error threshold between the actual landing point of the projectile and the future position of the dynamic target.

[0070] Total reward function upon landing The formula is: .

[0071] in, This is the maximum error threshold between the actual landing point of the thrown object and the future position of the target. Numerically equal to , It adjusts the weighting coefficient of the landing reward and sets it to a positive real number greater than 1, ensuring that the total landing reward is greater than the predicted total reward, thereby increasing the constraint and guidance on the actual throwing results.

[0072] By forming a two-stage reward function through the prediction total reward function and the landing total reward function, the prediction total reward function and the landing total reward function evaluate and guide the actions of the robotic arm in the pre-throw stage and the landing stage, respectively. In the presence of environmental disturbances such as air resistance, the robotic arm can be guided to balance the prediction accuracy before throwing and the actual hit effect after throwing, thereby improving the robustness and success rate of dynamic target throwing tasks.

[0073] S6: Store the state transition samples generated during the interaction into the priority experience replay experience pool, and train the neural network based on the predicted total reward function or the landing total reward function; repeat steps S1 to S5 for multiple rounds of interactive training until a convergent throwing control strategy is obtained.

[0074] Under the constraints of a parabolic motion prediction model and a two-stage reward function, the robotic arm is trained through multiple rounds of interactive training in a virtual simulation environment to perform dynamic target throwing tasks, so as to learn a stable control strategy that satisfies the dynamic target throwing constraints.

[0075] Specifically as follows: Before training begins, initialize the neural network parameters and initialize the priority experience replay pool M. The workflow for a single throwing round is described below: Step a: Initialize the joint states of the robotic arm and obtain the current state information of the dynamic target.

[0076] In each training round, the real-time position and velocity information of the target container moving with the conveyor device are acquired, and the joint state of the robotic arm is initialized to complete the pre-training preparation for a single throwing task.

[0077] Step b: Based on the state space of the robotic arm and the current state information of the dynamic target, the neural network outputs the joint angle increments to control the robotic arm.

[0078] Current state space The input is fed into an online policy network, which outputs the joint angle increments for the throwing task. .

[0079] Step c: Predict whether the future position of the dynamic target has been hit or the maximum number of steps in the current round has been reached based on the parabolic motion prediction model. If yes, proceed to step d; otherwise, return to step b.

[0080] robotic arm performs actions Then obtain the next state ,according to Based on the projectile motion model of the thrown object, the system predicts the landing position of the thrown object at a specified height, and calculates the predicted total reward by combining this with the future position of the dynamic target. It is used to guide the robotic arm to quickly hit a dynamic target before the object is released.

[0081] Step d: The end effector of the robotic arm releases the object.

[0082] If the robotic arm predicts that it can hit a dynamic target, the end effector releases the object, or the robotic arm releases the object when it reaches the maximum allowed number of steps in this round.

[0083] Step e: Determine whether the throw actually hits the future position of the dynamic target after landing at the specified height. If it does, the throw is successful and the throwing round ends; otherwise, the throw fails and the throwing round ends.

[0084] The thrown object undergoes a parabolic motion, and upon landing, the total landing reward is calculated based on the deviation between its actual landing point and the future position of the dynamic target, thus terminating the evaluation of the throwing result.

[0085] Based on the predicted total reward Or total landing reward Determine the total reward for the current stage This guides the robotic arm to gradually develop a control strategy that meets the requirements for throwing dynamic targets. It then determines whether the throwing task is complete; when the thrown object actually lands on the dynamic target, the interaction is marked as a successful completion of the throwing task, recorded as... .

[0086] The quintuple of state transition samples generated during the above interaction process The samples are stored in the priority experience replay pool, where samples related to successful dynamic target throwing or prediction have a large temporal difference error in value assessment, thus obtaining a higher sampling priority in subsequent sampling processes.

[0087] During the network training phase, state transition samples are sampled from the priority experience replay pool. The throwing-related state-action pairs are jointly evaluated based on the dual-value network, and the control strategy is optimized using a delayed update policy network. This allows the parameter update process to focus on learning the effective throwing timing and key joint actions, thereby improving the policy convergence stability and throwing success rate under the condition of continuous motion of dynamic targets.

[0088] Repeat the above process for multiple rounds of training until the success rate and reward value of the dynamic target throwing task tend to stabilize, and obtain the convergence strategy for actual robotic arm control.

[0089] S7: Deploy the converged policy network parameters to the physical robotic arm and control the robotic arm to perform dynamic target throwing tasks.

[0090] Specifically, by constructing a dynamic target throwing control closed loop for a physical robotic arm, the robotic arm state and dynamic target motion information obtained from visual perception are uniformly mapped to state inputs consistent with simulation training, and the throwing control commands are generated by the online policy neural network that has converged during training. Before the commands are executed, the throwing trajectory is forward-checked based on parabolic motion prediction, and the control commands are constrained and corrected or filtered according to the deviation between the predicted landing point and the future position of the target. The throwing execution and effect verification are completed under the condition of satisfying the robotic arm motion constraints, thereby realizing the stable deployment of reinforcement learning throwing strategies in real dynamic target scenarios.

[0091] Another aspect of the present invention provides a throwing system, comprising: a robotic arm, a control unit, a power supply unit, and a conveying device. The robotic arm includes: a robotic arm body and an end effector. The robotic arm body is used to perform the throwing action, and the end effector is mounted at the end of the robotic arm body for gripping and releasing the thrown object. The control unit is used to execute the above-described method for dynamic target throwing based on deep reinforcement learning and to generate corresponding robotic arm control commands based on environmental state information. The power supply unit is electrically connected to the robotic arm and is used to provide power support to the robotic arm. The conveying device is used to carry the target container and drive its movement to form a dynamic target.

[0092] The conveying device is equipped with a target container, which moves synchronously with the conveying device to make the target container a dynamic target.

[0093] An electronic device according to a third aspect of the present invention includes a memory and a processor, as well as computer instructions stored in the memory and running on the processor. When executed by the processor, the computer instructions perform the method described in the first aspect embodiment. For brevity, further details are omitted here.

[0094] It should be understood that in this embodiment, the processor can be a central processing unit (CPU), or it can be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc.

[0095] Memory may include read-only memory and random access memory, and provides instructions and data to the processor. A portion of memory may also include non-volatile random access memory. For example, memory may also store information about the device type.

[0096] A computer storage medium according to a fourth aspect of the present invention is used to store computer instructions, which, when executed by a processor, perform the method described in the first aspect embodiment.

[0097] The method in the first aspect embodiment can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules in the processor. The software modules can reside in readily available storage media in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. This storage medium is located in memory, and the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above method. To avoid repetition, a detailed description is not provided here.

[0098] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "illustrative embodiment," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example.

[0099] Although embodiments of the invention have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the claims and their equivalents.

Claims

1. A method for throwing with a robotic arm, the robotic arm comprising: A robotic arm body and an end effector, characterized in that the method includes: S1: Construct a neural network based on the TD3 algorithm and a priority experience replay pool through a virtual simulation environment; S2: Obtain the current state space of the robotic arm. The policy network of the neural network uses the current state space. Input, output action The robotic arm performs the action The next state space is then obtained. ; S3: Establish a parabolic motion prediction model, which is based on the next state space. Predict the predicted landing point of an object at a specified height; S4: Obtain the error between the predicted landing position of the thrown object and the future position of the dynamic target based on the predicted landing position, and calculate the total prediction reward function based on this error. When the error is less than a preset threshold, the end effector is triggered to release the object and a positive reward is given. S5: After the object falls to the specified height, calculate the error between the actual landing point and the future position of the dynamic target; calculate the total landing reward function based on this error. ; S6: Store the state transition samples generated during the interaction into the priority experience replay experience pool, and train the neural network based on the predicted total reward function or the landing total reward function; repeat steps S1 to S5 for multiple rounds of interactive training until a converged throwing control strategy is obtained. S7: Deploy the converged policy network parameters to the physical robotic arm and control the robotic arm to perform dynamic target throwing tasks.

2. The robotic arm throwing method according to claim 1, characterized in that, In step S1, the neural network includes an online network and a target network. The online network includes an online policy network and an online value network. The online policy network uses the current state space of the robotic arm. The input is used to control the movements of the robotic arm. The action includes joint angle increments and joint angular velocity increments, and the online value network combines the current state space. Action Conduct an assessment; The target network is used to delay or smooth the updates of the online network parameters, and to generate target actions and target value references.

3. The robotic arm throwing method according to claim 1, characterized in that, In step S2, the state space includes the angles and angular velocities of each joint of the robotic arm, the pose and velocity of the end effector, and the current position and velocity of the dynamic target.

4. The robotic arm throwing method according to claim 1, characterized in that, In step S3, the parabolic motion prediction model is established based on the equation of parabolic motion, which is: ， in, , These are the initial position coordinates and initial velocity of the object in the x, y, and z directions at the instant of its throw, respectively, where g is the gravitational acceleration. And, when the object falls to a specified height hour, The time t for the object to fall to the specified height is determined by the equation. We obtain the predicted landing point of the thrown object in the x and y axes based on time t. x_pre , y_pre ).

5. The robotic arm throwing method according to claim 1, characterized in that, In step S4, the predicted total reward function The formula is: ， in, The maximum error threshold between the predicted landing point of a thrown object and the future position of a dynamic target. The error between the predicted landing point of a thrown object and the future position of the target. It is the first reward adjustment constant and It is a real number greater than 1.

6. The robotic arm throwing method according to claim 5, characterized in that, In step S5, the total landing reward function The formula is: ， in, This is the maximum error threshold between the actual landing point of the thrown object and the future position of the dynamic target. It is a weighting coefficient and is set to a real number greater than 1. This represents the error between the actual landing point of the thrown object and the future position of the target. It is the second reward adjustment constant and Greater than .

7. A throwing system, characterized in that, include: A robotic arm includes: a robotic arm body and an end effector, wherein the robotic arm body is used to perform a throwing action, and the end effector is installed at the end of the robotic arm body for gripping and releasing the thrown object; The control unit is used to execute the dynamic target throwing method based on deep reinforcement learning as described in any one of claims 1-6, and to generate corresponding robotic arm control commands based on environmental state information. A power supply unit, electrically connected to the robotic arm, is used to provide power support for the robotic arm; A conveying device is used to carry a target container and drive its movement to form a dynamic target.

8. An electronic device, characterized in that, The device includes at least one processor coupled to a memory containing a computer program configured to be executed by the processor to perform the method of any one of claims 1-6.

9. A computer storage medium, characterized in that, It contains a computer program for performing the method described in any one of claims 1-6.

Images