Track planning method and track planning device for collision avoidance of equipotential operation mechanical arm

Abstract

The invention provides a trajectory planning method and device for collision avoidance of an equipotential operation mechanical arm, and the method comprises the steps: determining a state space, an action space and a reward function of the mechanical arm, the reward function is a function which considers the safe operation space of the mechanical arm under the hot-line work and the demand of the mechanical arm for avoiding the dangerous space on the basis of ensuring that the tail end of the mechanical arm moves to the target point position; based on the state space, the action space and the reward function, utilizing a depth deterministic strategy gradient algorithm to carry out iterative training to obtain a target strategy model; the target point position of the mechanical arm is obtained, the target point position is input into the target strategy model, and the optimal moving path of the mechanical arm is obtained; and the mechanical arm is controlled to move to the target point position according to the optimal moving path. The problem that the mechanical arm is difficult to avoid the dangerous space through a traditional method is solved.

Description

Technical Field

[0001] The present invention relates to the technical field of robot arm trajectory planning, and in particular to a trajectory planning method, a trajectory planning device, a computer-readable storage medium and a computer program product for collision avoidance of an equipotential working robot arm. Background Art

[0002] The safe and stable operation of the power system is related to the national economy and people's livelihood, and the autonomy and intelligence level of power robots are constantly increasing, playing an important role in ensuring the safe and stable operation of power equipment. In recent years, a number of key technologies have been broken through, and a series of prototypes have been developed. Pilot applications have been made in the fields of power generation, transmission, substation inspection robots, transmission line inspection drones, etc. have been widely promoted and applied. The autonomy and intelligence level of power robots are constantly increasing, and equipotential operation robots have played an important role in ensuring the safe and stable operation of power equipment.

[0003] As a widely used automated mechanical device, the robotic arm assists humans in performing many dangerous tasks. However, in most cases, the robotic arm is operated by humans, that is, the robot's motion trajectory is manually planned to play a role of teaching supervision [1]. This limits the distance between humans and the robotic arm to a certain range. Therefore, traditional robotic arms either rely on humans and work together with humans; or their trajectory planning follows a fixed and structured process. Therefore, in some dangerous, complex, and changeable environments, the trajectory planning of traditional robotic arms is easily affected by the changing environment, and its trajectory may deviate, which in turn causes serious consequences. Therefore, there is an urgent need to carry out research on autonomous robotic arms.

[0004] In recent years, domestic and foreign scholars have done a lot of research in the field of robot obstacle avoidance motion planning. In the prior art, there is an optimal learning generalization mechanism for obstacle avoidance motion planning, and the robot's intelligence level and autonomous operation ability are improved through a teaching learning system. However, the disadvantage of this method is that it is too dependent on the quality and collection method of collected samples. There is also an artificial potential field method for obstacle avoidance path planning for the robot, and virtual obstacles are added near the nearest point to solve the local extreme value problem. However, its disadvantage is that it lacks generalization and is difficult to adapt to complex environments. There is also an improved rapid expansion random tree algorithm for motion planning of dual robot arms. In addition to considering static obstacles, the motion posture of the main robot arm at each moment is regarded as a dynamic obstacle, so as to plan a feasible path for the slave robot arm. However, this method has the problems of large computational complexity and slow convergence speed. It is also proposed to combine the sampling-based path planning method with the flexible obstacle avoidance method for motion planning, change the posture of the obstacle during the planning process, and finally find a successful obstacle avoidance path. However, this method has safety hazards in that the robot arm comes into contact with the obstacle during the planning process. The above algorithm relies on real-time modeling of the robot and the environment, and its computational complexity increases exponentially as the robot's degrees of freedom increase. Therefore, in some dangerous, complex, and changeable environments, the trajectory planning of traditional robot arms is easily affected by the changing environment, and its trajectory may deviate, resulting in serious consequences. Summary of the invention

[0005] The main purpose of the present application is to provide a trajectory planning method, a trajectory planning device, a computer-readable storage medium and a computer program product for collision avoidance of an equipotential working robot arm, so as to at least solve the problem in the prior art that it is difficult to control the robot arm to avoid dangerous spaces.

[0006] In order to achieve the above-mentioned purpose, according to one aspect of the present application, a trajectory planning method for collision avoidance of an equipotential working robot arm is provided, including: determining the state space, action space and reward function of the robot arm, the state space is a state set of the environmental state of the working space in which the robot arm is located and the state of the robot arm itself, the action space is a set of actions of the robot arm moving in space, and the reward function is a function that takes into account the safe working space of the robot arm under live working and the requirement of the robot arm to avoid dangerous space on the basis of ensuring that the end of the robot arm moves to the target point position; iterative training is performed using a deep deterministic policy gradient algorithm based on the state space, the action space and the reward function to obtain a target policy model; the target point position of the robot arm is obtained, and the target point position is input into the target policy model to obtain the optimal moving path of the robot arm; the robot arm is controlled to move to the target point position according to the optimal moving path.

[0007] Optionally, determining the state space, action space and reward function of the robotic arm includes: determining the self-state of the robotic arm and the environmental state of the robotic arm, and determining the state space based on the self-state and the environmental state of the robotic arm, the state space at least including the self-state and the environmental state of the robotic arm, the self-state at least including the joint angle of the robotic arm and the spatial position of the end of the robotic arm, and the environmental state at least including the spatial position of the target point and the relative distance between the end of the robotic arm and the target point; determining the action space based on the joint angle of the robotic arm; calculating the distance between the spatial position of the end of the robotic arm and the target point to obtain the relative distance; simulating and calculating the spatial electric field of the working environment of the robotic arm to obtain the safe working space and the dangerous space; determining the reward function based on the position of the end of the robotic arm combined with the relative distance, the safe working space and the dangerous space.

[0008] Optionally, the spatial electric field of the working environment of the robot arm is simulated and calculated to obtain the safe working space and the dangerous space, including: using the finite element analysis method to simulate and model the electric field distribution of the robot arm within a set range to obtain an electric field model, wherein the set range is the set range of the hardware and wires before the robot arm enters the live working position; using the mirror method to calculate the electric field of the image charge in the electric field model to obtain the conductor electric field of the robot arm, and calculating the spatial electric field generated by the image charge; adding the conductor electric field and the spatial electric field to obtain the substation spatial electric field; based on the substation spatial electric field The electric field calculates the maximum electric field strength on the surface of the soft wire at different gap distances between the robotic arm and the adjacent soft wire; the key point method is used to calculate the distance between the joint center point of the robotic arm and the key point of the insulation safety boundary, and the smallest distance is taken as the minimum distance; the area that meets the first condition and the second condition is determined as the safe working space, the first condition is that the maximum electric field strength is less than or equal to the set maximum working electric field strength, and the second condition is that the minimum distance is greater than the set required distance; the area formed by the irregular sweeping body with the soft wire as the axis and a specific length as the radius is determined as the dangerous space.

[0009] Optionally, the electric field of the image charge in the electric field model is calculated by using the mirror method to obtain the conductor electric field of the manipulator, and the spatial electric field generated by the image charge is calculated, including: calculating the conductor electric field of the manipulator according to a first formula, wherein the first formula is: V represents the potential of the conductor in the electric field, E d represents the electric field of the conductor; and calculating the spatial electric field generated by the image charge according to the second formula, the second formula being: Eq represents the electric field in the space, k is the Coulomb constant, q i is the charge magnitude of the i-th image charge in the space electric field, r i is the distance from the i-th image charge to the conductor, is the unit vector pointing towards the conductor.

[0010] Optionally, the reward function is determined according to the position of the end of the robotic arm in combination with the relative distance, the safe operating space and the dangerous space, including: determining a first function, the first function being a piecewise constant function whose penalty value is a first set value when the robotic arm enters the dangerous space or exceeds the safe operating space, and whose penalty value is 0 when the robotic arm does not enter the dangerous space or does not exceed the safe operating space, and the first set value is a negative number; determining a second function, the second function being a piecewise constant function whose reward value is a second set value when the robotic arm reaches the position of the target point, and whose reward value is 0 when the robotic arm does not reach the position of the target point, and the second set value is a positive number; adding the relative distance, the first function and the second function to obtain the reward function.

[0011] Optionally, iterative training is performed using a deep deterministic policy gradient algorithm based on the state space, the action space and the reward function to obtain a target policy model, including: initializing an evaluation network, an action network, a target evaluation network and a target action network; a generation step, inputting the current sample state into the action network so that the action network generates an optimal action according to the current sample state, and the current sample state space is any sample state in the training sample set; an execution step, the evaluation network executes the optimal action and evaluates based on the reward function to obtain an evaluation score; an updating step, using the gradient descent method to update the parameters in the evaluation network, the action network, the target evaluation network and the target action network; a repeating step, repeating the generation step, the execution step and the updating step in sequence at least once until the end of the robotic arm reaches the set position, and taking the latest evaluation score as the cumulative reward value of the current iteration; repeating the generation step, the execution step, the updating step and the repeating step in sequence at least once until the number of iterations reaches the set maximum number of iterations or the cumulative reward value reaches the maximum value, to obtain the target policy model.

[0012] Optionally, the parameters in the evaluation network, the action network, the target evaluation network and the target action network are updated by using a gradient descent method, including: minimizing a third formula to update the evaluation network parameters in the evaluation network, wherein the third formula is N represents the total number of samples selected for learning from the training sample set, R i is the reward score calculated by the reward function, γ represents the discount parameter, Q(S i , A i |θ Q ) indicates that in the current sample state S i Execute the optimal action A i The evaluation score obtained after that, Q′((S i+1 |θ u′ )|θ Q′ ) indicates that in the current sample state S i The target score after executing the action, θ Q is the evaluation network parameter, θ Q′ is the target evaluation network parameter, θ u′ is the target action network parameter, S i is the current sample state, A i is the optimal action; and the action network parameters in the action network are updated according to the fourth formula, and the fourth formula is θ u represents the action network parameters, u(S|θ u ) represents the action network, Q(S, A|θ Q ) represents the evaluation network; updating the target evaluation network parameters in the target evaluation network according to the fifth formula, wherein the fifth formula is θ Q′ ←τθ Q +(1-τ)θ Q′ , τ represents the soft update parameter; according to the sixth formula, the target action network parameters in the target action network are updated, and the sixth formula is θ u′ ←τθ u +(1-τ)θ u′ .

[0013] According to another aspect of the present application, a trajectory planning device for collision avoidance of an equipotential working robot arm is provided, the device comprising: a determination unit, used to determine the state space, action space and reward function of the robot arm, the state space being a state set of the environmental state of the working space in which the robot arm is located and the state of the robot arm itself, the action space being a set of actions of the robot arm moving in space, and the reward function being a function that takes into account the safe working space of the robot arm under live working and the requirement of the robot arm to avoid dangerous space on the basis of ensuring that the end of the robot arm moves to the target point position; a training unit, used to perform iterative training using a deep deterministic policy gradient algorithm based on the state space, the action space and the reward function to obtain a target policy model; an acquisition unit, used to obtain the target point position of the robot arm, input the target point position into the target policy model, and obtain the optimal moving path of the robot arm; a control unit, used to control the robot arm to move to the target point position according to the optimal moving path.

[0014] According to another aspect of the present application, a computer-readable storage medium is provided, wherein the computer-readable storage medium includes a stored program, wherein when the program is executed, the device where the computer-readable storage medium is located is controlled to execute any one of the methods described.

[0015] According to another aspect of the present application, a computer program product is provided, comprising computer instructions, wherein when the computer instructions are executed by a processor, any one of the methods described above is implemented.

[0016] Applying the technical solution of the present application, in the trajectory planning method of collision avoidance of an equipotential operation manipulator, first, determine the state space, action space and reward function of the manipulator, the state space is the state set of the environmental state of the working space where the manipulator is located and the state of the manipulator itself, the action space is the action set of the manipulator moving in space, and the reward function is a function that considers the safe working space of the manipulator under live working and the need for the manipulator to avoid dangerous space on the basis of ensuring that the end of the manipulator moves to the target point position; then, based on the state space, the action space and the reward function, iterative training is performed using a deep deterministic policy gradient algorithm to obtain a target policy model; then, the target point position of the manipulator is obtained, and the target point position is input into the target policy model to obtain the optimal moving path of the manipulator; finally, the manipulator is controlled to move to the target point according to the optimal moving path. The present application designs a working space for the manipulator, and the working space includes a safe working space and a threat space. When the manipulator works in the safe working space, it can not only ensure the safety of the equipment and itself in the substation, but also effectively complete the task. By using reinforcement learning to train the robotic arm, a deep reinforcement learning algorithm is used to implement path planning for the equipotential robotic arm, fully considering the safe working environment of live working and the obstacle avoidance requirements of the robotic arm, ensuring that the robotic arm can eventually reach the target point in an orderly manner without collision. This application solves the problem that traditional methods of controlling the robotic arm are difficult to avoid dangerous spaces. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 A hardware structure block diagram of a mobile terminal for performing a trajectory planning method for collision avoidance of an equipotential operation manipulator provided in an embodiment of the present application is shown;

[0018] Figure 2 A schematic flow chart of a trajectory planning method for collision avoidance of an equipotential operation robot arm provided in accordance with an embodiment of the present application is shown;

[0019] Figure 3 A schematic diagram showing simulation results of an equipotential working robot arm provided according to an embodiment of the present application is shown;

[0020] Figure 4 A schematic diagram of the structure of an equipotential operation mechanical arm provided according to an embodiment of the present application is shown;

[0021] Figure 5 A schematic diagram of a safe operating space for a robotic arm in a 220kV electric field provided according to an embodiment of the present application is shown;

[0022] Figure 6 A structural diagram of a deep deterministic policy gradient algorithm provided according to an embodiment of the present application is shown;

[0023] Figure 7 A structural block diagram of a trajectory planning device for collision avoidance of an equipotential working robot arm provided according to an embodiment of the present application is shown.

[0024] The above drawings include the following reference numerals:

[0025] 102, processor; 104, memory; 106, transmission device; 108, input and output devices. DETAILED DESCRIPTION

[0026] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0027] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0028] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present application described here. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0029] For the convenience of description, some nouns or terms involved in the embodiments of the present application are explained below:

[0030] Equipotential work: refers to a safety measure that involves adjusting the body potential of the worker to the same level as the potential of the surrounding conductors with voltage. The purpose of this is to reduce the risk of electric shock, because in the equipotential state, there is no potential difference between the human body and the charged body, so no current will flow through the human body.

[0031] Equipotential operating robot arm: uses a robot arm to replace the human body and adjusts the potential of the robot arm to the same level as the potential of the surrounding conductors with voltage.

[0032] As introduced in the background technology, the robot arm trajectory planning in the prior art follows a fixed and structured process. Therefore, in some dangerous, complex and changeable environments, the trajectory planning of traditional robot arms is easily affected by the changing environment, and its trajectory may deviate, resulting in serious consequences. In order to solve the problem that traditional methods of controlling robot arms are difficult to avoid dangerous spaces, the embodiments of the present application provide a trajectory planning method, trajectory planning device, computer-readable storage medium and computer program product for collision avoidance of equipotential working robot arms.

[0033] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0034] The method embodiments provided in the embodiments of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Taking running on a mobile terminal as an example, Figure 1 1 is a hardware structure block diagram of a mobile terminal of a trajectory planning method for collision avoidance of an equipotential operation robot arm according to an embodiment of the present invention. Figure 1 As shown, the mobile terminal may include one or more ( Figure 1 Only one is shown in the figure) a processor 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA) and a memory 104 for storing data, wherein the mobile terminal may also include a transmission device 106 and an input / output device 108 for communication functions. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the mobile terminal. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations are shown.

[0035] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the trajectory planning method for collision avoidance of the equipotential operation robot arm in the embodiment of the present invention. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, the above method is realized. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories can be connected to the mobile terminal via a network. Examples of the above-mentioned network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof. The transmission device 106 is used to receive or send data via a network. The above-mentioned specific examples of the network may include a wireless network provided by a communication provider of the mobile terminal. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 may be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0036] In this embodiment, a trajectory planning method for collision avoidance of an equipotential working robot arm running on a mobile terminal, a computer terminal or a similar computing device is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0037] Figure 2 1 is a flow chart of a trajectory planning method for collision avoidance of an equipotential operation robot arm according to an embodiment of the present application. Figure 2 As shown, the method comprises the following steps:

[0038] Step S201, determine the state space, action space and reward function of the robot arm, where the state space is a state set of the environmental state of the working space where the robot arm is located and the state of the robot arm itself, the action space is a set of actions of the robot arm moving in space, and the reward function is a function that takes into account the safe working space of the robot arm under live working and the need for the robot arm to avoid dangerous spaces on the basis of ensuring that the end of the robot arm moves to the target point.

[0039] Specifically, a deep reinforcement learning algorithm is used to implement path planning for the equipotential manipulator. Specifically, a deep deterministic policy gradient algorithm is used to train the path planning model to generate the optimal path planning. Therefore, it is necessary to first obtain the parameters of the deep deterministic policy gradient algorithm, namely the above-mentioned state space, action space and reward function. The action space defines all possible actions that the manipulator can perform in space. These actions can be: the increment or absolute value of the joint angle; the speed or acceleration of the end effector; specific motion instructions, such as forward, backward, left turn, right turn, etc. The reward function includes positive rewards: the reduction of the distance between the end of the manipulator and the target point, the completion of a specific task or the achievement of a specific milestone; and negative rewards: the approach of the manipulator to an obstacle, the manipulator entering an unsafe operating area, and the unstable state of the manipulator, such as the joint angle exceeding the normal working range.

[0040] Step S202, based on the state space, the action space and the reward function, iterative training is performed using a deep deterministic policy gradient algorithm to obtain a target policy model.

[0041] Specifically, initialize the action network (Actor Network) and the evaluation network (Critic Network), initialize the target action network and the target evaluation network, and initialize the experience replay buffer (ReplayBuffer). Collect initial experience, such as randomly selecting actions, executing and collecting interaction data with the environment (state, action, reward, new state). Store the interaction data in the experience replay buffer. Randomly extract a batch of data from the experience replay buffer, use this batch of data to update the parameters of the evaluation network and the action network for training, and obtain the final target strategy model. After the training is completed, the obtained target strategy model can be used to guide the robot arm to perform tasks in the actual environment. Using deep reinforcement learning, the equipotential robot arm has the ability to automatically avoid obstacles, and in the high-voltage working environment, it is ensured that it does not enter the dangerous area, which plays an important role in ensuring the safe and stable operation of power equipment.

[0042] Step S203, obtaining the target point position of the above-mentioned robot arm, inputting the above-mentioned target point position into the above-mentioned target strategy model, and obtaining the optimal moving path of the above-mentioned robot arm.

[0043] Specifically, the visual system is used for rough positioning to determine the position that the end of the robot arm needs to reach. Determining the end of the robot arm is to determine the target point position that the robot arm needs to reach, so that the robot arm can work in a safe space and reach the target point position to perform subsequent related operations. After determining the target point position that the robot arm needs to reach, the target point position is input into the above-mentioned target strategy model to obtain the optimal movement path of the above-mentioned robot arm.

[0044] Step S204, controlling the robot arm to move to the target point position according to the optimal moving path.

[0045] Specifically, after obtaining the optimal moving path of the above-mentioned robot arm, the above-mentioned robot arm can be directly controlled to move to the above-mentioned target point position according to the above-mentioned optimal moving path. In addition, in order to verify the effectiveness of this application, an equipotential robot arm simulation experiment is carried out. Figure 3 As shown, Figure 3 The two points (point 1 and point 2) are the target points that the robot needs to reach. By using reinforcement learning to train the robot, the robot can eventually reach the target points in an orderly manner without collision.

[0046] In this embodiment, first, the state space, action space and reward function of the robot arm are determined. The state space is a state set of the environmental state of the working space where the robot arm is located and the state of the robot arm itself. The action space is a set of actions of the robot arm moving in space. The reward function is a function that considers the safe working space of the robot arm under live working and the need for the robot arm to avoid dangerous space on the basis of ensuring that the end of the robot arm moves to the target point position; then, based on the state space, the action space and the reward function, iterative training is performed using a deep deterministic policy gradient algorithm to obtain a target policy model; then, the target point position of the robot arm is obtained, and the target point position is input into the target policy model to obtain the optimal moving path of the robot arm; finally, the robot arm is controlled to move to the target point position according to the optimal moving path. This application designs an operating space for the robot arm, and the operating space includes a safe operating space and a threat space. When the robot arm works in the safe operating space, it can not only ensure the safety of the equipment and itself in the substation, but also effectively complete the task. By using reinforcement learning to train the robotic arm, a deep reinforcement learning algorithm is used to implement path planning for the equipotential robotic arm, fully considering the safe working environment of live working and the obstacle avoidance requirements of the robotic arm, ensuring that the robotic arm can eventually reach the target point in an orderly manner without collision. This application solves the problem that traditional methods of controlling the robotic arm are difficult to avoid dangerous spaces.

[0047] In order to enable those skilled in the art to more clearly understand the technical solution of the present application, the implementation process of the trajectory planning method for collision avoidance of the equipotential working robot arm of the present application will be described in detail below in combination with specific embodiments.

[0048] In order to realize the path planning of the equipotential manipulator, the safety of the equipment in the substation and the manipulator itself can be ensured while the task can be completed effectively. In an optional implementation, the above step S201 includes:

[0049] Step S2011, determining the self-state of the robotic arm and the environmental state of the robotic arm, and determining the state space according to the self-state and the environmental state of the robotic arm, wherein the state space at least includes the self-state and the environmental state of the robotic arm, wherein the self-state at least includes the joint angle of the robotic arm and the spatial position of the end of the robotic arm, and the environmental state at least includes the spatial position of the target point and the relative distance between the end of the robotic arm and the target point;

[0050] Step S2012, determining the action space according to the joint angles of the robot arm;

[0051] Step S2013, calculating the distance between the spatial position of the end of the robotic arm and the target point to obtain a relative distance;

[0052] Step S2014, performing simulation calculation on the spatial electric field of the working environment of the above-mentioned robot arm to obtain the above-mentioned safe working space and the above-mentioned dangerous space;

[0053] Step S20 1 5, determining the reward function according to the position of the end of the robot arm combined with the relative distance, the safe working space and the dangerous space.

[0054] In the above embodiment, the state space of the robot arm is defined as Figure 4As shown, the present invention uses a six-degree-of-freedom robotic arm for operation. The state space of the robotic arm refers to the state of the environment and the state of the robotic arm itself. The state space for the robotic arm mainly includes the angles of each joint, the position of the target point, and the position of the end of the robotic arm. They serve as input variables to enable the robotic arm to obtain relevant information and perform actions based on this information to obtain bonus points. The state of the robotic arm itself refers to the changing state of its own structure, including the size of the robotic arm joint angle, which can be expressed as A=(θ1, θ2, θ3, θ4, θ5, θ6). At the same time, the spatial coordinate position of the end of the robotic arm is defined as (x, y, z). The environmental state refers to the state obtained when the robotic arm interacts with the environment, mainly including the spatial position coordinates of the target point (a, b, c), and the relative distance r1 between the end of the robotic arm and the target point. Define the robotic arm motion space. In order to plan the trajectory of the end of the robotic arm moving from the initial point to the target point, first obtain various environmental state information, including the spatial coordinates of the target point and the end of the robotic arm. Then, after inverse kinematics calculations, the joint angles are obtained. The joint angles must be adjusted continuously with the purpose of shortening the distance between the end and the target point. Therefore, the action space is defined as A = (θ1, θ2, θ3, θ4, θ5, θ6), and the joint angles of the robot are the action space. Define the reward function: The key to autonomous trajectory planning of the robot arm lies in the way the reward value is obtained. The design of the reward function is mainly based on the relative coordinates of the end of the robot arm and the target point. Calculate the distance between the spatial positions of the above-mentioned robot arm end and the above-mentioned target point to obtain the relative distance, including: Substitute the spatial positions of the above-mentioned robot arm end and the above-mentioned target point into the seventh formula to calculate the above-mentioned relative distance. The above-mentioned seventh formula is r1 represents the above relative distance, (x, y, z) represents the spatial position coordinates of the end of the robot arm, and (a, b, c) represents the spatial position coordinates of the target point. Secondly, the equipotential robot arm should work in a safe working area and cannot touch other objects. The safe working space of the live working robot is not only limited by the narrow space between the substation equipment, but also by the electrical safety distance between phases and phases to the ground. Therefore, it is necessary to design a safe working space for the robot, in which the robot can ensure the safety of the equipment and itself in the substation while completing the task effectively. The safe working space and the dangerous space are planned by electromagnetic simulation of the environment, and the reward function is determined according to the relative distance and the position relationship between the position of the end of the robot arm and the safe working space and the dangerous space.

[0055] In order to plan a safe working space for the electromagnetic simulation of the robot's working environment to improve the safety of the substation equipment and the robot itself, in an optional implementation, the above step S2014 includes:

[0056] Step S20141, using finite element analysis to simulate and model the electric field distribution of the above-mentioned robot arm within a set range to obtain an electric field model, wherein the above-mentioned set range is a set range of hardware and wires before the above-mentioned robot arm enters the live working position;

[0057] Step S20142, using the mirror method to perform electric field calculation on the image charges in the electric field model, to obtain the conductor electric field of the manipulator, and to calculate the spatial electric field generated by the image charges;

[0058] Step S20143, adding the conductor electric field and the space electric field to obtain the substation space electric field;

[0059] Step S20144, calculating the maximum electric field strength on the surface of the flexible conductor at different gap distances between the mechanical arm and the adjacent flexible conductor based on the spatial electric field of the substation;

[0060] Step S20145, using a key point method to calculate the distance between the joint center point of the above-mentioned mechanical arm and the key point of the insulation safety boundary, and taking the smallest of the above-mentioned distances as the minimum distance;

[0061] Step S20146, determining an area that meets a first condition and a second condition as the safe working space, wherein the first condition is that the maximum electric field strength is less than or equal to a set maximum working electric field strength, and the second condition is that the minimum distance is greater than a set required distance;

[0062] Step S20147: The area formed by the irregular sweeping body with the above-mentioned soft wire as the axis and a specific length as the radius is determined as the above-mentioned dangerous space.

[0063] In the above embodiment, the image charge method is used to establish a model and calculate the surrounding space electric field. Specifically, the finite element analysis software is used to simulate and model the electric field distribution around the hardware and wires before the robot enters the target point. The image charge method is simple, practical, and highly accurate. This method does not require complex boundary conditions to solve the Laplace equation. It only needs to use image charges to replace various types of charged bodies and grounded bodies to obtain the potential and electric field strength at any point in the electric field. When calculating the spatial electric field around the busbar connecting conductor, the conductor can be regarded as a charged wire segment with uniformly distributed charges, and the image charge is located at the center of the conductor. When the influence of the earth is taken into account, the mirror method is used to calculate the electric field, and the conductor electric field and the spatial electric field generated by the image charge can be calculated. The conductor electric field and the spatial electric field generated by the image charge can be added to obtain the spatial electric field between the substation equipment where the robot is located, that is, the above-mentioned substation spatial electric field; according to the electric field solution result, the maximum working electric field strength E of the robot arm is set m , then the electric field strength E in the robot arm working space r Should meet E r≤E m . The robot arm operating space includes safe operating space and dangerous space. According to the calculation of the maximum electric field strength on the surface of the soft wire at different gap distances between the robot arm elbow joint and the adjacent soft wire, the safe operating space outside the dangerous space formed by the irregular sweep body with the soft wire as the axis and a specific length as the radius can be obtained. Generally speaking, it is a three-dimensional space with the soft wire as the center line and extending along the length direction of the wire. The boundary of this three-dimensional space is formed by the soft wire surface extending outward by a specific safe distance (that is, a radius of a specific length). Since the soft wire may not be completely straight and may have different bends and twists in space, the space area formed is irregular. In the safe operation of the power system, the area formed by this irregular sweep body is defined as a dangerous space, and a dangerous boundary plane (also known as a safety boundary, that is, if it does not exceed this plane, it is a safe area, and if it exceeds this plane, it is a dangerous area) is constructed. The plane is perpendicular to the ground and is located between the soft wire and the clamping robot arm. Let the distance between the dangerous boundary plane and the adjacent phase soft wire be Dissafe, such as Figure 5 As shown in the figure, the danger boundary is represented by a dotted line. The safety standard for live working is that the robot arm does not touch the danger boundary plane. This plane is a man-made virtual construction and does not actually exist. In the analysis process, the key point method is used to calculate the minimum distance between the robot and the insulation safety boundary. That is, the position of the end effector of the robot arm and the center point of each joint (x i ,y i , z i ), and the key points of the safety boundary (x l ,y l , z l ). Calculate the distances between all key points of the robot arm and all key points of the insulation safety boundary according to the distance formula. Among all the calculated distances, select the smallest one as the minimum distance d between the robot arm and the insulation safety boundary min The distance formula is Set the required distance to d m , then the minimum distance between the robot and the insulation safety boundary should satisfy d min >d m .

[0064] For the electric field distribution of the robot arm, in an optional implementation, the above step S20142 includes:

[0065] Step S201421, calculating the conductor electric field of the robotic arm according to a first formula, the first formula being: V represents the above potential of the conductor in the above electric field, E d represents the electric field of the above conductor;

[0066] Step S201422, calculating the spatial electric field generated by the image charge according to a second formula, the second formula is: E q represents the electric field in the above space, k is the Coulomb constant, q i is the charge magnitude of the i-th image charge in the above space electric field, r i is the distance from the i-th image charge to the conductor, is the unit vector pointing towards the conductor.

[0067] In the above embodiment, the simulated charge method uses simulated charges to replace various charged bodies and grounded bodies, so as to obtain the potential and electric field strength at any point in the field. When the influence of the earth is taken into account, the mirror method is used to obtain the electric field E at the conductor according to the known potential V of the conductor (the potential has been obtained using the simulated charge method) according to the first formula. d The spatial electric field generated by the image charge is calculated by the second formula.

[0068] The image method is a classic electromagnetic analysis method used to solve the electric field or potential distribution around a charged object. It is based on the mirror principle and simplifies electromagnetic problems on complex boundaries by mirroring the charge or current where the boundary conditions change. In electromagnetism, the image method is often used to solve electromagnetic field problems on the surface of a conductor or near a charged object. Specifically, the basic principle of the image method is: if there is a charge near a grounded conductor, it can be assumed that there is an equal but opposite charge (i.e., image charge) on the other side of the conductor, which can make the electric field on the surface of the conductor zero (because the electric field inside the conductor is always zero). By calculating the electric field generated by this image charge, the electric field distribution in the original problem can be approximately solved. By calculating the electric field under the combined action of the original charge and the image charge, the electric field strength and direction in the space near the charged object can be determined.

[0069] In order to fully consider the safe working environment of live working and the obstacle avoidance requirements of the robot arm, in an optional implementation manner, the above step S2015 includes:

[0070] Step S20151, determining a first function, wherein the first function is a piecewise constant function whose penalty value is a first set value when the robot arm enters the dangerous space or exceeds the safe operation space, and whose penalty value is 0 when the robot arm does not enter the dangerous space or does not exceed the safe operation space, and the first set value is a negative number;

[0071] Step S20152, determining a second function, wherein the second function is a piecewise constant function in which the reward value is a second set value when the robot reaches the position of the target point, and the reward value is 0 when the robot does not reach the position of the target point, and the second set value is a positive number;

[0072] Step S20153, adding the relative distance, the first function and the second function to obtain the reward function.

[0073] In the above embodiment, the relative distance is calculated according to the seventh formula, which is: r1 represents the relative distance, (x, y, z) represents the spatial coordinates of the end of the robot arm, (a, b, c) represents the spatial coordinates of the target point, and r2 is set when the robot arm touches the dangerous space or exceeds the safe working space, that is, the first function mentioned above, When the robot reaches the target point, r3 is set, which is the second function mentioned above. Finally, r1, r3, and r3 are added together to obtain the reward function R, R=r1+r2+r3, where r1 represents the relative distance between the end of the robot arm and the target point, r3 represents penalty points if the robot arm exceeds the safety range, and r3 represents reward points if the robot arm reaches the target point. The reward function fully considers the safe working environment of live working and the obstacle avoidance requirements of the robot arm.

[0074] In order to enhance the robustness and adaptability of the target policy model, in an optional implementation, the above step S202 includes:

[0075] Step S2021, initializing the evaluation network, the action network, the target evaluation network and the target action network;

[0076] Step S2022, a generation step, inputting the current sample state into the action network so that the action network generates an optimal action according to the current sample state, wherein the current sample state space is any sample state in the training sample set;

[0077] Step S2023, executing the step, the evaluation network executes the optimal action and evaluates based on the reward function to obtain an evaluation score;

[0078] Step S2024, an updating step, using a gradient descent method to update the parameters in the evaluation network, the action network, the target evaluation network, and the target action network;

[0079] Step S2025, repeating the steps, repeating the generation step, the execution step and the update step at least once in sequence, until the end of the robot arm reaches the set position, and taking the latest evaluation score as the cumulative reward value of the current iteration;

[0080] Step S2026, repeat the above generation step, the above execution step, the above update step and the above repetition step in sequence at least once, until the number of iterations reaches the set maximum number of iterations or the above cumulative reward value reaches the maximum value, to obtain the above target strategy model.

[0081] In the above embodiment, the purpose is to construct a DDPG algorithm model, that is, the above-mentioned target policy model. The Deep Deterministic Policy Gradient algorithm (DDPG) is one of the classic algorithms in DRL, which is the product of the combination of reinforcement learning and neural network technology. The reinforcement learning process can be expressed as a Markov decision process, which can be defined as a tuple (S, A, P, R, γ), whose elements represent the finite state set S, the action set A, the state transition probability P, the reward R, and the discount factor γ∈[0,1). The process can be described as: in each unit time step t, the agent obtains the current state information s t ∈S, s t Input strategy π, output new action a t ∈A executes and obtains a new state s t+1 ∈S, and return s t+1 Rewards t+1 The ultimate goal of the process is to obtain an optimal strategy π that maximizes the cumulative return. Figure 6 This is the structure diagram of the deep deterministic policy gradient algorithm. According to the specific network parameters (the evaluation network parameters are θ Q and action network parameters are θμ), the initial evaluation network Q(S, A|θ Q ), initialize the action network u(S, A|θ μ ), target evaluation network Q′ and target action network u′. The goal of the learning phase is to update the parameter node weights of the evaluation network and the action network. Input the current state into the action network, and the action network outputs the next optimal action based on the probability distribution; the evaluation network judges and scores the optimal action generated by the action network, and the action network modifies the probability of selecting each action according to the score of the evaluation network. Repeat the above process until the robot reaches the set target position and ends the current training. However, in order to ensure the accuracy of the target policy model, the set target position that the robot needs to reach can be adjusted for training again or only the parameters of the policy model can be adjusted to select the model with the maximum cumulative reward value as the target policy model.

[0082] In order to improve the accuracy of the trained target strategy model, in an optional implementation, the above step S2024 includes:

[0083] Step S20241, the third formula is minimized to update the evaluation network parameters in the evaluation network. The third formula is: N represents the total number of samples selected for learning from the training sample set, R i is the reward score calculated by the reward function, γ represents the discount parameter, Q(S i , A i |θ Q ) indicates that in the above current sample state S i Execute the optimal action A above i The above evaluation score obtained after i+1 |θ u′ )|θ Q′ ) indicates that in the above current sample state S i The target score after executing the action, θ Q is the above evaluation network parameter, θ Q′ is the target evaluation network parameter, θ u′ is the target action network parameter, S i is the current sample state, A i is the optimal action mentioned above;

[0084] Step S20242, updating the action network parameters in the action network according to the fourth formula, the fourth formula is: θ u represents the above action network parameters, u(S|θ u ) represents the above action network, Q(S, A|θ Q ) represents the above evaluation network;

[0085] Step S20243, updating the target evaluation network parameters in the target evaluation network according to the fifth formula, wherein the fifth formula is θ Q′ ←τθ Q +(1-τ)θ Q′ , τ represents the soft update parameter;

[0086] Step S20244, updating the target action network parameters in the target action network according to the sixth formula, wherein the sixth formula is θ u′ ←τθ u +(1-τ)θ u′ .

[0087] In the above embodiment, after the action network predicts action A, the evaluation network will give a Q(S, A) value as the evaluation of the current state S after executing action A. In order to enable the evaluation network to accurately give the corresponding value, the third formula is minimized to update the evaluation network parameters: That is to say, the evaluation network parameters are updated by minimizing L, that is, the evaluation network parameters θ in L are updated Q The third formula L gets the minimum value. L does not represent the evaluation network value. L has no actual meaning and is only used to represent this formula. In order to enable the action network to obtain a larger Q value in the action network, the fourth formula is used to update the parameters of the action network. The Q value is the sum of the cumulative future rewards expected to be obtained by taking a specific action A = u(S) in a specific state S. In other words, the Q value reflects the "goodness" or "pros and cons" of an action in a specific state. In reinforcement learning algorithms such as Q learning, the Q value is approached by continuous learning and updating to the optimal strategy. In the fourth formula, Represents the gradient, and then the fifth formula and the sixth formula are used to update the target evaluation network parameters and the target action network parameters respectively.

[0088] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0089] The embodiment of the present application also provides a trajectory planning device for collision avoidance of an equipotential working robot arm. It should be noted that the trajectory planning device for collision avoidance of an equipotential working robot arm in the embodiment of the present application can be used to execute the trajectory planning method for collision avoidance of an equipotential working robot arm provided in the embodiment of the present application. The device is used to implement the above-mentioned embodiments and preferred implementation modes, and those that have been described will not be repeated here. As used below, the term "module" can implement a combination of software and / or hardware for a predetermined function. Although the device described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceived.

[0090] The following introduces the trajectory planning device for collision avoidance of an equipotential working robot arm provided in an embodiment of the present application.

[0091] Figure 7 It is a structural block diagram of a trajectory planning device for collision avoidance of an equipotential working robot arm according to an embodiment of the present application.

[0092] like Figure 7 As shown, the device comprises:

[0093] The determination unit 10 is used to determine the state space, action space and reward function of the robot arm, wherein the state space is a state set of the environmental state of the working space in which the robot arm is located and the state of the robot arm itself, the action space is a set of actions of the robot arm moving in space, and the reward function is a function that takes into account the safe working space of the robot arm under live working and the need for the robot arm to avoid dangerous spaces on the basis of ensuring that the end of the robot arm moves to the target point.

[0094] Specifically, a deep reinforcement learning algorithm is used to implement path planning for the equipotential manipulator. Specifically, a deep deterministic policy gradient algorithm is used to train the path planning model to generate the optimal path planning. Therefore, it is necessary to first obtain the parameters of the deep deterministic policy gradient algorithm, namely the above-mentioned state space, action space and reward function. The action space defines all possible actions that the manipulator can perform in space. These actions can be: the increment or absolute value of the joint angle; the speed or acceleration of the end effector; specific motion instructions, such as forward, backward, left turn, right turn, etc. The reward function includes positive rewards: the reduction of the distance between the end of the manipulator and the target point, the completion of a specific task or the achievement of a specific milestone; and negative rewards: the approach of the manipulator to an obstacle, the manipulator entering an unsafe operating area, and the unstable state of the manipulator, such as the joint angle exceeding the normal working range.

[0095] The training unit 20 is used to perform iterative training based on the state space, the action space and the reward function using a deep deterministic policy gradient algorithm to obtain a target policy model.

[0096] Specifically, initialize the action network (Actor Network) and the evaluation network (Critic Network), initialize the target action network and the target evaluation network, and initialize the experience replay buffer (ReplayBuffer). Collect initial experience, such as randomly selecting actions, executing and collecting interaction data with the environment (state, action, reward, new state). Store the interaction data in the experience replay buffer. Randomly extract a batch of data from the experience replay buffer, use this batch of data to update the parameters of the evaluation network and the action network for training, and obtain the final target strategy model. After the training is completed, the obtained target strategy model can be used to guide the robot arm to perform tasks in the actual environment. Using deep reinforcement learning, the equipotential robot arm has the ability to automatically avoid obstacles, and in the high-voltage working environment, it is ensured that it does not enter the dangerous area, which plays an important role in ensuring the safe and stable operation of power equipment.

[0097] The acquisition unit 30 is used to acquire the target point position of the above-mentioned robot arm, input the above-mentioned target point position into the above-mentioned target strategy model, and obtain the optimal moving path of the above-mentioned robot arm.

[0098] Specifically, the visual system is used for rough positioning to determine the position that the end of the robot arm needs to reach. Determining the end of the robot arm is to determine the target point position that the robot arm needs to reach, so that the robot arm can work in a safe space and reach the target point position to perform subsequent related operations. After determining the target point position that the robot arm needs to reach, the target point position is input into the above-mentioned target strategy model to obtain the optimal movement path of the above-mentioned robot arm.

[0099] The control unit 40 is used to control the above-mentioned robot arm to move to the above-mentioned target point position according to the above-mentioned optimal moving path.

[0100] Specifically, after obtaining the optimal moving path of the above-mentioned robot arm, the above-mentioned robot arm can be directly controlled to move to the above-mentioned target point position according to the above-mentioned optimal moving path. In addition, in order to verify the effectiveness of this application, an equipotential robot arm simulation experiment is carried out. Figure 3 As shown, Figure 3 The two points (point 1 and point 2) are the target points that the robot needs to reach. By using reinforcement learning to train the robot, the robot can eventually reach the target points in an orderly manner without collision.

[0101] In this embodiment, a working space is designed for the robot arm, which includes a safe working space and a threat space. When the robot arm works in the safe working space, it can ensure the safety of the equipment and itself in the substation and complete the task effectively. The robot arm is trained by using reinforcement learning, and the path planning of the equipotential robot arm is realized by using a deep reinforcement learning algorithm. The safe working environment of live working and the obstacle avoidance requirements of the robot arm are fully considered to ensure that the robot arm can eventually reach the target point in an orderly manner without collision. This application solves the problem that it is difficult for the traditional method to control the robot arm to avoid dangerous spaces.

[0102] In order to realize the path planning of the equipotential manipulator, the safety of the equipment in the substation and the manipulator itself can be ensured while the task can be completed effectively. In an optional implementation, the determination unit includes:

[0103] A first determination subunit determines the self-state of the robotic arm and the environmental state of the robotic arm, and determines the state space according to the self-state and the environmental state of the robotic arm, wherein the state space at least includes the self-state and the environmental state of the robotic arm, wherein the self-state at least includes the joint angle of the robotic arm and the spatial position of the end of the robotic arm, and the environmental state at least includes the spatial position of the target point and the relative distance between the end of the robotic arm and the target point;

[0104] A second determining subunit determines the action space according to the joint angles of the robotic arm;

[0105] A first calculation subunit calculates the distance between the spatial position of the end of the robotic arm and the target point to obtain a relative distance;

[0106] A second calculation subunit performs simulation calculation on the spatial electric field of the working environment of the above-mentioned robot arm to obtain the above-mentioned safe working space and the above-mentioned dangerous space;

[0107] The third determination subunit determines the reward function according to the position of the end of the robotic arm in combination with the relative distance, the safe working space and the dangerous space.

[0108] In the above embodiment, the state space of the robot arm is defined as Figure 4 As shown, the present invention uses a six-degree-of-freedom robotic arm for operation. The state space of the robotic arm refers to the state of the environment and the state of the robotic arm itself. The state space for the robotic arm mainly includes the angles of each joint, the position of the target point, and the position of the end of the robotic arm. They serve as input variables to enable the robotic arm to obtain relevant information and perform actions based on this information to obtain bonus points. The state of the robotic arm itself refers to the changing state of its own structure, including the size of the robotic arm joint angle, which can be expressed as A=(θ1, θ2, θ3, θ4, θ5, θ6). At the same time, the spatial coordinate position of the end of the robotic arm is defined as (x, y, z). The environmental state refers to the state obtained when the robotic arm interacts with the environment, mainly including the spatial position coordinates of the target point (a, b, c), and the relative distance r1 between the end of the robotic arm and the target point. Define the robotic arm motion space. In order to plan the trajectory of the end of the robotic arm moving from the initial point to the target point, first obtain various environmental state information, including the spatial coordinates of the target point and the end of the robotic arm. Then, after inverse kinematics calculations, the joint angles are obtained. The joint angles must be adjusted continuously with the purpose of shortening the distance between the end and the target point. Therefore, the action space is defined as A = (θ1, θ2, θ3, θ4, θ5, θ6), and the joint angles of the robot are the action space. Define the reward function: The key to autonomous trajectory planning of the robot arm lies in the way the reward value is obtained. The design of the reward function is mainly based on the relative coordinates of the end of the robot arm and the target point. Calculate the distance between the spatial positions of the above-mentioned robot arm end and the above-mentioned target point to obtain the relative distance, including: Substitute the spatial positions of the above-mentioned robot arm end and the above-mentioned target point into the seventh formula to calculate the above-mentioned relative distance. The above-mentioned seventh formula is r1 represents the above relative distance, (x, y, z) represents the spatial position coordinates of the end of the robot arm, and (a, b, c) represents the spatial position coordinates of the target point. Secondly, the equipotential robot arm should work in a safe working area and cannot touch other objects. The safe working space of the live working robot is not only limited by the narrow space between the substation equipment, but also by the electrical safety distance between phases and phases to the ground. Therefore, it is necessary to design a safe working space for the robot, in which the robot can ensure the safety of the equipment and itself in the substation while completing the task effectively. The safe working space and the dangerous space are planned by electromagnetic simulation of the environment, and the reward function is determined according to the relative distance and the position relationship between the position of the end of the robot arm and the safe working space and the dangerous space.

[0109] In order to plan a safe working space for the electromagnetic simulation of the robot's working environment to improve the safety of the substation equipment and itself, in an optional implementation, the second computing subunit includes:

[0110] A simulation module, using finite element analysis to simulate and model the electric field distribution of the above-mentioned robot arm within a set range to obtain an electric field model, wherein the above-mentioned set range is the set range of the hardware and the wire before the above-mentioned robot arm enters the live working position;

[0111] A first calculation module uses a mirror method to perform electric field calculation on the mirror charge in the electric field model to obtain the conductor electric field of the manipulator and calculate the spatial electric field generated by the mirror charge;

[0112] A second calculation module is configured to add the conductor electric field and the space electric field to obtain the substation space electric field;

[0113] A third calculation module calculates the maximum electric field strength on the surface of the flexible conductor at different gap distances between the mechanical arm and the adjacent flexible conductor based on the spatial electric field of the substation;

[0114] A fourth calculation module, using a key point method to calculate the distance between the joint center point of the above-mentioned mechanical arm and the key point of the insulation safety boundary, and taking the minimum of the above-mentioned distance as the minimum distance;

[0115] A first determination module determines an area that meets a first condition and a second condition as the safe working space, wherein the first condition is that the maximum electric field strength is less than or equal to a set maximum working electric field strength, and the second condition is that the minimum distance is greater than a set required distance;

[0116] The second determination module determines the area formed by the irregular sweeping body with the above-mentioned soft wire as the axis and a specific length as the radius as the above-mentioned dangerous space.

[0117] In the above embodiment, the image charge method is used to establish a model and calculate the surrounding space electric field. Specifically, the finite element analysis software is used to simulate and model the electric field distribution around the hardware and wires before the robot enters the target point. The image charge method is simple, practical, and highly accurate. This method does not require complex boundary conditions to solve the Laplace equation. It only needs to use image charges to replace various types of charged bodies and grounded bodies to obtain the potential and electric field strength at any point in the electric field. When calculating the spatial electric field around the busbar connecting conductor, the conductor can be regarded as a charged wire segment with uniformly distributed charges, and the image charge is located at the center of the conductor. When the influence of the earth is taken into account, the mirror method is used to calculate the electric field, and the conductor electric field and the spatial electric field generated by the image charge can be calculated. The conductor electric field and the spatial electric field generated by the image charge can be added to obtain the spatial electric field between the substation equipment where the robot is located, that is, the above-mentioned substation spatial electric field; according to the electric field solution result, the maximum working electric field strength E of the robot arm is set m , then the electric field strength E in the robot arm working space r Should meet E r ≤E m . The robot arm's operating space includes a safe operating space and a dangerous space. According to the calculation of the maximum electric field strength on the surface of the soft wire at different gap distances between the robot arm's elbow joint and the adjacent soft wire, the safe operating space outside the dangerous space formed by the irregular sweeping body with the soft wire as the axis and a specific length as the radius can be obtained. Generally speaking, it is a three-dimensional space with the soft wire as the center line and extending along the length of the wire. The boundary of this three-dimensional space is formed by the soft wire surface extending outward by a specific safe distance (that is, a radius of a specific length). Since the soft wire may not be completely straight and may have different bends and twists in space, the space area formed is irregular. Construct a dangerous boundary plane (also known as a safe boundary, that is, the area that does not exceed this plane is a safe area, and the area that exceeds this plane is a dangerous area). The plane is perpendicular to the ground and is located between the soft wire and the clamping robot arm. Let the distance between the dangerous boundary plane and the adjacent soft wire be Dissafe, such as Figure 5 As shown in the figure, the danger boundary is represented by a dotted line. The safety standard for live working is that the robot arm does not touch the danger boundary plane. This plane is a man-made virtual construction and does not actually exist. In the analysis process, the key point method is used to calculate the minimum distance between the robot and the insulation safety boundary. That is, the position of the end effector of the robot arm and the center point of each joint (x i ,y i , z i ), and the key points of the safety boundary (x l ,y l , z l). Calculate the distances between all key points of the robot arm and all key points of the insulation safety boundary according to the distance formula. Among all the calculated distances, select the smallest one as the minimum distance d between the robot arm and the insulation safety boundary min The distance formula is Set the required distance to d m , then the minimum distance between the robot and the insulation safety boundary should satisfy d min >d m .

[0118] In order to calculate the electric field distribution of the robot arm, in an optional implementation, the second calculation module includes:

[0119] The first calculation submodule calculates the conductor electric field of the robotic arm according to a first formula. The first formula is: V represents the above potential of the conductor in the above electric field, E d represents the electric field of the above conductor;

[0120] The second calculation submodule calculates the spatial electric field generated by the image charge according to the second formula. The second formula is: E q represents the electric field in the above space, k is the Coulomb constant, q i is the charge magnitude of the i-th image charge in the above space electric field, r i is the distance from the i-th image charge to the conductor, is the unit vector pointing towards the conductor.

[0121] In the above embodiment, the simulated charge method uses simulated charges to replace various charged bodies and grounded bodies, so as to obtain the potential and electric field strength at any point in the field. When the influence of the earth is taken into account, the mirror method is used to obtain the electric field E at the conductor according to the known potential V of the conductor (the potential has been obtained using the simulated charge method) according to the first formula. d The spatial electric field generated by the image charge is calculated by the second formula.

[0122] The image method is a classic electromagnetic analysis method used to solve the electric field or potential distribution around a charged object. It is based on the mirror principle and simplifies electromagnetic problems on complex boundaries by mirroring the charge or current where the boundary conditions change. In electromagnetism, the image method is often used to solve electromagnetic field problems on the surface of a conductor or near a charged object. Specifically, the basic principle of the image method is: if there is a charge near a grounded conductor, it can be assumed that there is an equal but opposite charge (i.e., image charge) on the other side of the conductor, which can make the electric field on the surface of the conductor zero (because the electric field inside the conductor is always zero). By calculating the electric field generated by this image charge, the electric field distribution in the original problem can be approximately solved. By calculating the electric field under the combined action of the original charge and the image charge, the electric field strength and direction in the space near the charged object can be determined.

[0123] In order to fully consider the safe working environment of live working and the obstacle avoidance requirements of the robot arm, in an optional implementation manner, the third determination subunit includes:

[0124] A third determination module determines a first function, wherein the first function is a piecewise constant function in which a penalty value is a first set value when the robot arm enters the dangerous space or exceeds the safe operation space, and the penalty value is 0 when the robot arm does not enter the dangerous space or does not exceed the safe operation space, and the first set value is a negative number;

[0125] a fourth determination module, determining a second function, wherein the second function is a piecewise constant function in which the reward value is a second set value when the robot arm reaches the position of the target point, and the reward value is 0 when the robot arm does not reach the position of the target point, and the second set value is a positive number;

[0126] An adding module adds the relative distance, the first function and the second function to obtain the reward function.

[0127] In the above embodiment, the relative distance is calculated according to the seventh formula, which is: r1 represents the relative distance, (x, y, z) represents the spatial coordinates of the end of the robot arm, (a, b, c) represents the spatial coordinates of the target point, and r2 is set when the robot arm touches the dangerous space or exceeds the safe working space, that is, the first function mentioned above, When the robot reaches the target point, r3 is set, which is the second function mentioned above. Finally, r1, r3, r3 are added together to get the reward function R, R = r i+r2+r3, r1 represents the relative distance between the end of the robot arm and the target point, r3 represents penalty points if the robot arm exceeds the safety range, and r3 represents bonus points if the robot arm reaches the target point. The reward function fully considers the safe working environment of live working and the obstacle avoidance requirements of the robot arm.

[0128] In order to enhance the robustness and adaptability of the target policy model, in an optional implementation, the training unit includes:

[0129] Initialize subunits, initialize the evaluation network, action network, target evaluation network and target action network;

[0130] A generating subunit, configured to execute the generating step, inputting the current sample state into the action network, so that the action network generates an optimal action according to the current sample state, wherein the current sample state space is any sample state in the training sample set;

[0131] An execution subunit, used to execute the execution step, wherein the evaluation network executes the optimal action and evaluates based on the reward function to obtain an evaluation score;

[0132] An updating subunit, used to perform an updating step, and to update the parameters in the evaluation network, the action network, the target evaluation network, and the target action network by using a gradient descent method;

[0133] A first repeating subunit is used to execute a repeating step, repeating the generating step, the executing step and the updating step in sequence at least once, until the end of the robotic arm reaches a set position, and taking the latest evaluation score as the cumulative reward value of the current iteration;

[0134] The second repeating subunit is used to sequentially repeat the above-mentioned generation step, the above-mentioned execution step, the above-mentioned update step and the above-mentioned repetition step at least once until the number of iterations reaches a set maximum number of iterations or the above-mentioned cumulative reward value reaches a maximum value, thereby obtaining the above-mentioned target strategy model.

[0135] In the above embodiment, the purpose is to construct a DDPG algorithm model, that is, the above-mentioned target policy model. The Deep Deterministic Policy Gradient algorithm (DDPG) is one of the classic algorithms in DRL, which is the product of the combination of reinforcement learning and neural network technology. The reinforcement learning process can be expressed as a Markov decision process, which can be defined as a tuple (S, A, P, R, γ), whose elements represent the finite state set S, the action set A, the state transition probability P, the reward R, and the discount factor γ∈[0,1). The process can be described as: in each unit time step t, the agent obtains the current state information st ∈S, s t Input strategy π, output new action a t ∈A executes and obtains a new state s t+1 ∈S, and return s t+1 Rewards t+1 The ultimate goal of the process is to obtain an optimal strategy π that maximizes the cumulative return. Figure 6 This is the structure diagram of the deep deterministic policy gradient algorithm. According to the specific network parameters (the evaluation network parameters are θ Q and action network parameters are θμ), the initial evaluation network Q(S, A|θ Q ), initialize the action network u(S, A|θ μ ), target evaluation network Q′ and target action network u′. The goal of the learning phase is to update the parameter node weights of the evaluation network and the action network. Input the current state into the action network, and the action network outputs the next optimal action based on the probability distribution; the evaluation network judges and scores the optimal action generated by the action network, and the action network modifies the probability of selecting each action according to the score of the evaluation network. Repeat the above process until the robot reaches the set target position and ends the current training. However, in order to ensure the accuracy of the target policy model, the set target position that the robot needs to reach can be adjusted for training again or only the parameters of the policy model can be adjusted to select the model with the maximum cumulative reward value as the target policy model.

[0136] In order to improve the accuracy of the trained target strategy model, in an optional implementation manner, the above step of updating the subunit includes:

[0137] The first updating module minimizes the third formula to update the evaluation network parameters in the evaluation network. The third formula is: N represents the total number of samples selected for learning from the training sample set, R i is the reward score calculated by the reward function, γ represents the discount parameter, Q(S i , A i |θ Q ) indicates that in the above current sample state S i Execute the optimal action A above i The above evaluation score obtained after i+1 |θ u′ )|θ Q ) indicates that in the above current sample state S i The target score after executing the action, θ Q is the above evaluation network parameter, θ Q′ is the target evaluation network parameter, θ u′ is the target action network parameter, S iis the current sample state, A i is the optimal action mentioned above;

[0138] The second updating module updates the action network parameters in the action network according to the fourth formula. The fourth formula is: θ u represents the above action network parameters, u(S|θ u ) represents the above action network, Q(S, A|θ Q ) represents the above evaluation network;

[0139] The third updating module updates the target evaluation network parameters in the target evaluation network according to the fifth formula. The fifth formula is θ Q′ ←τθ Q +(1-τ)θ Q′ , τ represents the soft update parameter;

[0140] The fourth updating module updates the target action network parameters in the target action network according to the sixth formula, where the sixth formula is θ u′ ←τθ u +(1-τ)θ u′ .

[0141] In the above embodiment, after the action network predicts action A, the evaluation network will give a Q(S, A) value as the evaluation of the current state S after executing action A. In order to enable the evaluation network to accurately give the corresponding value, the third formula is minimized to update the evaluation network parameters: That is to say, the evaluation network parameters are updated by minimizing L, that is, the evaluation network parameters θ in L are updated Q The third formula L gets the minimum value. L does not represent the evaluation network value. L has no actual meaning and is only used to represent this formula. In order to enable the action network to obtain a larger Q value in the action network, the fourth formula is used to update the parameters of the action network. The Q value is the sum of the cumulative future rewards expected to be obtained by taking a specific action A = u(S) in a specific state S. In other words, the Q value reflects the "goodness" or "pros and cons" of an action in a specific state. In reinforcement learning algorithms such as Q learning, the Q value is approached by continuous learning and updating to the optimal strategy. In the fourth formula, Represents the gradient, and then the fifth formula and the sixth formula are used to update the target evaluation network parameters and the target action network parameters respectively.

[0142] The above-mentioned trajectory planning device for collision avoidance of an equipotential operation robot arm includes a processor and a memory. The above-mentioned determination unit, training unit, and acquisition unit are all stored in the memory as program units, and the processor executes the above-mentioned program units stored in the memory to realize corresponding functions. The above-mentioned modules are all located in the same processor; or, the above-mentioned modules are located in different processors in any combination.

[0143] The processor contains a kernel, which calls the corresponding program unit from the memory. One or more kernels can be set, and the problem that the traditional method of controlling the robot arm is difficult to avoid dangerous spaces can be solved by adjusting the kernel parameters.

[0144] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0145] An embodiment of the present invention provides a computer-readable storage medium, which includes a stored program, wherein when the program is running, the device where the computer-readable storage medium is located is controlled to execute the trajectory planning method for collision avoidance of the equipotential working robot arm.

[0146] An embodiment of the present invention provides a processor, which is used to run a program, wherein the trajectory planning method for collision avoidance of the equipotential working robot arm is executed when the program is running.

[0147] An embodiment of the present invention provides a collision avoidance trajectory planning system for an equipotential operation robot arm. The collision avoidance trajectory planning system for an equipotential operation robot arm includes a processor, a memory, and a program stored in the memory and executable on the processor. When the processor executes the program, at least the following steps are implemented:

[0148] Step S201, determining the state space, action space and reward function of the robot arm, wherein the state space is a state set of the environmental state of the working space where the robot arm is located and the state of the robot arm itself, the action space is a set of actions of the robot arm moving in space, and the reward function is a function that takes into account the safe working space of the robot arm under live working and the requirement of the robot arm to avoid dangerous space on the basis of ensuring that the end of the robot arm moves to the target point position;

[0149] Step S202, performing iterative training using a deep deterministic policy gradient algorithm based on the state space, the action space, and the reward function to obtain a target policy model;

[0150] Step S203, obtaining the target point position of the robotic arm, inputting the target point position into the target strategy model, and obtaining the optimal moving path of the robotic arm;

[0151] Step S204, controlling the robot arm to move to the target point position according to the optimal moving path.

[0152] The present application also provides a computer program product, which, when executed on a data processing device, is suitable for executing a program for initializing at least the following method steps:

[0153] Step S201, determining the state space, action space and reward function of the robot arm, wherein the state space is a state set of the environmental state of the working space where the robot arm is located and the state of the robot arm itself, the action space is a set of actions of the robot arm moving in space, and the reward function is a function that takes into account the safe working space of the robot arm under live working and the requirement of the robot arm to avoid dangerous space on the basis of ensuring that the end of the robot arm moves to the target point position;

[0154] Step S202, performing iterative training using a deep deterministic policy gradient algorithm based on the state space, the action space, and the reward function to obtain a target policy model;

[0155] Step S203, obtaining the target point position of the robotic arm, inputting the target point position into the target strategy model, and obtaining the optimal moving path of the robotic arm;

[0156] Step S204, controlling the robot arm to move to the target point position according to the optimal moving path.

[0157] Obviously, those skilled in the art should understand that the above modules or steps of the present invention can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order than here, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present invention is not limited to any specific combination of hardware and software.

[0158] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0159] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0160] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0161] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0162] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0163] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0164] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0165] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0166] From the above description, it can be seen that the above embodiments of the present application achieve the following technical effects:

[0167] 1) The trajectory planning method for collision avoidance of the equipotential operation robot arm of the present application designs an operation space for the robot arm, which includes a safe operation space and a threat space. When the robot arm works in the safe operation space, it can ensure the safety of the equipment and itself in the substation, and can effectively complete the task. The robot arm is trained by using reinforcement learning, and the path planning of the equipotential robot arm is realized using a deep reinforcement learning algorithm. The safe working environment of live operation and the obstacle avoidance requirements of the robot arm are fully considered to ensure that the robot arm can eventually reach the target point in an orderly manner without collision. This application solves the problem that it is difficult for traditional methods to control the robot arm to avoid dangerous spaces.

[0168] 2) The trajectory planning device for collision avoidance of the equipotential operation robot arm of the present application designs an operation space for the robot arm, and the operation space includes a safe operation space and a threat space. When the robot arm works in the safe operation space, it can ensure the safety of the equipment and itself in the substation, and can effectively complete the task. The robot arm is trained by using reinforcement learning, and the path planning of the equipotential robot arm is realized using a deep reinforcement learning algorithm. The safe working environment of live operation and the obstacle avoidance requirements of the robot arm are fully considered, ensuring that the robot arm can eventually reach the target point in an orderly manner without collision. The present application solves the problem that it is difficult for the traditional method to control the robot arm to avoid dangerous spaces.

[0169] The above description is only the preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A trajectory planning method for collision avoidance of an equipotential operation robot arm, characterized in that: include: Determine the state space, action space and reward function of the robot arm, wherein the state space is a state set of the environmental state of the working space where the robot arm is located and the state of the robot arm itself, the action space is a set of actions of the robot arm moving in space, and the reward function is a function that takes into account the safe working space of the robot arm under live working and the requirement for the robot arm to avoid dangerous space on the basis of ensuring that the end of the robot arm moves to the target point position; Iteratively training is performed using a deep deterministic policy gradient algorithm based on the state space, the action space, and the reward function to obtain a target policy model; Obtaining the target point position of the robotic arm, inputting the target point position into the target strategy model, and obtaining the optimal movement path of the robotic arm; The robot arm is controlled to move to the target point position according to the optimal moving path.

2. The method according to claim 1, characterized in that Determine the state space, action space, and reward function of the robot arm, including: Determine the self state of the robot arm and the environmental state of the robot arm, and determine the state space according to the self state and the environmental state of the robot arm, wherein the state space at least includes the self state and the environmental state of the robot arm, the self state at least includes the joint angle of the robot arm and the spatial position of the end of the robot arm, and the environmental state at least includes the spatial position of the target point and the relative distance between the end of the robot arm and the target point; Determining the action space according to the joint angles of the robotic arm; Calculating the distance between the end of the robotic arm and the spatial position of the target point to obtain a relative distance; Performing simulation calculation on the spatial electric field of the working environment of the robot arm to obtain the safe working space and the dangerous space; The reward function is determined according to the position of the end of the robot arm combined with the relative distance, the safe operating space and the dangerous space.

3. The method according to claim 2, characterized in that The spatial electric field of the working environment of the robot arm is simulated and calculated to obtain the safe working space and the dangerous space, including: The electric field distribution of the robot arm within a set range is simulated and modeled by using a finite element analysis method to obtain an electric field model, wherein the set range is the set range of the hardware and the wire before the robot arm enters the live working position; Using the mirror method to calculate the electric field of the mirror charge in the electric field model, obtain the conductor electric field of the manipulator, and calculate the spatial electric field generated by the mirror charge; The conductor electric field and the space electric field are added together to obtain the substation space electric field; Calculate the maximum electric field strength on the surface of the flexible conductor at different gap distances between the mechanical arm and the adjacent flexible conductor based on the spatial electric field of the substation; The distance between the joint center point of the robot arm and the key point of the insulation safety boundary is calculated by using the key point method, and the minimum distance is taken as the minimum distance; Determine an area that meets a first condition and a second condition as the safe working space, wherein the first condition is that the maximum electric field strength is less than or equal to a set maximum working electric field strength, and the second condition is that the minimum distance is greater than a set required distance; The area formed by the irregular sweeping body with the soft wire as the axis and a specific length as the radius is determined as the dangerous space.

4. The method according to claim 3, characterized in that The image method is used to calculate the electric field of the image charge in the electric field model to obtain the conductor electric field of the manipulator, and the spatial electric field generated by the image charge is calculated, including: The conductor electric field of the robotic arm is calculated according to a first formula, where the first formula is: V represents the potential of the conductor in the electric field, E d represents the electric field of the conductor; The spatial electric field generated by the image charge is calculated according to a second formula, where the second formula is: E q represents the electric field in the space, k is the Coulomb constant, q i is the charge magnitude of the i-th image charge in the space electric field, r i is the distance from the i-th image charge to the conductor, is the unit vector pointing towards the conductor.

5. The method according to claim 2, characterized in that: The reward function is determined according to the position of the end of the robot arm combined with the relative distance, the safe working space and the dangerous space, including: Determine a first function, wherein the first function is a piecewise constant function in which a penalty value is a first set value when the robot arm enters the dangerous space or exceeds the safe operating space, and the penalty value is 0 when the robot arm does not enter the dangerous space or does not exceed the safe operating space, and the first set value is a negative number; Determine a second function, where the second function is a piecewise constant function in which the reward value is a second set value when the robot arm reaches the position of the target point, and the reward value is 0 when the robot arm does not reach the position of the target point, and the second set value is a positive number; The reward function is obtained by adding the relative distance, the first function and the second function.

6. The method according to claim 1, characterized in that Iterative training is performed using a deep deterministic policy gradient algorithm based on the state space, the action space, and the reward function to obtain a target policy model, including: Initialize the evaluation network, action network, target evaluation network and target action network; A generating step, inputting the current sample state into the action network, so that the action network generates an optimal action according to the current sample state, wherein the current sample state space is any sample state in the training sample set; An execution step, wherein the evaluation network executes the optimal action and evaluates based on the reward function to obtain an evaluation score; An updating step, using a gradient descent method to update the parameters in the evaluation network, the action network, the target evaluation network and the target action network; Repeating step, repeating the generating step, the executing step and the updating step in sequence at least once, until the end of the robot arm reaches the set position, and taking the latest evaluation score as the cumulative reward value of the current iteration; The generating step, the executing step, the updating step and the repeating step are sequentially repeated at least once until the number of iterations reaches a set maximum number of iterations or the cumulative reward value reaches a maximum value, thereby obtaining the target strategy model.

7. The method according to claim 6, characterized in that The parameters in the evaluation network, the action network, the target evaluation network and the target action network are updated using a gradient descent method, including: The third formula is minimized to update the evaluation network parameters in the evaluation network. The third formula is: N represents the total number of samples selected for learning from the training sample set, R i is the reward score calculated by the reward function, γ represents the discount parameter, Q(S i ,A i |θ Q ) indicates that in the current sample state S i Execute the optimal action A i The evaluation score obtained after that, Q′((S i+1 |θ u′ )|θ Q′ ) indicates that in the current sample state S i The target score after executing the action, θ Q is the evaluation network parameter, θ Q′ is the target evaluation network parameter, θ u′ is the target action network parameter, S i is the current sample state, A i is the optimal action; The action network parameters in the action network are updated according to the fourth formula, and the fourth formula is: θ u represents the action network parameters, u(S|θ u ) represents the action network, Q(S,A|θ Q ) represents the evaluation network; The target evaluation network parameters in the target evaluation network are updated according to the fifth formula, and the fifth formula is θ Q′ ←τθ Q +(1-τ)θ Q′ , τ represents the soft update parameter; The target action network parameters in the target action network are updated according to the sixth formula, and the sixth formula is θ u′ ←τθ u +(1-τ)θ u′ .

8. A trajectory planning device for collision avoidance of an equipotential operation robot arm, characterized in that: The device comprises: A determination unit, used to determine a state space, an action space and a reward function of the robot arm, wherein the state space is a state set of the environmental state of the working space where the robot arm is located and the state of the robot arm itself, the action space is a set of actions of the robot arm moving in space, and the reward function is a function that takes into account the safe working space of the robot arm under live working and the requirement of the robot arm to avoid dangerous space on the basis of ensuring that the end of the robot arm moves to the target point position; A training unit, configured to perform iterative training using a deep deterministic policy gradient algorithm based on the state space, the action space, and the reward function to obtain a target policy model; An acquisition unit, used for acquiring a target point position of the robotic arm, inputting the target point position into the target strategy model, and obtaining an optimal moving path of the robotic arm; A control unit is used to control the robotic arm to move to the target point position according to the optimal moving path.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored program, wherein when the program is executed, the device where the computer-readable storage medium is located is controlled to execute the method according to any one of claims 1 to 7.

10. A computer program product comprising computer instructions, characterized in that: When the computer instructions are executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images