Steel structural member robot path planning method and device and electronic equipment

Abstract

The invention provides a steel structural member robot path planning method and device and electronic equipment, and the method comprises the steps: obtaining the task environment information of a robot, including the initial state, the target state and the obstacle distance corresponding to the robot; the task environment information is input into a path planning model, execution actions of the robot are predicted according to the task environment information, the path planning model is obtained by adjusting model parameters according to reward values obtained after the robot executes the actions, and the model parameters comprise decision network parameters and evaluation network parameters; determining the next state of the robot according to the initial state and the execution action, and continuously predicting the subsequent state of the robot by taking the next state as the current state; and when the follow-up state is the target state, the path of the robot is determined, so that the robot completes welding of the steel structural part according to the path. Intelligent obstacle avoidance in the welding process of the robot is achieved, and therefore the welding quality and efficiency of the steel structural part are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of path planning, and in particular to a steel structure robot path planning method, device and electronic equipment. Background Art

[0002] With the development of construction technology and material science, steel structure buildings have been widely used due to their light weight, high strength, good seismic performance and fast construction speed. The main frame of steel structure parts is composed of welded steel, rolled steel columns, ribs, support brackets, connecting brackets, etc., and the structure is relatively complex. In recent years, the proportion of steel structure buildings has gradually increased, and its forms have become increasingly diverse, and the demand for steel structure parts has also increased.

[0003] At present, the assembly and welding of steel structures mainly rely on manual operation. It is necessary to first assemble the parts such as the corbels and ribs, and then weld them together with the steel columns. This welding process involves multiple complex steps, including grabbing, spot welding, flipping, etc., which are difficult to operate and require auxiliary lifting and flipping, especially for large-sized and large-tonnage steel components. At the same time, during the welding process, welding at corners, seams and other locations is also difficult, resulting in low overall welding efficiency and unstable welding quality. In order to ensure the welding efficiency and quality of steel structures, it is an inevitable trend to use robots for automated welding of steel structures. However, when using robots for automatic welding, path planning of the robot is crucial. If the welding path is not planned well, it is not only easy to collide with the welding workpiece, affecting the welding quality, but also the welding efficiency cannot be guaranteed.

[0004] Therefore, it is necessary to propose a robot welding path planning scheme to enable the robot to avoid obstacles, so as to improve the welding quality and efficiency, thereby promoting the coordinated development of digitalization, intelligence and greening of the steel structure industry. Summary of the invention

[0005] In order to provide a basic understanding of some aspects of the disclosed embodiments, a brief summary is given below. The summary is not an extensive review, nor is it intended to identify key / critical components or delineate the scope of protection of these embodiments, but rather serves as a prelude to the detailed description that follows.

[0006] In view of the above-mentioned shortcomings of the prior art, the present invention discloses a steel structure robot path planning method, device and electronic equipment, which are used to solve the technical problem in the prior art that the welding path design of the steel structure robot cannot meet the welding quality and efficiency requirements.

[0007] In the first aspect, the present application provides a robot path planning method for steel structure parts, the method comprising: obtaining task environment information of the robot, the task environment information including a starting state, a target state and an obstacle distance corresponding to the robot; inputting the task environment information into a path planning model to predict the execution action of the robot according to the task environment information, the path planning model adjusting model parameters according to a reward value obtained after the robot executes the action, the model parameters including decision network parameters and evaluation network parameters corresponding to the robot action; determining the next state of the robot according to the starting state and the execution action, and continuously predicting the subsequent state of the robot using the next state as the current state; until the subsequent state is the target state, determining the path of the robot so that the robot completes the welding of steel structure parts according to the path.

[0008] In one embodiment of the present invention, the method for obtaining the path planning model includes: constructing a reinforcement learning model, the reinforcement learning model includes a decision network and an evaluation network corresponding to the robot action, the decision network is used to generate a motion strategy, including a current decision network and a target decision network, and the evaluation network is used to evaluate the motion strategy, including a current evaluation network and a target evaluation network; using the current decision network and the current evaluation network to interact with the robot and the task environment to obtain batch quadruple data, the quadruple data is represented as (S, a, r, S′), wherein S represents the current state of the robot, a represents the execution action of the robot in the current state, r represents the reward value of whether a collision occurs in the process of the robot performing the action in the current state to obtain the next state, and S′ represents the next state of the robot after performing the action in the current state; training the current decision network and the current evaluation network according to the batch quadruple data, and updating the decision network parameters and the evaluation network parameters; using the updated decision network parameters and the updated evaluation network parameters to update the target decision network and the target evaluation network until the preset training termination condition is reached, and the path planning model is obtained.

[0009] In one embodiment of the present invention, the current decision network includes a first fully connected layer, a convolutional neural network layer and a second fully connected layer, and the current evaluation network includes a third fully connected layer, a fourth fully connected layer and a fifth fully connected layer; the use of the current decision network and the current evaluation network to interact the robot with the task environment includes: using the first fully connected layer and the convolutional neural network layer to respectively learn the robot's position features and the distance features of the obstacles, and using the second fully connected layer to learn the robot's spatial features based on the position features and the distance features, thereby describing the robot's actions; using the third fully connected layer and the fourth fully connected layer to process the robot's state and execution action, respectively, and using the fifth fully connected layer to learn the evaluation features of the processed state and execution action, thereby describing the reward obtained by executing the action.

[0010] In one embodiment of the present invention, the method for obtaining the four-tuple data includes: setting a reinforcement learning component, the component including a state component, an action component and a reward function component, the state component is represented by S t =(P t-1 , P t , P e , D t ), where p t-1 Indicates the previous state of the robot, p t Indicates the current state of the robot, p e is the target state of the robot, D t represents the obstacle distance, and the action component is represented by a t =(α t , β t , d t ), where α t represents the horizontal movement angle of the robot in the radar coordinate system, β t represents the vertical movement angle of the robot in the radar coordinate system, α t With β t Together they represent the direction of movement of the robot, d t Indicates the moving distance of the robot in the moving direction, α t , β t d t All are continuous variables, α t With β t The range is [-60°, 60°], d t The range is [0, d max ],d maxis a preset single maximum moving distance; using the current decision network, extracting spatial features of the state component, and determining the execution action in the four-tuple data according to the spatial features and the action component; determining the next state in the four-tuple data according to the current state in the state component and the execution action in the four-tuple data; using the current evaluation network, processing the current state in the state component and the execution action in the four-tuple data, and using the reward function component to calculate the reward value in the four-tuple data.

[0011] In one embodiment of the present invention, the training of the current decision network and the current evaluation network according to the batch quadruple data, and updating of the decision network parameters and the evaluation network parameters include: inputting the next state in the quadruple data into the current decision network to obtain the next execution action; inputting the current state and execution action in the quadruple data into the current evaluation network to calculate the current reward value, and inputting the next state and the next execution action in the quadruple data into the current evaluation network to calculate the next reward value; with the goal of maximizing the current reward value, calculating the first gradient of the current reward value with respect to the decision network parameters, and updating the decision network parameters according to the first gradient; calculating the next reward value according to the preset instant reward and discount factor to obtain the target reward value, and with the goal of minimizing the loss between the current reward value and the target reward value, calculating the second gradient of the loss function with respect to the evaluation network parameters, and updating the evaluation network parameters according to the second gradient.

[0012] In one embodiment of the present invention, the target decision network and the target evaluation network are updated using the updated decision network parameters and the updated evaluation network parameters, including: calculating the decision network parameters, the updated decision network parameters and a preset learning rate to obtain target decision network parameters, and calculating the evaluation network parameters, the updated evaluation network parameters and the learning rate to obtain target evaluation network parameters; assigning the target decision network parameters to the target decision network to complete the update of the target decision network, and assigning the target evaluation parameters to the target evaluation network to complete the update of the target evaluation network.

[0013] In one embodiment of the present invention, the obstacle distance is an obstacle distance matrix D measured in multiple directions. t ; The calculation formula of the reward value is:

[0014]

[0015] Among them, s t Represents the current state of the robot; at Represents the actions performed by the robot in the current state; t+1 Represents the next state of the robot after performing an action in the current state; r t (s t ,a t ,s t+1 ) represents the reward value of whether a collision occurs when the robot performs an action in the current state to obtain the next state; R is the reward value of a collision when the robot performs an action in the current state to obtain the next state; L t Represents the distance from the current state of the robot to the target state; L t+1 Indicates the distance from the next state to the target state of the robot; mean(D t+1 ) represents the average value of the obstacle distance matrix of the robot in the next state; mean(D t ) represents the average value of the obstacle distance matrix of the robot in the current state; e is the preset minimum value.

[0016] In one embodiment of the present invention, the obstacle distance matrix D t The method for obtaining the obstacle monitoring angle range includes: dividing a preset obstacle monitoring angle range into a plurality of range sub-intervals, each of the range sub-intervals representing a monitoring direction; setting a direction-distance matrix, and detecting whether there is an obstacle in each of the monitoring directions; if there is an obstacle in any monitoring direction, detecting the obstacle distance corresponding to the arbitrary monitoring direction and recording it in the direction-distance matrix; if there is no obstacle in any monitoring direction, setting the obstacle distance corresponding to the arbitrary monitoring direction in the direction-distance matrix to a preset distance threshold.

[0017] In the second aspect, the present application provides a robot path planning device for steel structure parts, the device comprising: an information acquisition module, used to acquire the robot's task environment information, the task environment information including the robot's corresponding starting state, target state and obstacle distance; an action prediction module, used to input the task environment information into a path planning model to predict the robot's execution action based on the task environment information, the path planning model is obtained by adjusting model parameters according to the reward value obtained after the robot executes the action, the model parameters including decision network parameters and evaluation network parameters corresponding to the robot action; a path prediction module, used to determine the next state of the robot based on the starting state and the execution action, and use the next state as the current state to continuously predict the subsequent state of the robot; a path determination module, used to determine the robot's path until the subsequent state is the target state, so that the robot completes the welding of steel structure parts according to the path.

[0018] In the third aspect, an electronic device is provided, characterized in that the electronic device comprises: one or more processors; a storage device for storing one or more programs, and when the one or more programs are executed by the one or more processors, the electronic device implements the steel structure robot path planning method described in the first aspect.

[0019] As described above, the steel structure robot path planning method, device and electronic equipment provided by the embodiments of the present invention have the following beneficial effects:

[0020] Firstly, the task environment information of the robot is obtained, which includes the robot's corresponding starting state, target state and obstacle distance. Then, the task environment information is input into the path planning model to predict the robot's next execution action according to the task environment information. The path planning model adjusts the decision network parameters and the evaluation network parameters in the model according to the reward value obtained after the robot executes the action, so that the execution action can effectively avoid obstacles and move towards the target state. Then, the next state of the robot is determined according to the starting state and the execution action, and the next state is used as the current state to continuously predict the subsequent state of the robot. When the subsequent state is the target state, the robot's path is formed based on each state. Through the path planning model including the decision network and the evaluation network of the execution action, the robot path planning is performed based on the purpose of avoiding obstacles and reaching the end point as soon as possible, so that the robot can reach the target state from the starting state without collision and efficiently according to the path, complete the welding of steel structures, realize intelligent obstacle avoidance in the robot welding process, thereby improving the welding quality and efficiency of steel structures.

[0021] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The drawings herein are incorporated into the specification and constitute a part of the specification, showing embodiments consistent with the present application, and together with the specification, are used to explain the principles of the present application. Obviously, the drawings described below are only some embodiments of the present application, and for those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative work. In the drawings:

[0023] Figure 1 It is a schematic diagram of an implementation environment of a steel structure robot path planning device shown in an exemplary embodiment of the present application;

[0024] Figure 2 is a flow chart of a steel structure robot path planning method shown in an exemplary embodiment of the present application;

[0025] Figure 3 is a schematic diagram of a robot task environment simulation platform shown in an exemplary embodiment of the present application;

[0026] Figure 4 is a flowchart of a reinforcement learning algorithm shown in an exemplary embodiment of the present application;

[0027] Figure 5 is a block diagram of a steel structure robot path planning device shown in an exemplary embodiment of the present application;

[0028] Figure 6 is a block diagram of a steel structure assembly and welding system shown in an exemplary embodiment of the present application;

[0029] Figure 7 It is a structural schematic diagram of an electronic device shown as an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0030] The following will describe the embodiments of the present invention with reference to the accompanying drawings and preferred embodiments. Those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be understood that the preferred embodiments are only for illustrating the present invention, not for limiting the scope of protection of the present invention.

[0031] It should be noted that the illustrations provided in the following embodiments are only used to illustrate the basic concept of the present invention in a schematic manner, and thus the illustrations only show components related to the present invention rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed arbitrarily, and the component layout may also be more complicated.

[0032] In the following description, numerous details are discussed to provide a more thorough explanation of the embodiments of the present invention. However, it is obvious to those skilled in the art that the embodiments of the present invention can be implemented without these specific details. In other embodiments, well-known structures and devices are shown in the form of block diagrams rather than in detail to avoid making the embodiments of the present invention difficult to understand.

[0033] In the assembly and welding of steel structures, there are many complex steps involved, including grabbing, spot welding, flipping, etc., which are difficult to operate. At the same time, in the welding process, welding at corners, gaps and other locations is also difficult, which makes the overall welding efficiency low and the welding quality unstable. The use of robots for automated welding of steel structures has guaranteed the welding efficiency and quality of steel structures to a certain extent. However, the inventors of this application have found that the use of robots for automatic welding is crucial for the path planning of the robots. If the welding path is not planned well, it is not only easy to collide with the welding workpiece, affecting the welding quality, but also the welding efficiency cannot be guaranteed.

[0034] Therefore, see Figure 1 , Figure 1 FIG. 1 is a schematic diagram of an implementation environment of a steel structure robot path planning device according to an exemplary embodiment of the present application. Figure 1 As shown, the implementation environment may include a steel structure robot path planning device 110 and a computer device 120. The steel structure robot path planning device 110 may be set in the computer device 120 to plan the path of the steel structure robot. The computer device 120 may be at least one of a desktop graphics processing unit (GPU) computer, a GPU computing cluster, a neural network computer, etc. The steel structure robot path planning device 110 uses a path planning model including a decision network and an evaluation network for executing actions to plan the robot path based on the purpose of avoiding obstacles and reaching the end point as quickly as possible, so that the robot can reach the target state from the starting state without collision and efficiently according to the path, complete the welding of the steel structure, and realize intelligent obstacle avoidance during the robot welding process, thereby improving the welding quality and efficiency of the steel structure.

[0035] See also Figure 2 , Figure 2 is a flow chart of a steel structure robot path planning method shown in an exemplary embodiment of the present application. The method can be applied to Figure 1 The implementation environment shown is specifically implemented by the steel structure robot path planning device in the implementation environment. It should be understood that the method can also be applied to other exemplary implementation environments and be specifically implemented by devices in other implementation environments. This embodiment does not limit the implementation environment to which the method is applicable.

[0036] like Figure 2 As shown, in an exemplary embodiment, the steel structure robot path planning method includes at least steps S210 to S240, which are described in detail as follows:

[0037] Step S210, obtaining the task environment information of the robot, where the task environment information includes the robot's corresponding starting state, target state and obstacle distance.

[0038] Among them, the task environment refers to the entire environment in which the robot is welding steel structures; the starting state refers to the starting position of the robot; the target state refers to the position of the steel structure, that is, the target position that the robot needs to reach; the obstacle distance refers to the distance between the obstacle and the robot. The robot will avoid obstacles and move from the starting state to the target state.

[0039] In a possible embodiment, before obtaining the task environment information of the robot, it also includes: analyzing the various equipment included in the steel structure welding environment, and obtaining the specifications and dimensions of each equipment, wherein the equipment includes multiple robots, robot loading mechanisms, robot loading mechanism moving tracks, steel structure loading mechanisms and other auxiliary equipment; modeling according to the layout position of each equipment, and building a task environment simulation platform, wherein each device in the task environment simulation platform can control its movement in the form of program instructions.

[0040] See also Figure 3 , Figure 3 FIG. 1 is a schematic diagram of a robot task environment simulation platform shown in an exemplary embodiment of the present application. Figure 3 As shown, the steel structure is loaded in the corresponding loading mechanism, the robot for welding the steel structure is loaded on the corresponding loading structure, and a corresponding moving track is provided to realize the movement of the robot. It should be understood that Figure 3 Only one set of welding mechanisms in the task environment simulation platform is shown.

[0041] It should also be noted that the robot can weld the steel structure loaded on the corresponding steel structure loading mechanism, and the steel structure has at least one welding position.

[0042] As a possible embodiment, in the task environment simulation platform, obstacles include various equipment and steel structures in the entire task environment. It should be noted that during the welding process, the robot not only needs to avoid colliding with various equipment, but also needs to avoid hitting steel structures, especially large-sized and complex steel structures, and the robot's path needs to be planned.

[0043] In a possible embodiment, various states of the robot are expressed in the form of position coordinates, that is, the starting state is the starting position coordinates, the target state is the target position coordinates, each coordinate is a three-dimensional coordinate in space, and the obstacle distance is measured by a radar in the task environment.

[0044] As a possible implementation example, when building a task environment simulation platform, a robot three-dimensional coordinate system with the robot as the origin, a radar three-dimensional coordinate system with the radar as the origin, and a workpiece three-dimensional coordinate system with the steel structure as the origin are established.

[0045] It should be noted that the directions of the X-axis, Y-axis and Z-axis of the robot 3D coordinate system, the radar 3D coordinate system and the workpiece 3D coordinate system may be the same or different. In addition, it is necessary to calculate the conversion relationship between the radar 3D coordinate system and the robot 3D coordinate system, i.e., the first conversion matrix, and it is necessary to calculate the conversion relationship between the workpiece 3D coordinate system and the robot 3D coordinate system, i.e., the second conversion matrix.

[0046] As a possible embodiment, a radar is provided on each robot end (i.e., welding end), for example, the longitudinal direction of the radar is set to be the same as the end direction of the robot. Based on the radar three-dimensional coordinate system, the radar detects and obtains the obstacle distance. At this time, the detected obstacle distance is the distance between the end and the obstacle, so it can be converted into the distance between the robot and the obstacle according to the above-mentioned first conversion matrix. It should be understood that when the radar detects the obstacle distance, the obstacle included is based on the direction of the radar three-dimensional coordinate system.

[0047] Step S220 , inputting the task environment information into the path planning model to predict the execution action of the robot according to the task environment information.

[0048] Among them, the path planning model is obtained by adjusting the model parameters according to the reward value obtained after the robot performs the action. The model parameters include the decision network parameters and evaluation network parameters corresponding to the robot action.

[0049] It should be noted that the path planning model includes a decision network and an evaluation network. The decision network is provided with decision network parameters, and the evaluation network is provided with evaluation network parameters. The decision network is used to decide on an action to avoid obstacles under a given state, and the evaluation network is used to evaluate the quality of the action, that is, to give a reward value for whether the action encounters an obstacle. The larger the reward value, the less likely it is that the robot will collide with an obstacle when performing the action.

[0050] As a possible embodiment, the execution action can be represented by three-dimensional coordinates (α, β, d), where α and β represent angles, that is, the moving direction based on the radar coordinate system, which can be set to counterclockwise rotation angle as positive and clockwise rotation angle as negative, and d represents the moving distance in this direction. In this way, the path of the robot is composed of at least one execution action, that is, at least one three-dimensional coordinate.

[0051] In one embodiment, a path planning model is obtained by: constructing a reinforcement learning model, the reinforcement learning model includes a decision network and an evaluation network corresponding to the robot action, the decision network is used to generate a motion strategy, including a current decision network and a target decision network, and the evaluation network is used to evaluate the motion strategy, including a current evaluation network and a target evaluation network; using the current decision network and the current evaluation network to interact with the robot and the task environment to obtain batch four-tuple data, the four-tuple data including the current state, the execution action, the reward value and the next state; training the current decision network and the current evaluation network according to the batch four-tuple data, and updating the decision network parameters and the evaluation network parameters; using the updated decision network parameters and the updated evaluation network parameters to update the target decision network and the target evaluation network until the preset training termination condition is reached, and the path planning model is obtained.

[0052] In this embodiment, the path planning model is a model based on DDPG (Deep Deterministic Policy Gradient). When constructing a reinforcement learning model, the current decision network and the target decision network are the same at the beginning of the design, and the current evaluation network and the target evaluation network are also the same at the beginning of the design. By training the current decision network and the current evaluation network, the parameters of the two networks are updated, and then the parameters of the target decision network and the target evaluation network are updated respectively using the parameters of the two networks. In this way, by setting the current decision network, the target decision network, the current evaluation network, and the target evaluation network to perform model training, the model learning process can be stabilized, the fluctuations in the training process can be reduced, and the learning stability can be improved.

[0053] In this embodiment, the robot interacts with the task environment, that is, the robot is regarded as an intelligent agent, and the steel structure and other auxiliary equipment constitute the environment of reinforcement learning, which describes all possible situations that the robot can encounter when performing actions.

[0054] In this embodiment, the training termination condition is that the number of training times or the reward value converges to an error threshold.

[0055] In addition, in this embodiment, the four-tuple data includes the current state, the executed action, the reward value and the next state. The four-tuple data is represented as (S, a, r, S′), wherein S represents the current state of the robot, a represents the executed action of the robot in the current state, r represents the reward value of whether a collision occurs in the process of the robot executing the action in the current state to obtain the next state, and S′ represents the next state of the robot after executing the action in the current state.

[0056] In one embodiment, the current decision network includes a first fully connected layer, a convolutional neural network layer and a second fully connected layer, and the current evaluation network includes a third fully connected layer, a fourth fully connected layer and a fifth fully connected layer; the current decision network and the current evaluation network are used to interact with the robot and the task environment, including: using the first fully connected layer and the convolutional neural network layer to respectively learn the position characteristics of the robot and the distance characteristics of the obstacles, and using the second fully connected layer to learn the spatial characteristics of the robot based on the position characteristics and the distance characteristics, thereby describing the robot's actions; using the third fully connected layer and the fourth fully connected layer to process the robot's state and execution action respectively, and using the fifth fully connected layer to learn the evaluation characteristics of the processed state and execution action, thereby describing the reward obtained by executing the action.

[0057] In this embodiment, the current decision network includes a first fully connected layer, a convolutional neural network layer and a second fully connected layer. Specifically, the first fully connected layer is used to learn the position features of the robot, the convolutional neural network layer is used to learn the distance features between the robot and the obstacle, and the second fully connected layer is used to learn the spatial features of the robot according to the position features and the distance features, so as to achieve the description of the robot's action. That is, the output dimension of the current decision network is 3, which is expressed as (α t , β t , d t ), where α t With β t represents the angle based on the radar coordinate system, d t It represents the distance between the robot and the obstacle in the radar coordinate system; the current evaluation network includes the third fully connected layer, the fourth fully connected layer and the fifth fully connected layer. Specifically, the third fully connected layer is used to process the state of the robot, the fourth fully connected layer is used to process the execution action of the robot, and the fifth fully connected layer is used to learn the evaluation features of the processed state and execution action, so as to realize the evaluation description of the execution action, that is, the output dimension of the current evaluation network is 1, which represents the reward value of performing the execution action under the current state.

[0058] In this way, the robot interacts with the task environment through the various hierarchical structures in the current decision network and the current evaluation network to perform feature learning, and the neural network can be used to efficiently and accurately describe the robot's actions and evaluate the robot's actions.

[0059] In one embodiment, the method for obtaining the four-tuple data includes: setting a reinforcement learning component, the component includes a state component, an action component and a reward function component, and the state component is represented by S t =(P t-1 , P t , P e , D t ), where pt-1 Indicates the previous state of the robot, p t Indicates the current state of the robot, p e is the target state of the robot, D t Represents the obstacle distance, and the action component is represented by a t =(α t , β t , d t ), where α t represents the horizontal movement angle of the robot in the radar coordinate system, β t represents the vertical movement angle of the robot in the radar coordinate system, α t With β t Together they represent the direction of movement of the robot, d t Indicates the moving distance of the robot in the moving direction, α t , β t d t All are continuous variables, α t With β t The range is [-60°, 60°], d t The range is [0, d max ],d max is the preset single maximum moving distance; using the current decision network, the spatial features of the state component are extracted, and the execution action in the four-tuple data is determined based on the spatial features and the action component; according to the current state in the state component and the execution action in the four-tuple data, the next state in the four-tuple data is determined; using the current evaluation network, the current state in the state component and the execution action in the four-tuple data are processed, and the reward function component is used to calculate the reward value in the four-tuple data.

[0060] In this embodiment, for the intelligent robot, three components of the Markov decision process in reinforcement learning are designed: state, action, and reward function.

[0061] Among them, the state S t =(P t-1 , P t , P e , D t ), p t is the current position coordinate of the robot, that is, the current state, p t-1 is the position coordinate of the robot at the previous moment, that is, the previous state, p e is the target position coordinate of the robot, that is, the target state, D t Indicates the distance between the robot and the obstacle; action a t =(α t , β t , d t ), αt With β t represents the moving direction, α t represents the horizontal movement angle of the robot in the radar coordinate system, β t represents the vertical movement angle of the robot in the radar coordinate system, d t represents the moving distance in the moving direction, α t With β t The angle range can be set according to the specific situation of the task environment, for example, it can be set to [-60°, 60°], d t The range can be set according to the specific situation of the task environment, for example, it can be set to [0, d max ],d max is the preset maximum single moving distance; the reward function is r t (s t , a t ,s t+1 ), characterized in s t Execute action a in state t Reaching state s t+1 The reward value at that time.

[0062] In addition, the robot is initially at the starting point p s , since there is no state at the previous moment, the position coordinates at the previous moment are set to a point with a unit length in the opposite direction from the starting position to the target position, as initialization, and are set to: Among them, p s Indicates the position coordinates of the robot at the starting point, p e Indicates the target location coordinates.

[0063] It should be noted that the current decision network includes a first fully connected layer, a convolutional neural network layer and a second fully connected layer, and the current evaluation network includes a third fully connected layer, a fourth fully connected layer and a fifth fully connected layer.

[0064] In this embodiment, the current decision network is used to extract spatial features of the state component, and the execution action in the quadruple data is determined based on the spatial features and the action component. Specifically, in the current decision network, the first fully connected network is used to extract P in the state component. t-1 , P t , P e The feature 1 of the three coordinates, that is, the position feature of the robot, is then extracted using a convolutional neural network. t The feature feature2, that is, the distance feature between the robot and the obstacle, is finally connected to feature1 and feature2 using the second fully connected network to extract the spatial features of the robot, realize the description of the robot state by the action component, and output the action (αt , β t , d t ), thereby determining the execution action in the quadruple data.

[0065] In this embodiment, the next state in the four-tuple data is determined according to the current state in the state component and the execution action in the four-tuple data. Specifically, at the current position coordinate, according to the execution action (α t , β t , d t ) is changed to obtain the position coordinates at the next moment, thereby determining the next state in the four-tuple data.

[0066] In this embodiment, the current evaluation network is used to process the current state in the state component and the execution action in the four-tuple data, and the reward function component is used to calculate the reward value in the four-tuple data. Specifically, the third fully connected layer of the 2-layer is used to evaluate the robot's s t Processing, using the fourth fully connected layer of the 2nd layer to the robot's a t Processing and using the fifth fully connected layer of 2 layers to process the processed s t with a t The splicing is performed and the evaluation features are learned. The activation function can use the ReLU function (Rectified Linear Unit) to realize the evaluation description of the execution action by the reward function component and output s t Next t The reward value of is determined, thereby determining the reward value in the quadruple data.

[0067] In this embodiment, the current state, the executed action, the next state and the reward value are integrated to form a four-tuple data.

[0068] It should also be noted that, in this embodiment, the current decision network and the current evaluation network are continuously used to allow the robot to interact with the environment, obtain the state S′ and reward r at the next moment, and put the obtained four-tuple data (S, a, r, S′) into the experience pool.

[0069] In one embodiment, the obstacle distance is an obstacle distance matrix D measured in multiple directions. t .

[0070] Specifically, the obstacle distance matrix D tThe method for obtaining the obstacle monitoring angle range includes: dividing a preset obstacle monitoring angle range into a plurality of range sub-intervals, each range sub-interval representing a monitoring direction; setting a direction-distance matrix, and detecting whether there is an obstacle in each monitoring direction; if there is an obstacle in any monitoring direction, detecting the obstacle distance corresponding to any monitoring direction and recording it in the direction-distance matrix; if there is no obstacle in any monitoring direction, setting the obstacle distance corresponding to any monitoring direction in the direction-distance matrix to a preset distance threshold.

[0071] In this embodiment, for example, the obstacle monitoring angle range is -60° to 60° in the X and Y axis directions of the radar three-dimensional coordinate system, and the range is divided into 13 range sub-intervals with a grid of 10° each. The direction-distance matrix is ​​a 13×13 matrix, and the values ​​in the matrix are the distances of obstacles detected in front of this direction.

[0072] In this way, by detecting obstacles in multiple directions through radar, multi-directional obstacle monitoring is achieved for the robot, which can effectively avoid collisions of the robot.

[0073] Furthermore, the calculation formula of the reward value is:

[0074]

[0075] Among them, s t Represents the current state of the robot; a t Represents the actions performed by the robot in the current state; t+1 Represents the next state of the robot after performing an action in the current state; r t (s t ,a t ,s t+1 ) represents the reward value of whether a collision occurs when the robot performs an action in the current state to obtain the next state; R is the reward value of a collision when the robot performs an action in the current state to obtain the next state; L t Represents the distance from the current state of the robot to the target state; L t+1 Indicates the distance from the next state to the target state of the robot; mean(D t+1 ) represents the average value of the obstacle distance matrix of the robot in the next state; mean(D t ) represents the average value of the obstacle distance matrix of the robot in the current state; e is the preset minimum value.

[0076] In this embodiment, if the robot collides when executing an action from the current state to the next state, the reward value is a preset value, such as -1000. If the robot does not collide when executing an action from the current state to the next state, the reward value is ΔL plus the difference of the safety distance, where ΔL is p t to p e The distance L t Subtract p t+1 to p e The distance L t+1 ), the difference in safety distance is Where e is a minimum value set to prevent the denominator from being 0, and the mean() function represents the calculation of the average value of a set of data.

[0077] In this way, maximizing L t -L t+1 ,improve Reduce The robot moves closer to the target position by maximizing the forward distance and avoids collision by maximizing the distance from obstacles. This ensures that the robot's actions will not cause collisions and will get closer to the target position, thereby ensuring the reliability of the path planning model.

[0078] In one embodiment, the current decision network and the current evaluation network are trained according to batch quadruple data, and the decision network parameters and the evaluation network parameters are updated, including: inputting the next state in the quadruple data into the current decision network to obtain the next execution action; inputting the current state and execution action in the quadruple data into the current evaluation network to calculate the current reward value, and inputting the next state and the next execution action in the quadruple data into the current evaluation network to calculate the next reward value; with the goal of maximizing the current reward value, calculating the first gradient of the current reward value with respect to the decision network parameters, and updating the decision network parameters according to the first gradient; calculating the next reward value according to the preset instant reward and discount factor to obtain the target reward value, and with the goal of minimizing the loss between the current reward value and the target reward value, calculating the second gradient of the loss function with respect to the evaluation network parameters, and updating the evaluation network parameters according to the second gradient.

[0079] In this embodiment, the priority experience replay strategy is used to continuously extract samples (S, a, r, S') from the experience pool to train the current decision network and the current evaluation network. Input S and a in (S, a, r, S') into the current evaluation network to obtain the reward value of the current execution action, that is, the current reward value Q(S, a), and then input S' into the current decision network to obtain action a ′ , put S′, a ′ Input into the current evaluation network to get a ′ The reward value, that is, the next reward value Q(S′,a′ ).

[0080] In this embodiment, let the decision network parameter be w, which represents all parameters in the current decision network. With the goal of maximizing the current reward value, calculate the first gradient of the current reward value with respect to the decision network parameter, and update the decision network parameter according to the first gradient, that is, update the current decision network to maximize the output Q(S, a), and use the composite function derivation rule to find (the gradient of the reward value Q to action a) × (the gradient of action a to parameter w), that is:

[0081]

[0082] Then, the decision network parameters are updated according to the first gradient obtained using the gradient ascent method.

[0083] In this embodiment, the evaluation network parameter is θ, which represents all parameters in the current evaluation network. The next reward value, the preset instant reward and the preset discount factor are calculated to obtain the target reward value, and the calculation formula of the target reward value is:

[0084] Q ′ =r+γQ(S′,a ′ ) Formula (3)

[0085] Among them, Q ′ represents the target reward value, r represents the immediate reward, γ represents the discount factor, Q(S′, a ′ ) represents the next reward value.

[0086] In addition, the loss function between the current reward value and the target reward value is:

[0087] L = (Q(S, a) - Q ′ ) 2 Formula (4)

[0088] Then, with the goal of minimizing the loss, the back propagation algorithm is used to calculate the second gradient of the loss function with respect to the evaluation network parameters, and finally the gradient descent method is used to update the evaluation network parameters according to the second gradient.

[0089] In this embodiment, based on reinforcement learning, the reliability of the path planning of the path planning model is guaranteed, thereby improving the reliability of the robot's intelligent obstacle avoidance.

[0090] In one embodiment, the target decision network and the target evaluation network are updated using the updated decision network parameters and the updated evaluation network parameters, including: calculating the decision network parameters, the updated decision network parameters and the preset learning rate to obtain the target decision network parameters, and calculating the evaluation network parameters, the updated evaluation network parameters and the learning rate to obtain the target evaluation network parameters; assigning the target decision network parameters to the target decision network to complete the update of the target decision network, and assigning the target evaluation parameters to the target evaluation network to complete the update of the target evaluation network.

[0091] In this embodiment, all parameters of the current decision network and the target decision network are represented by w and w respectively. ′ Indicates that all parameters of the current evaluation network and the target evaluation network are θ and θ respectively. ′ Indicates. A soft update method is adopted, that is, a learning rate τ is introduced, for example, τ = 0.1. The learning rate can be set according to the specific situation. The decision network parameters in the old target decision network and the new corresponding network parameters are weighted and then assigned to the target decision network. In this way, the stability of parameter update is guaranteed by soft update, thereby ensuring the accuracy of model training.

[0092] In this embodiment, the calculation formula of the target evaluation network parameter is:

[0093] w″=τw+(1-τ)w′ Formula (5)

[0094] Among them, w″ represents the target evaluation network parameters, τ represents the learning rate, w represents the decision network parameters, and w′ represents the updated decision network parameters.

[0095] The calculation formula of the target decision network parameters is:

[0096] θ″=τθ+(1-τ)θ′ Formula (6)

[0097] Among them, θ″ represents the target decision network parameters, τ represents the learning rate, θ represents the evaluation network parameters, and θ′ represents the updated evaluation network parameters.

[0098] In a possible embodiment, the target decision network and the target evaluation network are updated based on a preset update frequency. For example, the update frequency is set to 10, that is, the current decision network and the current evaluation network are updated 10 times, and the target decision network and the target evaluation network are updated once. In this way, the stability of parameter updates can be guaranteed, thereby ensuring the accuracy of model training.

[0099] Step S230, determining the next state of the robot according to the initial state and the execution action, and taking the next state as the current state to continuously predict the subsequent state of the robot.

[0100] In this embodiment, first, the next state is continuously used as the current state s t Predict the corresponding execution action a t , and then according to the current state s t and perform action a t Determine the follow-up status.

[0101] Step S240, until the subsequent state is the target state, determine the path of the robot so that the robot completes the welding of the steel structure according to the path.

[0102] In this embodiment, each time a subsequent state is obtained, the subsequent state is matched with the target state until the subsequent state and the target state can match each other. The path of the robot is determined based on all previous state points or execution actions, so that the robot can reach the specified target point from the starting point without collision according to the path, thereby completing the welding task and realizing the intelligent and automatic welding of steel structures.

[0103] In this way, the welding efficiency and quality are improved, which can further promote the coordinated development of digitalization, intelligence and greening of the steel structure industry.

[0104] See also Figure 4 , Figure 4 is a flowchart of a reinforcement learning algorithm shown in an exemplary embodiment of the present application. Figure 4 As shown in the figure, based on the reinforcement learning model, four neural networks are first constructed, namely the current decision network, the current evaluation network, the target decision network and the target evaluation network, and the network parameters are initialized. Then, it is judged whether the training times are reached or convergence (no judgment is required during the first training, and judgment is made once for each subsequent training). Then, the learning environment is initialized (that is, the robot is in the starting state), and actions are performed according to the action strategy to obtain process data and put them into the experience pool. Subsequently, data is sampled from the experience pool to train the neural network, thereby updating the current neural network (that is, the current decision network and the current evaluation network), and updating the target network (that is, the target decision network and the target evaluation network) based on the interval time (that is, the update frequency is reached), and judging whether the robot has reached the end point, if not, returning to the action step according to the action strategy, if yes, judging whether the training times are reached or the model converges, if not, continuing training, if yes, ending training, and obtaining a path planning model.

[0105] The above-mentioned steel structure robot path planning method first obtains the robot's task environment information, which includes the robot's corresponding starting state, target state and obstacle distance, and then inputs the task environment information into the path planning model to predict the robot's next execution action according to the task environment information. The path planning model adjusts the decision network parameters and the evaluation network parameters in the model according to the reward value obtained after the robot executes the action, so that the execution action can effectively avoid obstacles and move towards the target state. Then, the next state of the robot is determined according to the starting state and the execution action, and the next state is used as the current state to continuously predict the subsequent state of the robot. When the subsequent state is the target state, the robot's path is formed based on each state. Through the path planning model including the decision network and the evaluation network of the execution action, the robot path planning is performed based on the purpose of avoiding obstacles and reaching the end point as soon as possible, so that the robot can reach the target state from the starting state without collision and efficiently according to the path, complete the welding of steel structures, and realize intelligent obstacle avoidance in the robot welding process, thereby improving the welding quality and efficiency of steel structures.

[0106] See also Figure 5 , Figure 5 is a block diagram of a steel structure robot path planning device shown in an exemplary embodiment of the present application. The device can be applied to Figure 1 The implementation environment shown is a schematic diagram. It should be understood that the device may also be applicable to other exemplary implementation environments, and this embodiment does not limit the implementation environment to which the device is applicable.

[0107] like Figure 6 As shown, in an exemplary embodiment, the steel structure robot path planning device 500 includes at least an information acquisition module 510, an action prediction module 520, a path prediction module 530 and a path determination module 540, which are described in detail as follows:

[0108] The information acquisition module 510 is used to acquire the task environment information of the robot, where the task environment information includes the robot's corresponding starting state, target state and obstacle distance;

[0109] The action prediction module 520 is used to input the task environment information into the path planning model to predict the execution action of the robot according to the task environment information. The path planning model is obtained by adjusting the model parameters according to the reward value obtained after the robot executes the action. The model parameters include the decision network parameters and the evaluation network parameters corresponding to the robot action;

[0110] A path prediction module 530 is used to determine the next state of the robot according to the starting state and the execution action, and to continuously predict the subsequent state of the robot using the next state as the current state;

[0111] The path determination module 540 is used to determine the path of the robot until the subsequent state is the target state, so that the robot completes the welding of the steel structure according to the path.

[0112] The above-mentioned steel structure robot path planning device first obtains the robot's task environment information, which includes the robot's corresponding starting state, target state and obstacle distance, and then inputs the task environment information into the path planning model to predict the robot's next execution action according to the task environment information. The path planning model adjusts the decision network parameters and evaluation network parameters in the model according to the reward value obtained after the robot executes the action, so that the execution action can effectively avoid obstacles and move towards the target state. Then, the next state of the robot is determined according to the starting state and the execution action, and the next state is used as the current state to continuously predict the subsequent state of the robot. When the subsequent state is the target state, the robot's path is formed based on each state. Through the path planning model including the decision network and the evaluation network of the execution action, the robot path planning is performed based on the purpose of avoiding obstacles and reaching the end point as soon as possible, so that the robot can reach the target state from the starting state without collision and efficiently according to the path, complete the welding of steel structures, and realize intelligent obstacle avoidance during robot welding, thereby improving the welding quality and efficiency of steel structures.

[0113] It should be noted that the steel structure robot path planning device provided in the above embodiment and the steel structure robot path planning method provided in the above embodiment belong to the same concept, and the contents of the operations performed by each module have been described in detail in the method embodiment and will not be repeated here.

[0114] See also Figure 6 , Figure 6 FIG. 1 is a block diagram of a steel structure welding system according to an exemplary embodiment of the present application. Figure 6 As shown, the steel structure welding system 600 includes at least a path planning module 610 and a task execution module 620, which are described in detail as follows:

[0115] A path planning module 610 is used to plan the path of each robot in the task environment simulation platform;

[0116] The task execution module 620 is used to control the robots to travel from their respective starting states to target states without collision according to the paths of the robots, and to control the actuators of the robots to perform welding operations.

[0117] It should be noted that the path planning module 610 is the above-mentioned steel structure robot path planning device, and uses the above-mentioned steel structure robot path planning method.

[0118] In this way, for the welding environment of the entire steel structure, each robot completes welding according to its own path, which can realize automatic task allocation and collaborative operation of multiple sites, multiple robots and multiple tasks, and promote the coordinated development of digitalization, intelligence and greening of the steel structure industry.

[0119] See also Figure 7 , Figure 7 It is a structural schematic diagram of an electronic device shown as an exemplary embodiment of the present application. Figure 7 The structure diagram of the computer system suitable for implementing the electronic device of the embodiment of the present application is shown. It should be noted that: Figure 7 The computer system 700 of the electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0120] like Figure 7 As shown, computer system 700 includes a central processing unit (CPU) 701, which can perform various appropriate actions and processes according to the program stored in read-only memory (ROM) 702 or the program loaded from storage part 708 to random access memory (RAM) 703, such as executing the method in the above embodiment. In RAM 703, various programs and data required for system operation are also stored. CPU 701, ROM 702 and RAM 703 are connected to each other through bus 704. Input / output (I / O) interface 705 is also connected to bus 704.

[0121] The following components are connected to the I / O interface 705: an input section 706 including a keyboard, a mouse, etc.; an output section 707 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 708 including a hard disk, etc.; and a communication section 709 including a network interface card such as a LAN (Local Area Network) card, a modem, etc. The communication section 709 performs communication processing via a network such as the Internet. A drive 710 is also connected to the I / O interface 705 as needed. A removable medium 711, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 710 as needed so that a computer program read therefrom is installed into the storage section 708 as needed.

[0122] In particular, according to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a computer program for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication section 709, and / or installed from a removable medium 711. When the computer program is executed by a central processing unit (CPU) 701, various functions defined in the system of the present application are executed.

[0123] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present invention. Among them, each box in the flowchart or block diagram can represent a module, a program segment, or a part of the code, and the above-mentioned module, program segment, or a part of the code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart, can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0124] The units involved in the embodiments of the present invention may be implemented by software or hardware, and the units described may also be arranged in a processor. The names of these units do not, in some cases, limit the units themselves.

[0125] The above embodiments are merely illustrative of the principles and effects of the present invention, and are not intended to limit the present invention. Anyone familiar with the technology may modify or change the above embodiments without violating the spirit and scope of the present invention. Therefore, all equivalent modifications or changes made by a person of ordinary skill in the art without departing from the spirit and technical ideas disclosed by the present invention shall still be covered by the claims of the present invention.

Claims

1. A steel structure robot path planning method, characterized in that: The method comprises: Acquire task environment information of the robot, wherein the task environment information includes a starting state, a target state, and an obstacle distance corresponding to the robot; The task environment information is input into a path planning model to predict the execution action of the robot according to the task environment information, and the path planning model is obtained by adjusting model parameters according to the reward value obtained after the robot executes the action, and the model parameters include decision network parameters and evaluation network parameters corresponding to the robot action; Determine the next state of the robot according to the starting state and the execution action, and use the next state as the current state to continuously predict the subsequent state of the robot; When the subsequent state is the target state, the path of the robot is determined so that the robot completes the welding of the steel structure according to the path.

2. The steel structure robot path planning method according to claim 1, characterized in that: The method for obtaining the path planning model includes: Constructing a reinforcement learning model, wherein the reinforcement learning model includes a decision network and an evaluation network corresponding to the robot action, wherein the decision network is used to generate a motion strategy, including a current decision network and a target decision network, and the evaluation network is used to evaluate the motion strategy, including a current evaluation network and a target evaluation network; The robot interacts with the task environment by using the current decision network and the current evaluation network to obtain batch quaternary data, wherein the quaternary data is represented as (S, a, r, S′), wherein S represents the current state of the robot, a represents the execution action of the robot in the current state, r represents the reward value of whether a collision occurs in the process of the robot performing the action in the current state to obtain the next state, and S′ represents the next state of the robot after performing the action in the current state; Training the current decision network and the current evaluation network according to the batch quadruple data, and updating the decision network parameters and the evaluation network parameters; The target decision network and the target evaluation network are updated using the updated decision network parameters and the updated evaluation network parameters until a preset training termination condition is reached to obtain the path planning model.

3. The steel structure robot path planning method according to claim 2, characterized in that: The current decision network includes a first fully connected layer, a convolutional neural network layer and a second fully connected layer, and the current evaluation network includes a third fully connected layer, a fourth fully connected layer and a fifth fully connected layer; The method of utilizing the current decision network and the current evaluation network to interact the robot with the task environment includes: Using the first fully connected layer and the convolutional neural network layer to respectively learn the position features of the robot and the distance features of the obstacles, and using the second fully connected layer to learn the spatial features of the robot according to the position features and the distance features, so as to describe the movements of the robot; The third fully connected layer and the fourth fully connected layer are used to process the state and execution action of the robot respectively, and the fifth fully connected layer is used to learn the evaluation features of the processed state and execution action, so as to describe the reward obtained by executing the action.

4. The steel structure robot path planning method according to claim 2, characterized in that: The method for obtaining the quadruple data includes: Set up reinforcement learning components, including state components, action components, and reward function components. The state component is represented by S t =(P t-1 , P t , P e , D t ), where p t-1 Indicates the previous state of the robot, p t Indicates the current state of the robot, p e is the target state of the robot, D t represents the obstacle distance, and the action component is represented by a t =(α t , β t , d t ), where α t represents the horizontal movement angle of the robot in the radar coordinate system, β t represents the vertical movement angle of the robot in the radar coordinate system, α t With β t Together they represent the direction of movement of the robot, d t Indicates the moving distance of the robot in the moving direction, α t , β t d t All are continuous variables, α t With β t The range is [-60°, 60°], d t The range is [0, d max ],d max The preset maximum single moving distance; Using the current decision network, extracting spatial features from the state component, and determining the execution action in the four-tuple data according to the spatial features and the action component; Determine a next state in the four-tuple data according to a current state in the state component and an execution action in the four-tuple data; The current evaluation network is used to process the current state in the state component and the execution action in the four-tuple data, and the reward function component is used to calculate and obtain the reward value in the four-tuple data.

5. The steel structure robot path planning method according to claim 2, characterized in that: The step of training the current decision network and the current evaluation network according to the batch quadruple data and updating the decision network parameters and the evaluation network parameters includes: Inputting the next state in the four-tuple data into the current decision network to obtain the next execution action; Input the current state and the execution action in the four-tuple data into the current evaluation network to calculate the current reward value, and input the next state and the next execution action in the four-tuple data into the current evaluation network to calculate the next reward value; Taking maximizing the current reward value as a goal, calculating a first gradient of the current reward value with respect to the decision network parameter, and updating the decision network parameter according to the first gradient; According to the preset instant reward and discount factor, the next reward value is calculated to obtain the target reward value, and with the goal of minimizing the loss between the current reward value and the target reward value, the second gradient of the loss function with respect to the evaluation network parameters is calculated, and the evaluation network parameters are updated according to the second gradient.

6. The steel structure robot path planning method according to claim 2, characterized in that: The updating of the target decision network and the target evaluation network by using the updated decision network parameters and the updated evaluation network parameters includes: Calculating the decision network parameters, the updated decision network parameters, and a preset learning rate to obtain target decision network parameters, and calculating the evaluation network parameters, the updated evaluation network parameters, and the learning rate to obtain target evaluation network parameters; The target decision network parameters are assigned to the target decision network to complete the update of the target decision network, and the target evaluation parameters are assigned to the target evaluation network to complete the update of the target evaluation network.

7. The steel structure robot path planning method according to any one of claims 1 to 6, characterized in that: The obstacle distance is the obstacle distance matrix D measured in multiple directions t ; The calculation formula of the reward value is: Among them, s t Represents the current state of the robot; a t Represents the actions performed by the robot in the current state; t+1 Represents the next state of the robot after performing an action in the current state; r t (s t ,a t ,s t+1 ) represents the reward value of whether a collision occurs when the robot performs an action in the current state to obtain the next state; R is the reward value of a collision when the robot performs an action in the current state to obtain the next state; L t Represents the distance from the current state of the robot to the target state; L t+1 Indicates the distance from the next state to the target state of the robot; mean(D t+1 ) represents the average value of the obstacle distance matrix of the robot in the next state; mean(D t ) represents the average value of the obstacle distance matrix of the robot in the current state; e is the preset minimum value.

8. The steel structure robot path planning method according to claim 7, characterized in that: The obstacle distance matrix D t Ways to obtain include: Dividing a preset obstacle monitoring angle range into a plurality of range sub-intervals, each of the range sub-intervals representing a monitoring direction; Setting a direction-distance matrix and detecting whether there is an obstacle in each of the monitoring directions; If there is an obstacle in any monitoring direction, the obstacle distance corresponding to the arbitrary monitoring direction is detected and recorded in the direction-distance matrix; if there is no obstacle in any monitoring direction, the obstacle distance corresponding to the arbitrary monitoring direction in the direction-distance matrix is ​​set to a preset distance threshold.

9. A steel structure robot path planning device, characterized in that: The device comprises: An information acquisition module is used to acquire the task environment information of the robot, wherein the task environment information includes the starting state, target state and obstacle distance corresponding to the robot; An action prediction module is used to input the task environment information into a path planning model to predict the execution action of the robot according to the task environment information, wherein the path planning model is obtained by adjusting model parameters according to the reward value obtained after the robot executes the action, and the model parameters include decision network parameters and evaluation network parameters corresponding to the robot action; A path prediction module, used to determine the next state of the robot according to the starting state and the execution action, and to continuously predict the subsequent state of the robot using the next state as the current state; The path determination module is used to determine the path of the robot until the subsequent state is the target state, so that the robot completes the welding of the steel structure according to the path.

10. An electronic device, characterized in that: The electronic device comprises: one or more processors; A storage device for storing one or more programs, when the one or more programs are executed by the one or more processors, enables the electronic device to implement the steel structure robot path planning method as described in any one of claims 1 to 8.

Images