Robot motion planning method, device, equipment and storage medium

Abstract

The present invention provides a robot motion planning method, device, equipment, and storage medium, which are applied to the field of robot assembly technology. The method includes: obtaining multimodal information collected by the robot at the current moment, the position and posture of the object to be assembled at the current moment, and environmental information corresponding to the object to be assembled; the above environmental information includes whether there are obstacles around the object to be assembled; determining the state of the robot at the current moment based on the multimodal information at the current moment; using a deep reinforcement learning network to plan the robot's motion path based on the robot's current state, the position and posture of the object to be assembled, and the environmental information, to determine the robot's motion trajectory as it moves from its current position to the object to be assembled; wherein, the deep reinforcement learning network uses multiple parallel environments to jointly plan the motion path. Adopting the technical solution of the present invention can improve the accuracy and efficiency of path planning.

Description

Technical Field

[0001] The present invention relates to the field of robot assembly technology, and in particular to a robot motion planning method, device, equipment and storage medium. Background Art

[0002] Robotic assembly involves robots performing operations such as gripping, inserting, and tightening industrial parts to assemble them. This typically requires motion planning before a robot can perform an assembly task. After motion planning, the robot is then controlled to execute the task according to the planned motion.

[0003] When planning robot motion to perform assembly tasks, some technologies rely on dynamic models to accurately grasp the robot's kinematic and dynamic parameters, and use forward and inverse kinematics to calculate and control the robot's motion trajectory to complete the assembly task. Other technologies use pre-set information such as the robot's joint motion sequence and visual feedback to implement motion planning and complete the assembly task.

[0004] However, the above technology has the problem that the obtained motion planning is not accurate enough and has low efficiency. Summary of the Invention

[0005] The present invention provides a robot motion planning method, device, equipment and storage medium, which are used to solve the defects in the existing technology that the motion planning obtained by implementing robot motion planning based on dynamic models or pre-set robot joint sequences is not accurate and efficient. The method determines the state of the robot through the multimodal information collected by the robot, and combines the position and posture of the object to be assembled and the environmental information to use a deep reinforcement learning network to plan the robot's motion path to obtain a more accurate planned path. The deep reinforcement learning network uses multiple parallel environments to jointly perform motion path planning, which can significantly improve the efficiency of path planning.

[0006] The present invention provides a robot motion planning method, comprising:

[0007] Obtaining the multimodal information collected by the robot at the current moment, the current position of the object to be assembled, and the environmental information corresponding to the object to be assembled; the environmental information includes whether there are obstacles around the object to be assembled;

[0008] Determining the state of the robot at the current moment based on the multimodal information at the current moment; the state of the robot at the current moment includes the position and posture of the robot at the current moment;

[0009] Based on the robot's current state, the position of the object to be assembled, and environmental information, a deep reinforcement learning network is used to plan the robot's motion path and determine the robot's motion trajectory from its current position to the object to be assembled.

[0010] Among them, the deep reinforcement learning network is trained using a distributed proximal policy optimization algorithm, in which multiple parallel environments are used to jointly plan the motion path.

[0011] According to a robot motion planning method provided by the present invention, a deep reinforcement learning network is used to plan the robot's motion path based on the robot's current state, the position and posture of the object to be assembled, and environmental information, and the motion trajectory of the robot moving from its current position and posture to the object to be assembled is determined, including:

[0012] Based on the robot's current state, the position and posture of the object to be assembled, and environmental information, a deep reinforcement learning network is used to plan the robot's motion path and determine the robot's state at the next moment; the state at the next moment includes the robot's position and posture at the next moment;

[0013] Determine reward functions corresponding to the various assembly factors according to the various assembly factors corresponding to the object to be assembled during the assembly process;

[0014] Based on the reward function, the robot's state at the next moment, the posture of the object to be assembled, and environmental information, a deep reinforcement learning network is used to iteratively plan the robot's motion path and determine the motion trajectory.

[0015] According to a robot motion planning method provided by the present invention, the reward function corresponding to the multiple assembly factors corresponding to the object to be assembled during the assembly process is determined, including:

[0016] Obtain the assembly requirements and various assembly factors corresponding to the object to be assembled;

[0017] Determine the weight corresponding to each assembly factor according to assembly requirements;

[0018] The reward function is determined based on each assembly requirement and the weight corresponding to each assembly requirement.

[0019] According to a robot motion planning method provided by the present invention, the method further includes:

[0020] When the robot moves to the vicinity of the object to be assembled according to the motion trajectory, a visual image of the object to be assembled is acquired by the camera, and a target posture corresponding to the object to be assembled is determined based on the visual image;

[0021] Get the current first pose of the robot's end effector;

[0022] According to the first posture and the target posture, the first posture of the robot is adjusted to determine the second posture; the accuracy of the second posture is higher than the accuracy of the first posture;

[0023] Send a control instruction to the robot; the control instruction includes a second posture, which is used to control the robot to adjust the posture of the end effector according to the second posture.

[0024] According to a robot motion planning method provided by the present invention, the method further includes:

[0025] The assembly task corresponding to the object to be assembled is divided into multiple subtasks, and multiple policy networks are configured in the deep reinforcement learning network; each subtask corresponds to a policy network, and the multiple subtasks include the first subtask of moving the robot from its current position to the object to be assembled;

[0026] When it is determined that the robot has completed the first subtask, the current posture of the robot is fed back to the deep reinforcement learning network so that the deep reinforcement learning network optimizes the policy network corresponding to the second subtask based on the current posture of the robot; the second subtask is the next subtask to be performed after the first subtask is completed among the multiple subtasks;

[0027] The second subtask is performed according to the policy network corresponding to the second subtask optimized by the deep reinforcement learning network.

[0028] According to a robot motion planning method provided by the present invention, the updating method of the strategy network of each subtask includes:

[0029] Determine the degree of difference between the first strategy and the second strategy according to the first strategy output by the strategy network in the current time and the second strategy output in the previous time;

[0030] Determine the target action currently selected according to the first strategy, and evaluate the advantage of the target action according to the advantage estimation function;

[0031] The policy network is updated based on the degree of difference, the advantage of the target action, the number of parallel environments currently used, and the number of time steps to determine the updated policy network.

[0032] According to a robot motion planning method provided by the present invention, the multimodal information includes a red, green, and blue (RGB) image of an object to be assembled, a depth image of the object to be assembled, an infrared image of the object to be assembled, mechanical feature information collected by a force sensor of the robot, and tactile feature information collected by a tactile sensor of the robot. Determining the state of the robot at the current moment based on the multimodal information at the current moment includes:

[0033] A convolutional neural network is used to extract features from the current RGB image and infrared image, respectively, to determine the first feature corresponding to the RGB image and the second feature corresponding to the infrared image; the first feature is used to characterize the surface texture and color information of the object to be assembled, and the second feature is used to characterize the contour information of the object to be assembled;

[0034] A point cloud processing network is used to perform feature extraction processing on the depth image to determine a third feature corresponding to the depth image; the third feature is used to characterize the distance between the robot and the object to be assembled and the three-dimensional shape of the object to be assembled;

[0035] A feature fusion operation is performed on the first feature, the second feature, the third feature, the mechanical feature information, and the tactile feature information to determine the state of the robot at the current moment.

[0036] The present invention also provides a robot motion planning device, comprising the following modules:

[0037] An acquisition module is used to obtain the multimodal information collected by the robot at the current moment, the position and posture of the object to be assembled at the current moment, and the environmental information corresponding to the object to be assembled; the above environmental information includes whether there are obstacles around the object to be assembled;

[0038] A state determination module, configured to determine the state of the robot at a current moment based on the multimodal information at the current moment; the state of the robot at the current moment includes the position and posture of the robot at the current moment;

[0039] The motion planning module is used to plan the robot's motion path based on the robot's current state, the position of the object to be assembled, and environmental information using a deep reinforcement learning network, and determine the robot's motion trajectory from its current position to the object to be assembled;

[0040] Among them, the deep reinforcement learning network is trained using a distributed proximal policy optimization algorithm, in which multiple parallel environments are used to jointly plan the motion path.

[0041] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the robot motion planning method described above is implemented.

[0042] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the above-described robot motion planning methods.

[0043] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements any one of the above-mentioned robot motion planning methods.

[0044] The robot motion planning method, device, equipment and storage medium provided by the present invention obtain the multimodal information collected by the robot at the current moment, the posture corresponding to the object to be assembled at the current moment and the environmental information corresponding to the object to be assembled, determine the state of the robot at the current moment according to the multimodal information at the current moment, and use a deep reinforcement learning network to plan the motion path of the robot according to the state of the robot at the current moment, the posture of the object to be assembled and the environmental information to determine the motion trajectory of the robot moving from the posture at the current moment to the object to be assembled; wherein, the state of the robot at the current moment includes its posture at the current moment, and the deep reinforcement learning network is trained using a distributed proximal strategy optimization algorithm, and the distributed proximal strategy optimization algorithm uses multiple parallel environments to jointly plan the motion path. In this method, the robot's state can be determined through multimodal information collected by the robot. Combined with the position and environmental information of the object to be assembled, a deep reinforcement learning network is used to plan the robot's motion path. This multimodal information, combined with other information, allows the robot to plan a highly accurate motion trajectory during the motion planning process, improving the accuracy of motion planning and subsequent object assembly. This approach also eliminates the need for a precise robot dynamics model and can adapt to changes in the physical properties of different robots and environmental changes. It is more adaptable to assembly parts of various shapes and sizes and different assembly scenarios, thereby reducing system maintenance costs. Furthermore, path planning and assembly tasks can be effectively completed even when the assembly object is offset, obstacles appear, or lighting conditions change. Furthermore, because the deep reinforcement learning network in this method uses multiple parallel environments to jointly plan motion paths, this significantly improves the efficiency of path planning. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the technical solutions in the present invention or the prior art, a brief introduction is given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0046] Figure 1 This is one of the flow charts of the robot motion planning method provided by the present invention.

[0047] Figure 2 This is the second flow chart of the robot motion planning method provided by the present invention.

[0048] Figure 3 It is a structural schematic diagram of the robot motion planning device provided by the present invention.

[0049] Figure 4 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0050] To make the objectives, technical solutions, and advantages of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the embodiments described are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0051] Currently, motion planning for robotic assembly is typically based on precise robot dynamics models or handcrafted feature-based methods. Dynamics-model-based methods require precise knowledge of the robot's kinematic and dynamic parameters, such as joint velocities, accelerations, forces, and torques. The robot's motion trajectory is then calculated and controlled through forward and inverse kinematics to complete the assembly task. In the area of visual servoing, some existing technologies utilize visual sensor feedback to adjust the robot's motion based on the deviation between the target and current position using classical control theory (such as proportional-integral-derivative (PID) control). In the area of reinforcement learning, some work has begun to explore the use of deep reinforcement learning (DRL) for robotic task planning. However, combining it with visual servoing for motion planning in assembly tasks is still at a relatively early stage, and most efforts fail to fully consider the specific characteristics of assembly tasks and the advantages of visual servoing.

[0052] In some traditional robotic assembly systems, robot motion planning and control rely primarily on preprogrammed trajectories and simple environmental perception, resulting in limited adaptability and flexibility. In visual servoing, traditional handcrafted feature-based approaches require manual feature design, and feature extraction can be inaccurate and inflexible for complex assembly parts and scenarios. For example, in some automated assembly lines, assembly is performed using pre-programmed robot joint motion sequences and simple visual feedback (such as the target's two-dimensional position deviation). This makes it difficult to handle variations in part shape and pose, as well as complex environmental interference. Among the explorations of deep reinforcement learning, some studies have only applied it to simple robotic manipulation tasks, such as grasping and placing, without delving into complex assembly tasks. Furthermore, these studies lack sufficient understanding of state representation, action space design, reward function design, and integration with visual servoing. For example, using a simple state representation may not fully reflect the complexity of the assembly task, and the reward function may not adequately consider multiple factors, such as assembly accuracy, time efficiency, and collision risk, resulting in poor learning outcomes.

[0053] The aforementioned methods based on precise dynamic models require precise modeling of the robot. Changes in the robot's physical properties (such as wear and tear, component replacement) or environmental changes require model readjustment, resulting in high system maintenance costs and poor adaptability. Traditional visual servoing methods rely on handcrafted features. The extraction and design of these features are difficult for complex assembly scenarios and are easily affected by factors such as lighting and occlusion, resulting in insufficient motion planning accuracy and robustness. Simple deep reinforcement learning applications in robotic assembly can suffer from inadequate state and action space design, an inability to effectively process high-dimensional visual information, an incomplete reward function, and low learning efficiency, making it difficult to achieve accurate and efficient assembly motion planning. This is particularly true for multi-component assembly, high-precision assembly requirements, and complex environments. In summary, the motion planning methods used in these technologies face numerous challenges when dealing with complex assembly components, diverse environmental conditions, and high-precision assembly requirements. For example, in multi-component assembly scenarios with varying component shapes and sizes, and the presence of environmental interference and obstacles, these motion planning methods based on precise dynamic models or handcrafted features struggle to maintain accuracy and adaptability, and their motion planning efficiency is also low. Based on this, embodiments of the present invention provide a robot motion planning method, apparatus, device, and storage medium, which can solve the above technical problems.

[0054] The following combination Figure 1-Figure 2 The robot motion planning method of the present invention is described.

[0055] It should be noted that the execution subject of the embodiments of the present invention can be a robot motion planning device, an electronic device that includes the robot motion plan, or other devices or equipment. The electronic device here can be a terminal or a server. In the case of a terminal, the electronic device can be a robot or an electronic device in a robot. The following embodiments are described using an electronic device as an example of the execution subject.

[0056] Figure 1 This is one of the flow charts of the robot motion planning method provided by the present invention, such as Figure 1 As shown, the method includes the following steps:

[0057] Step 102 : obtaining the multimodal information collected by the robot at the current moment, the posture of the object to be assembled at the current moment, and the environmental information corresponding to the object to be assembled; the environmental information includes whether there are obstacles around the object to be assembled.

[0058] The multimodal information collected by the robot at the current moment may be of different types, including image information collected by a camera or video camera on the robot, or sensor information collected by mechanical or tactile sensors on the robot. Optionally, the multimodal information may include a red, green, and blue (RGB) image of the object to be assembled, a depth image of the object to be assembled, an infrared image of the object to be assembled, mechanical feature information collected by the robot's force sensors, and tactile feature information collected by the robot's tactile sensors.

[0059] The object to be assembled can be a component to be assembled, such as a screw or bolt. The specific shape and size of the object to be assembled can be determined based on the actual assembly scenario. The object to be assembled can have any shape and size. A camera or video camera, for example, on a robot can capture an image of the object to be assembled at the current moment, and the pose of the object to be assembled at the current moment can be calculated based on the captured image. The pose of the object to be assembled at the current moment can include its position and posture at the current moment, and the pose of the object to be assembled at the current moment can be the pose perceived by the robot.

[0060] The environmental information corresponding to the object to be assembled can be collected by a camera or video camera on the robot, etc., of the object to be assembled and the environmental images around the robot, and the environmental information of the robot and the object to be assembled can be identified through the collected environmental images. The environmental information may include whether there are obstacles around the robot and the object to be assembled, the distance to the obstacles when there are obstacles, the number or shape of the obstacles, and other information.

[0061] Step 104 , determining the state of the robot at the current moment based on the multimodal information at the current moment; the state of the robot at the current moment includes the position and posture of the robot at the current moment.

[0062] In this step, the robot can also obtain its own position at the current moment through the positioning module, and at the same time, it can obtain the robot's posture at the current moment through other sensors (such as speed sensors, angle sensors, etc.). The robot's current position and posture can be used to obtain the robot's current posture.

[0063] After obtaining the multimodal information collected by the robot at the current moment, all of the multimodal information can be combined with the robot's current position and posture to directly determine the robot's current state. Alternatively, a portion of the multimodal information can be selected and combined with the robot's current position and posture to determine the robot's current state. Alternatively, a portion of the modal information can be further processed and then combined with the robot's current position and posture to determine the robot's current state. In short, the robot's current state can be determined using the multimodal information collected by the robot at the current moment.

[0064] Step 106 , based on the current state of the robot, the position of the object to be assembled, and environmental information, a deep reinforcement learning network is used to plan the motion path of the robot to determine the motion trajectory of the robot moving from its current position to the object to be assembled.

[0065] The specific architecture and type of the deep reinforcement learning network are not specifically limited here; for example, it can be a deep Q-network. In this step, the deep reinforcement learning network can be pre-trained. Specifically, the deep reinforcement learning network is trained using the Distributed Proximal Policy Optimization (DPPO) algorithm, which employs multiple parallel environments for motion path planning. Specifically, during each training cycle of the deep reinforcement learning network, multiple environments are trained in parallel to accelerate gradient updates of the deep reinforcement learning network, thereby improving training efficiency. This, in turn, enhances the efficiency of subsequent motion path planning and object assembly performed using the trained deep reinforcement learning network.

[0066] The input of the above-mentioned deep reinforcement learning network can be the state at a certain moment and other information, and the output can be the policy distribution and / or executed actions and / or value estimation, etc.

[0067] Specifically, after the deep reinforcement learning network is trained, the robot's current state, the position of the object to be assembled at the current moment, and the environmental information obtained above can be input into the deep reinforcement learning network for motion path planning. The action that the robot needs to perform at the next moment is output, and then the robot's state at the next moment is determined based on this. The state is then input into the deep reinforcement learning network for motion path planning. The actions performed by the robot at each moment are obtained through repeated iterations, and the robot's motion trajectory is obtained through the position combination corresponding to these actions. The motion trajectory here can be the motion trajectory of the robot moving from its current position to the position of the object to be assembled. In this way, the robot's motion path planning can be achieved.

[0068] As can be seen from the above description, the robot's state data incorporates information from multiple different modalities. This provides the robot with more diverse and richer information for path planning, motion planning, or object assembly, thereby improving the accuracy of subsequent path planning, motion planning, or object assembly. Furthermore, this richer information eliminates the need for a precise robot dynamics model, allowing the motion planning process in this embodiment to adapt to variations in the physical properties of different robots and environmental changes, providing greater adaptability to assembly parts of various shapes and sizes and in diverse assembly scenarios.

[0069] In this embodiment, by obtaining the multimodal information collected by the robot at the current moment, the posture corresponding to the object to be assembled at the current moment, and the environmental information corresponding to the object to be assembled, the state of the robot at the current moment is determined according to the multimodal information at the current moment, and according to the state of the robot at the current moment, the posture of the object to be assembled and the environmental information, a deep reinforcement learning network is used to plan the motion path of the robot to determine the motion trajectory of the robot moving from the posture at the current moment to the object to be assembled; wherein, the state of the robot at the current moment includes its posture at the current moment, and the deep reinforcement learning network is trained using a distributed proximal strategy optimization algorithm, and the distributed proximal strategy optimization algorithm uses multiple parallel environments to jointly plan the motion path. In this method, the robot's state can be determined through multimodal information collected by the robot. Combined with the position and environmental information of the object to be assembled, a deep reinforcement learning network is used to plan the robot's motion path. This multimodal information, combined with other information, allows the robot to plan a highly accurate motion trajectory during the motion planning process, improving the accuracy of motion planning and subsequent object assembly. This approach also eliminates the need for a precise robot dynamics model and can adapt to changes in the physical properties of different robots and environmental changes. It is more adaptable to assembly parts of various shapes and sizes and different assembly scenarios, thereby reducing system maintenance costs. Furthermore, path planning and assembly tasks can be effectively completed even when the assembly object is offset, obstacles appear, or lighting conditions change. Furthermore, because the deep reinforcement learning network in this method uses multiple parallel environments to jointly plan motion paths, this significantly improves the efficiency of path planning.

[0070] The following embodiment describes a possible implementation method for motion path planning.

[0071] In some embodiments, step 106 may include the following steps: planning the robot's motion path using a deep reinforcement learning network based on the robot's current state, the position of the object to be assembled, and environmental information to determine the robot's motion trajectory from its current position to the object to be assembled.

[0072] Step A1: Based on the robot's current state, the position and posture of the object to be assembled, and environmental information, a deep reinforcement learning network is used to plan the robot's motion path and determine the robot's state at the next moment; the state at the next moment includes the robot's position and posture at the next moment.

[0073] In this step, the robot's current state, the position and posture of the object to be assembled, and environmental information can be input into a deep reinforcement learning network for motion path planning, outputting the robot's strategy distribution and / or actions to be executed and / or estimated value of the actions to be executed at the next moment. The robot's position and posture are then adjusted based on the strategy distribution and / or actions to be executed and / or estimated value of the actions to be executed at the next moment, and multimodal information collected by the robot at the next moment is re-collected. The robot's state and position at the next moment are then determined based on the multimodal information collected at the next moment.

[0074] Step A2: determining reward functions corresponding to the various assembly factors according to the various assembly factors corresponding to the object to be assembled during the assembly process.

[0075] In this step, before assembling the object, an assembly reward function can be set. The reward function here is related to multiple assembly factors. As an optional embodiment, the assembly requirements and each assembly factor corresponding to the object to be assembled can be obtained; the weight corresponding to each assembly factor can be determined according to the assembly requirements; and the reward function can be determined according to each assembly requirement and the weight corresponding to each assembly requirement.

[0076] Specifically, assembly requirements can represent requirements for various assembly factors, including assembly accuracy, assembly time, assembly energy consumption, collision risk during assembly, and operational stability during assembly. Assembly requirements can be pre-determined based on the assembly task, such as requiring high assembly accuracy or fast assembly time.

[0077] Each assembly factor can be pre-set with a default weight. After the assembly requirements are determined, the default weights of each assembly factor can be adjusted according to the assembly requirements to obtain the final weight. For example, in high-precision electronic component assembly tasks, the assembly requirements indicate extremely high requirements for assembly accuracy. The weight of assembly accuracy can be increased to make the robot pay more attention to assembly accuracy. In large-scale production scenarios where assembly requirements have high production efficiency requirements, in order to improve overall production efficiency, the weight of assembly time can be appropriately increased. In this way, the final weight of the assembly factor can be determined. Afterwards, each assembly factor can be multiplied by its corresponding weight, and the products can be added together to obtain the reward function. For example, the reward function can be expressed as follows:

[0078] .

[0079] in, R represents the reward function, P Indicates assembly accuracy, which is a key indicator for measuring assembly quality, ensuring that parts can be installed accurately in the designated positions;T Indicates assembly time, which can reflect assembly efficiency. In large-scale production, shortening assembly time can significantly improve production efficiency; E Indicates assembly energy consumption. In the context of increasingly tight energy resources, reducing energy consumption is crucial for the sustainable development of enterprises. C Indicates the collision risk during assembly, preventing the robot from colliding with the surrounding environment or parts, which can reduce the risk of equipment damage and production interruption; S Indicates operational stability during assembly. Stable operation helps improve assembly quality and consistency. 、 、 、 、 They represent the weights of the assembly factors, namely assembly accuracy, assembly time, assembly energy consumption, collision risk during assembly, and operational stability during assembly. They can be flexibly adjusted according to the needs of specific assembly tasks. Usually .

[0080] The design of the above-mentioned reward function can motivate the robot to pursue higher assembly accuracy, shorter assembly time, lower energy consumption, smaller collision risk and higher operational stability while ensuring assembly success, thereby achieving more optimized motion planning and control.

[0081] In step A3, based on the reward function, the state of the robot at the next moment, the position of the object to be assembled, and the environmental information, a deep reinforcement learning network is used to iteratively plan the robot's motion path and determine the motion trajectory.

[0082] In this step, a reward function can be designed for the deep reinforcement learning network according to the above-mentioned reward function. When the current state of the robot, the posture of the object to be assembled, and the environmental information are input into the deep reinforcement learning network for motion path planning, the deep reinforcement learning network can obtain the reward obtained after performing an action based on the current state based on the above-mentioned reward function. Then, the strategy in the deep reinforcement learning network can be adjusted based on the reward to make the current strategy optimal, and based on the optimal strategy, the strategy distribution and / or the action to be performed and / or the value estimate of the action to be performed by the robot at the next moment can be output. Then, in this way, the strategy can be updated and the motion path can be planned by continuously iterating the reward function and the state, and finally the motion trajectory of the robot moving from the current posture to the object to be assembled can be obtained.

[0083] In this embodiment, the robot's current position, the position of the object to be assembled, and environmental information are combined with a deep reinforcement learning network to determine the robot's state at the next moment. Simultaneously, a reward function constructed based on multiple assembly factors is used to plan the robot's motion path and obtain the robot's motion trajectory. This method, using reward functions based on multiple assembly factors, allows the robot to plan its motion path more accurately and efficiently, while meeting the actual assembly requirements. Furthermore, the weights of each assembly factor are determined based on the assembly requirements, and a reward function including these factors is constructed accordingly. This allows the robot to meet the assembly requirements corresponding to each assembly factor as much as possible while ensuring successful assembly, achieving more optimized motion path planning and control.

[0084] The above embodiment describes the process of moving the robot from its current position to the position of the object to be assembled. This process implements coarse-grained path planning. In order to achieve more refined path planning, the following embodiment describes the process of performing refined motion path planning based on deep reinforcement learning combined with visual servoing.

[0085] Figure 2 This is the second flow chart of the robot motion planning method provided by the present invention, such as Figure 2 As shown, the above method may further include the following steps:

[0086] Step 202 : When the robot moves to the vicinity of the object to be assembled according to the motion trajectory, a visual image of the object to be assembled captured by a camera is acquired, and a target posture corresponding to the object to be assembled is determined based on the visual image.

[0087] In this step, when the robot moves from its current position to the vicinity of the object to be assembled, the camera mounted on the robot can capture images of the object to be assembled, obtaining a captured visual image, which can be an RGB image, a depth image, etc. The electronic device can then obtain the visual image captured by the camera and process it, for example, by identifying the position and posture of the object to be assembled in the image to obtain the pose in the image, and then convert the pose in the image into physical space to obtain the target pose of the object to be assembled in physical space.

[0088] Step 204: Obtain the first pose currently corresponding to the end effector of the robot.

[0089] In this step, the position data, velocity data, and angle data of the robot can be collected by devices such as position sensors, velocity sensors, and angle sensors at the robot's end effector. The current position, velocity, and angle data of the robot's end effector can be obtained from the collected position data, velocity data, and angle data, and recorded as the first pose. For example, the currently collected position, velocity, and angle of the robot can be used as its first pose.

[0090] Step 206: Adjust the first posture of the robot according to the first posture and the target posture to determine a second posture; the accuracy of the second posture is higher than the accuracy of the first posture.

[0091] In this step, after obtaining the first pose of the robot end effector and the target pose of the object to be assembled, the deviation between the first pose and the target pose can be calculated, and then the first pose can be precisely adjusted based on the deviation and the preset gain to obtain a second pose with higher accuracy.

[0092] For example, the formula Adjust the posture (including position and attitude) of the robot's end effector, where u is the adjusted second posture and K is the preset gain, which is the gain factor in the robot's visual servo system. It mainly determines the range of motion when the robot makes fine adjustments and can be predetermined by the robot's visual servo system. is the deviation between the first pose and the target pose.

[0093] Step 208: Send a control instruction to the robot; the control instruction includes a second posture, which is used to control the robot to adjust the posture of the end effector according to the second posture.

[0094] In this step, after adjusting a more precise second posture for the robot, the second posture can be encapsulated in a control instruction and sent to the robot. After receiving the control instruction, the robot can control its end effector to adjust the current first posture to a more precise second posture, so as to better assemble the assembly object in the future, such as grasping, placing down, tightening, etc.

[0095] After the robot accurately adjusts its posture according to the control instructions of the visual servo system, the adjusted posture can be fed back to the deep reinforcement learning network, so that the deep reinforcement learning network can analyze the execution effect of the current output strategy based on the adjusted posture, and continuously optimize the strategy network according to the execution effect, thereby realizing a closed loop of the entire assembly process.

[0096] For example, in the case of automobile parts assembly, the deep reinforcement learning network first plans a rough grasping and placement path based on environmental information, guiding the robot to approach the target part. During the approach process, the robot's visual servo system obtains the target position and posture information of the part in real time through the camera, and calculates the deviation between its end effector and the target position. Then, according to the control law Visual servoing sends precise adjustment commands to the robot, fine-tuning the position and posture of the robot's end effector to ensure accurate grasping and placement of parts. Deep reinforcement learning then analyzes the effectiveness of the strategy based on the feedback from visual servoing, summarizes lessons learned, and optimizes subsequent strategies. Through this closed-loop optimization mechanism, the robot can continuously improve assembly accuracy and efficiency.

[0097] In this embodiment, when the robot moves roughly around the object to be assembled, the camera on the robot captures the visual image of the object to obtain its target posture and the first posture of the robot's end effector, and combines these two postures to adjust the posture of the robot's end effector so that it can approach the object to be assembled more accurately. In this way, through the close collaboration of the visual servo system and the deep reinforcement learning network, precise assembly of the object to be assembled can be achieved.

[0098] The following embodiment illustrates a process of setting multiple strategy networks based on an assembly task to execute the assembly task.

[0099] In some embodiments, the above method may further include the following steps:

[0100] The assembly task corresponding to the object to be assembled is divided into multiple subtasks, and multiple policy networks are configured in the deep reinforcement learning network; each subtask corresponds to a policy network, and the multiple subtasks include the first subtask of moving the robot from its current position to the object to be assembled;

[0101] When it is determined that the robot has completed the first subtask, the current posture of the robot is fed back to the deep reinforcement learning network so that the deep reinforcement learning network optimizes the policy network corresponding to the second subtask based on the current posture of the robot; the second subtask is the next subtask to be performed after the first subtask is completed among the multiple subtasks;

[0102] The second subtask is performed according to the policy network corresponding to the second subtask optimized by the deep reinforcement learning network.

[0103] After determining the assembly task for the object to be assembled, the task can be divided into multiple subtask levels, such as a coarse positioning subtask, a fine positioning subtask, an insertion subtask, and a tightening subtask. The coarse positioning subtask is the task of moving the robot from its current position to the vicinity of the object to be assembled according to the planned motion trajectory, and can be recorded as the first subtask.

[0104] In this embodiment, a corresponding strategy network and value network are pre-configured for each subtask in the deep learning network according to multiple subtasks, so as to execute the corresponding motion planning subtask.

[0105] After the robot completes the first subtask, it can proceed to the second subtask, which is a subtask executed consecutively after the first subtask. For example, the first subtask is a coarse positioning task, the second subtask is a precise positioning task, and the third subtask can be a grasping and inserting subtask. After completing the first subtask, the electronic device can obtain the robot's current position and posture and feed it back to the deep reinforcement learning network. The deep reinforcement learning network adjusts and optimizes the policy network for the second subtask based on the input current position and posture, thereby obtaining the optimal policy network for the second subtask. After determining the optimal policy network for the second subtask, the second subtask can be executed based on the optimal policy network for the second subtask.

[0106] It is understandable that the policy network of each subtask can be continuously optimized and updated. For each subtask policy network, its update method may include the following steps:

[0107] Based on the first strategy output by the policy network in the current time and the second strategy output in the previous time, the degree of difference between the first strategy and the second strategy is determined; the target action currently selected is determined based on the first strategy, and the advantage of the target action is evaluated according to the advantage estimation function; based on the degree of difference, the advantage of the target action, the number of parallel environments currently used, and the number of time steps, the policy network is updated to determine the updated policy network.

[0108] The policy network for each subtask can be updated according to the following formula:

[0109] .

[0110] in, represents the policy loss function, represents the policy loss function Parameters gradient; N Indicates the number of parallel environments used to calculate the loss function in each training cycle, by NRun in parallel across multiple environments to accelerate gradient updates and improve training efficiency; T is the number of time steps, which represents the number of times the agent interacts with the environment in each training cycle; n Indicates the number of parallel environments, ranging from 1 to N , t Indicates time / moment, ranging from 1 to T ; It represents the probability ratio, which is used to measure the difference between the first strategy output by the policy network in the current time and the second strategy output in the previous time. It can be calculated using the following formula:

[0111] .

[0112] in, s t Indicates that the robot is t The state of the moment, Indicates that the status s t Next, select the action; Indicates the first strategy of the current output; Represents the second strategy of the previous output; each strategy output includes the current state and the action selected under the current state.

[0113] represents the advantage estimate, i.e., the estimated advantage, used to evaluate the advantage of the current action; clip () function means limiting the probability ratio to a certain range to prevent its value from exceeding the preset range, thereby avoiding excessive strategy updates leading to training failure; Represents the clipping parameter, which is used to limit the amplitude of policy updates and ensure the stability of training.

[0114] Specifically, the above can obtain the estimated advantage by estimating the advantage of the current target action through the policy currently output by the policy network, which includes the current state of the robot and the target action selected in the current state; at the same time, the probability ratio between the current output policy and the previous output policy is calculated, and the policy network is updated in combination with the current number of time steps and the number of parallel environments, clipping parameters, etc., to obtain an updated policy network, and the performance of the updated policy network is better.

[0115] During the actual assembly of complex mechanical products, such as an engine, the policy network at the coarse positioning level quickly guides the robot to the approximate location of the target part based on environmental information, laying the foundation for subsequent precise operations. Subsequently, the policy network at the fine positioning level leverages more detailed environmental perception information to further adjust the robot's position, bringing it closer to the part. At critical operation levels like insertion and tightening, the policy network at the corresponding level generates precise motion control instructions based on the part's shape, size, and assembly requirements.

[0116] Furthermore, the policy and value networks corresponding to these multiple levels or subtasks can be pre-trained and trained using the Distributed Proximal Policy Optimization (DPPO) algorithm. The DPPO algorithm, leveraging the advantages of distributed training, can fully leverage multiple parallel environments for simultaneous training, significantly accelerating learning and enabling the robot to more quickly learn effective strategies for different assembly stages (coarse positioning, fine positioning, insertion, tightening, etc.), thereby better navigating complex, high-dimensional state and action spaces.

[0117] In this embodiment, by dividing the assembly task into multiple subtasks and configuring a corresponding strategy network and value network for each subtask, it is convenient to provide a better strategy for each subtask to perform the corresponding task, thereby improving the accuracy of subtask execution and the efficiency of the entire assembly task execution.

[0118] The following embodiment illustrates a process of obtaining the robot's current state based on multimodal information collected by the robot at the current moment.

[0119] In some embodiments, determining the robot's current state based on the multimodal information at the current moment in step 104 may include the following steps:

[0120] A convolutional neural network is used to extract features from the current RGB image and infrared image, respectively, to determine the first feature corresponding to the RGB image and the second feature corresponding to the infrared image; the first feature is used to characterize the surface texture and color information of the object to be assembled, and the second feature is used to characterize the contour information of the object to be assembled;

[0121] A point cloud processing network is used to perform feature extraction processing on the depth image to determine a third feature corresponding to the depth image; the third feature is used to characterize the distance between the robot and the object to be assembled and the three-dimensional shape of the object to be assembled;

[0122] A feature fusion operation is performed on the first feature, the second feature, the third feature, the mechanical feature information, and the tactile feature information to determine the state of the robot at the current moment.

[0123] Among them, the above-mentioned convolutional neural network can be, for example, a CNN network, and the point cloud processing network can be, for example, a PointNet network, etc.

[0124] Specifically, after acquiring the RGB image at the current moment, the key features such as the surface texture and color information of the object to be assembled in the RGB image can be extracted through a convolutional neural network to obtain the extracted features, which are recorded as the first features. At the same time, after acquiring the infrared image, the same convolutional neural network as above or a different convolutional neural network can be used to extract the contour feature information of the object to be assembled in the infrared image to obtain the extracted features, which are recorded as the second features.

[0125] The robot can also use devices such as lidar or depth cameras to collect the depth image of the object to be assembled at the current moment, and then use the point cloud processing network to extract the spatial structural features of the depth image to obtain the distance between the robot and the object to be assembled, the three-dimensional shape of the object to be assembled, the position of the object to be assembled and other feature information, and then use this feature information as the third feature.

[0126] At the same time, sensor devices such as force sensors and tactile sensors can also be installed on the robot's end effector (such as a manipulator). When the robot grasps the object to be assembled, the mechanical sensor can be used to collect the force information at the current moment when the robot grasps the object to be assembled, and the tactile sensor can be used to collect the tactile information at the current moment when the robot grasps the object to be assembled. The force information can be recorded as mechanical characteristic information, and the tactile information can be recorded as tactile characteristic information.

[0127] After obtaining the first feature, second feature, third feature, mechanical feature information, and tactile feature information at the current moment, a feature fusion operation can be performed on these features. For example, the following formula can be used to perform the feature fusion operation to obtain a fused high-dimensional feature, and the high-dimensional feature is used as the current state of the robot:

[0128] .

[0129] in, Indicates the state of the robot at the current moment; Represents feature fusion operation; Represents an RGB image; represents a depth image; Indicates infrared image; Represents mechanical feature information and tactile feature information.

[0130] Taking the complex assembly workshop environment as an example, when a robot needs to grasp a part with complex surface texture and changing surrounding light, RGB images can provide surface texture and color information of the part, depth images can accurately measure the distance between the part and the robot and the three-dimensional shape of the part, infrared images can assist in identifying the contours of the part in poor light, and force sensors and tactile sensors can provide feedback on the force conditions when the robot grasps the part.

[0131] In this embodiment, the multimodal information collected by the robot at the current moment is subjected to feature extraction and feature fusion to obtain the state of the robot at the current moment. In this way, through the fusion of multimodal feature information, the robot can perceive the assembly scene more comprehensively and accurately, greatly improving the accuracy and robustness of the state representation, and providing a solid and reliable data basis for the robot's motion planning.

[0132] It can be seen from the description of the above embodiments that in the embodiments of the present invention, the robot collects information about the environment and assembly parts through multimodal sensors, including RGB images, depth images, infrared images and other sensor information, and uses this information to perform feature extraction and feature fusion to obtain the state of the robot, which is a high-dimensional state representation. Afterwards, a deep reinforcement learning network / architecture is used to generate a preliminary motion plan based on the state information, and at the same time, the assembly task is decomposed into multiple levels, and each level uses a different policy network and value network for learning. In this process, a reward function related to each assembly factor of the assembly requirements is added to provide feedback based on the performance of the assembly, guiding the strategy optimization of deep reinforcement learning. At the same time, the motion trajectory planned by deep reinforcement learning is fine-tuned in real time through visual servoing, and the feedback of visual servoing reacts to the strategy update of deep reinforcement learning, forming a closed-loop optimization, and ultimately achieving precise assembly operations.

[0133] The embodiments of the present invention have the following technical effects:

[0134] 1. Improve assembly efficiency and precision.

[0135] Through multimodal information fusion and improved deep reinforcement learning architecture, the robot can learn effective assembly strategies more quickly and maintain high assembly accuracy in complex environments, reducing assembly time and errors to meet the requirements of high-precision assembly tasks.

[0136] 2. Enhance adaptability and robustness.

[0137] Without requiring a precise robot dynamics model, the system can adapt to variations in the physical characteristics of different robots and environmental changes, providing greater adaptability to assembly parts of various shapes and sizes and in diverse assembly scenarios, reducing system maintenance costs. Assembly tasks can be completed effectively even when parts are misaligned, obstacles are present, or lighting conditions change.

[0138] 3. Optimize resource utilization.

[0139] A reward function that comprehensively considers various assembly factors can enable robots to optimize resource utilization, reduce energy consumption and costs while completing assembly tasks. For large-scale automated assembly production lines, it can significantly save energy and operating costs.

[0140] The robot motion planning device provided by the present invention is described below. The robot motion planning device described below and the robot motion planning method described above can be referenced to each other.

[0141] Figure 3 This is a schematic diagram of the structure of the robot motion planning device provided by the present invention, see Figure 3 As shown, the device may include:

[0142] An acquisition module 310 is configured to acquire the multimodal information collected by the robot at the current moment, the position and posture of the object to be assembled at the current moment, and the environmental information corresponding to the object to be assembled; the environmental information includes whether there are obstacles around the object to be assembled;

[0143] A state determination module 320 is configured to determine the state of the robot at a current moment based on the multimodal information at the current moment; the state of the robot at the current moment includes the position and posture of the robot at the current moment;

[0144] The motion planning module 330 is used to plan the robot's motion path based on the robot's current state, the position of the object to be assembled, and environmental information using a deep reinforcement learning network to determine the robot's motion trajectory from its current position to the object to be assembled;

[0145] Among them, the deep reinforcement learning network is trained using a distributed proximal policy optimization algorithm, in which multiple parallel environments are used to jointly plan the motion path.

[0146] In some embodiments, the motion planning module 330 is specifically configured to plan a motion path for the robot using a deep reinforcement learning network based on the current state of the robot, the position and posture of the object to be assembled, and environmental information, and determine the state of the robot at the next moment; the state at the next moment includes the position and posture of the robot at the next moment;

[0147] Determine reward functions corresponding to the various assembly factors according to the various assembly factors corresponding to the object to be assembled during the assembly process;

[0148] Based on the reward function, the robot's state at the next moment, the posture of the object to be assembled, and environmental information, a deep reinforcement learning network is used to iteratively plan the robot's motion path and determine the motion trajectory.

[0149] Optionally, the motion planning module 330 is specifically configured to obtain assembly requirements and assembly factors corresponding to the object to be assembled; determine weights corresponding to the assembly factors according to the assembly requirements; and determine a reward function according to the assembly requirements and the weights corresponding to the assembly requirements.

[0150] In some embodiments, the apparatus further comprises:

[0151] The target posture determination module is used to obtain a visual image of the object to be assembled after the camera captures the image when the robot moves to the vicinity of the object to be assembled according to the motion trajectory, and determine the target posture corresponding to the object to be assembled based on the visual image;

[0152] The robot posture acquisition module is used to obtain the first posture corresponding to the robot's end effector;

[0153] The posture adjustment module is used to adjust the first posture of the robot according to the first posture and the target posture to determine the second posture; the accuracy of the second posture is higher than the accuracy of the first posture;

[0154] The control module is used to send control instructions to the robot; the above control instructions include a second posture, which is used to control the robot to adjust the posture of the end effector according to the second posture.

[0155] In some embodiments, the apparatus further comprises:

[0156] A partitioning module is configured to divide the assembly task corresponding to the object to be assembled into multiple subtasks and configure multiple policy networks in the deep reinforcement learning network; each subtask corresponds to a policy network, and the multiple subtasks include a first subtask of moving the robot from its current position to the object to be assembled;

[0157] A feedback module is configured to, upon determining that the robot has completed the first subtask, feed back the robot's current position to the deep reinforcement learning network, so that the deep reinforcement learning network optimizes the policy network corresponding to the second subtask based on the robot's current position; the second subtask being the next subtask to be performed after the first subtask is completed among the multiple subtasks;

[0158] An execution module is configured to execute the second subtask according to the policy network corresponding to the second subtask optimized by the deep reinforcement learning network.

[0159] In some embodiments, the apparatus further comprises an updating module, the updating module being configured to update the policy network of each subtask;

[0160] The update module is specifically used to determine the degree of difference between the first strategy and the second strategy output by the policy network in the current time and the second strategy output in the previous time; determine the target action currently selected based on the first strategy, and evaluate the advantage of the target action based on the advantage estimation function; update the policy network based on the degree of difference, the advantage of the target action, the number of parallel environments currently used, and the number of time steps, and determine the updated policy network.

[0161] In some embodiments, the multimodal information includes an RGB image of the object to be assembled, a depth image of the object to be assembled, an infrared image of the object to be assembled, mechanical feature information collected by the force sensor of the robot, and tactile feature information collected by the tactile sensor of the robot. The state determination module 320 is specifically used to use a convolutional neural network to perform feature extraction on the RGB image and the infrared image at the current moment, respectively, to determine the first feature corresponding to the RGB image and the second feature corresponding to the infrared image; the first feature is used to characterize the surface texture and color information of the object to be assembled, and the second feature is used to characterize the contour information of the object to be assembled; the point cloud processing network is used to perform feature extraction processing on the depth image to determine the third feature corresponding to the depth image; the third feature is used to characterize the distance between the robot and the object to be assembled and the three-dimensional shape of the object to be assembled; the first feature, the second feature, the third feature, the mechanical feature information and the tactile feature information are subjected to feature fusion operation to determine the state of the robot at the current moment.

[0162] It should be noted here that the above-mentioned device provided by the embodiment of the present invention can implement all the method steps implemented by the above-mentioned method embodiment and can achieve the same technical effect. The parts and beneficial effects of this embodiment that are the same as those of the method embodiment will not be described in detail here.

[0163] Figure 4 An example of a physical structure diagram of an electronic device is shown below. Figure 4As shown, the electronic device may include: a processor (processor) 410 , a communication interface (Communications Interface) 420 , a memory (memory) 430 and a communication bus 440 , wherein the processor 410 , the communication interface 420 , and the memory 430 communicate with each other via the communication bus 440 . The processor 410 can call the logic instructions in the memory 430 to execute the robot motion planning method, which includes: obtaining the multimodal information collected by the robot at the current moment, the posture corresponding to the object to be assembled at the current moment, and the environmental information corresponding to the object to be assembled; the above-mentioned environmental information includes whether there are obstacles around the object to be assembled; determining the state of the robot at the current moment based on the multimodal information at the current moment; the above-mentioned state of the robot at the current moment includes the posture of the robot at the current moment; based on the state of the robot at the current moment, the posture of the object to be assembled and the environmental information, using a deep reinforcement learning network to plan the motion path of the robot, and determine the motion trajectory of the robot moving from its current posture to the object to be assembled; wherein the deep reinforcement learning network is trained using a distributed proximal strategy optimization algorithm, and the distributed proximal strategy optimization algorithm uses multiple parallel environments to jointly plan the motion path.

[0164] Furthermore, the logic instructions in the aforementioned memory 430 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product, stored in a storage medium, includes instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, a mobile hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0165] On the other hand, the present invention also provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the robot motion planning method provided by the above methods, which includes: obtaining the multimodal information collected by the robot at the current moment, the posture corresponding to the object to be assembled at the current moment, and the environmental information corresponding to the object to be assembled; the above environmental information includes whether there are obstacles around the object to be assembled; determining the state of the robot at the current moment based on the multimodal information at the current moment; the state of the robot at the current moment includes the posture of the robot at the current moment; based on the state of the robot at the current moment, the posture of the object to be assembled and the environmental information, a deep reinforcement learning network is used to plan the motion path of the robot, and determine the motion trajectory of the robot moving from its current posture to the object to be assembled; wherein the deep reinforcement learning network is trained using a distributed proximal strategy optimization algorithm, and the distributed proximal strategy optimization algorithm uses multiple parallel environments to jointly plan the motion path.

[0166] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the robot motion planning method provided by the above-mentioned methods, the method comprising: obtaining multimodal information collected by the robot at the current moment, the posture corresponding to the object to be assembled at the current moment, and the environmental information corresponding to the object to be assembled; the above-mentioned environmental information includes whether there are obstacles around the object to be assembled; determining the state of the robot at the current moment based on the multimodal information at the current moment; the above-mentioned state of the robot at the current moment includes the posture of the robot at the current moment; based on the state of the robot at the current moment, the posture of the object to be assembled and the environmental information, using a deep reinforcement learning network to plan the motion path of the robot, and determining the motion trajectory of the robot moving from its current posture to the object to be assembled; wherein the deep reinforcement learning network is trained using a distributed proximal strategy optimization algorithm, and the distributed proximal strategy optimization algorithm uses multiple parallel environments to jointly plan the motion path.

[0167] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one location or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0168] Through the description of the above embodiments, those skilled in the art will clearly understand that each embodiment can be implemented using software plus a necessary general-purpose hardware platform, or of course, hardware. Based on this understanding, the essence of the above technical solution, or the portion that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, or an optical disk, and includes a number of instructions for causing a computer device (such as a personal computer, server, or network device) to execute the methods described in each embodiment or certain portions of the embodiments.

[0169] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A robot motion planning method, characterized in that: include: Acquire the multimodal information collected by the robot at the current moment, the posture of the object to be assembled at the current moment, and the environmental information corresponding to the object to be assembled; The environmental information includes whether there are obstacles around the object to be assembled; Determining a state of the robot at a current moment according to the multimodal information at a current moment; the state of the robot at a current moment includes the position and posture of the robot at a current moment; Based on the current state of the robot, the position of the object to be assembled, and the environmental information, a deep reinforcement learning network is used to plan the motion path of the robot to determine the motion trajectory of the robot moving from its current position to the object to be assembled; The deep reinforcement learning network is trained using a distributed proximal strategy optimization algorithm, wherein the distributed proximal strategy optimization algorithm uses multiple parallel environments to jointly plan the motion path; The method further comprises: When the robot moves to the vicinity of the object to be assembled according to the motion trajectory, a visual image of the object to be assembled is acquired after being captured by a camera, and a posture of the object to be assembled in the visual image is determined based on the visual image, and the posture in the visual image is converted into a physical space to determine a target posture corresponding to the object to be assembled in the physical space; Obtaining the first pose currently corresponding to the end effector of the robot; Calculating a deviation between the first posture and the target posture based on the first posture and the target posture, and adjusting the first posture of the robot based on the deviation and a preset gain to determine a second posture; the accuracy of the second posture is higher than the accuracy of the first posture, and the preset gain is a gain factor in the robot visual servo system; A control instruction is sent to the robot; the control instruction includes the second posture, which is used to control the robot to adjust the posture of the end effector according to the second posture; after the robot accurately adjusts its posture according to the control instruction of the visual servo system, the adjusted posture is fed back to the deep reinforcement learning network, so that the deep reinforcement learning network analyzes the execution effect of the currently output strategy based on the adjusted posture, and continuously optimizes the strategy network therein according to the execution effect, thereby realizing a closed loop of the entire assembly process.

2. The robot motion planning method according to claim 1, characterized in that: The method includes: planning a motion path for the robot using a deep reinforcement learning network based on the current state of the robot, the position of the object to be assembled, and the environmental information, and determining a motion trajectory of the robot moving from its current position to the object to be assembled. Based on the current state of the robot, the position and posture of the object to be assembled, and the environmental information, a deep reinforcement learning network is used to plan the motion path of the robot to determine the state of the robot at the next moment; the state at the next moment includes the position and posture of the robot at the next moment; Determining reward functions corresponding to the multiple assembly factors according to the multiple assembly factors corresponding to the object to be assembled during the assembly process; According to the reward function, the state of the robot at the next moment, the position and posture of the object to be assembled, and the environmental information, a deep reinforcement learning network is used to iteratively plan the motion path of the robot to determine the motion trajectory.

3. The robot motion planning method according to claim 2, characterized in that: The step of determining the reward functions corresponding to the multiple assembly factors corresponding to the object to be assembled during the assembly process includes: Obtaining assembly requirements and assembly factors corresponding to the object to be assembled; Determining the weight corresponding to each of the assembly factors according to the assembly requirements; The reward function is determined according to each of the assembly requirements and the weight corresponding to each of the assembly requirements.

4. The robot motion planning method according to claim 1, characterized in that: The method further comprises: Dividing the assembly task corresponding to the object to be assembled into a plurality of subtasks, and configuring a plurality of policy networks in the deep reinforcement learning network; each of the subtasks corresponds to a policy network, and the plurality of subtasks include a first subtask of moving the robot from its current position to the object to be assembled; When it is determined that the robot has completed the first subtask, feeding back the current posture of the robot to the deep reinforcement learning network so that the deep reinforcement learning network optimizes the policy network corresponding to the second subtask based on the current posture of the robot; the second subtask is the next subtask to be performed after the completion of the first subtask among the multiple subtasks; The second subtask is performed according to the policy network corresponding to the second subtask optimized by the deep reinforcement learning network.

5. The robot motion planning method according to claim 4, characterized in that: The updating method of the policy network for each subtask includes: Determining a degree of difference between the first strategy and the second strategy according to a first strategy currently output by the strategy network and a second strategy previously output by the strategy network; determining a target action currently selected according to the first strategy, and evaluating the advantage of the target action according to an advantage estimation function; The policy network is updated according to the degree of difference, the advantage of the target action, the number of parallel environments currently used, and the number of time steps to determine an updated policy network.

6. The robot motion planning method according to claim 1, characterized in that: The multimodal information includes a red, green, and blue (RGB) image of the object to be assembled, a depth image of the object to be assembled, an infrared image of the object to be assembled, mechanical feature information collected by a force sensor of the robot, and tactile feature information collected by a tactile sensor of the robot. Determining the state of the robot at a current moment based on the multimodal information at a current moment includes: Using a convolutional neural network to perform feature extraction on the RGB image and the infrared image at the current moment, respectively, to determine a first feature corresponding to the RGB image and a second feature corresponding to the infrared image; the first feature is used to characterize the surface texture and color information of the object to be assembled, and the second feature is used to characterize the contour information of the object to be assembled; Performing feature extraction processing on the depth image using a point cloud processing network to determine a third feature corresponding to the depth image; the third feature is used to characterize the distance between the robot and the object to be assembled and the three-dimensional shape of the object to be assembled; A feature fusion operation is performed on the first feature, the second feature, the third feature, the mechanical feature information, and the tactile feature information to determine the state of the robot at a current moment.

7. A robot motion planning device, characterized in that: include: An acquisition module is used to acquire the multimodal information collected by the robot at the current moment, the posture corresponding to the object to be assembled at the current moment, and the environmental information corresponding to the object to be assembled; The environmental information includes whether there are obstacles around the object to be assembled; a state determination module, configured to determine the state of the robot at a current moment based on the multimodal information at the current moment; the state of the robot at the current moment includes the position and posture of the robot at the current moment; A motion planning module is configured to plan a motion path for the robot using a deep reinforcement learning network based on the current state of the robot, the position of the object to be assembled, and the environmental information, and determine a motion trajectory for the robot to move from its current position to the object to be assembled; The deep reinforcement learning network is trained using a distributed proximal strategy optimization algorithm, wherein the distributed proximal strategy optimization algorithm uses multiple parallel environments to jointly plan the motion path; The device further comprises: a target posture determination module, configured to, when the robot moves to the vicinity of the object to be assembled according to the motion trajectory, obtain a visual image of the object to be assembled captured by a camera, determine the posture of the object to be assembled in the visual image based on the visual image, convert the posture in the visual image into a physical space, and determine the target posture corresponding to the object to be assembled in the physical space; A robot posture acquisition module is used to obtain the first posture currently corresponding to the end effector of the robot; a posture adjustment module, configured to calculate a deviation between the first posture and the target posture based on the first posture and the target posture, and adjust the first posture of the robot based on the deviation and a preset gain to determine a second posture; the accuracy of the second posture is higher than the accuracy of the first posture, and the preset gain is a gain factor in the robot visual servo system; A control module is configured to send a control instruction to the robot; the control instruction includes the second posture, and is configured to control the robot to adjust the posture of the end effector according to the second posture; after the robot accurately adjusts its posture according to the control instruction of the visual servo system, the adjusted posture is fed back to the deep reinforcement learning network, so that the deep reinforcement learning network analyzes the execution effect of the currently output strategy based on the adjusted posture, and continuously optimizes the strategy network therein according to the execution effect, thereby realizing a closed loop of the entire assembly process.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the computer program, the robot motion planning method according to any one of claims 1 to 6 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the robot motion planning method according to any one of claims 1 to 6 is implemented.

Images