Obstacle avoidance method and device of mechanical arm, electronic equipment and storage medium

Abstract

The application provides an obstacle avoidance method and device of a mechanical arm, electronic equipment and a storage medium, wherein the method comprises: obtaining an initial running track of the mechanical arm and a to-be-detected image; determining an obstacle avoidance in the initial running track based on an attention mechanism and positions of each arm end of the mechanical arm in the initial running track and positions of each obstacle in the to-be-detected image; and adjusting the initial running track based on the obstacle avoidance to obtain a to-be-executed running track. The method and device provided by the application can perform online re-planning of the running track of the mechanical arm in a multi-obstacle scene through the attention mechanism, realize fast and accurate obstacle avoidance planning in the presence of a plurality of sudden obstacles, and improve the universality of obstacle avoidance planning of the mechanical arm.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of mechanical arm control, and in particular to an obstacle avoidance method and device for a mechanical arm, an electronic device and a storage medium. BACKGROUND

[0002] With the increasingly wide application of mechanical arms in various fields, the obstacle avoidance trajectory planning problem of the mechanical arms has attracted more attention. The obstacle avoidance method of the mechanical arm is usually to generate a feasible trajectory to guide the manipulator to approach the target. Overall, the mainstream methods include search-based and optimization-based methods, which find an obstacle avoidance path in the known robot workspace to achieve the obstacle avoidance of the mechanical arm.

[0003] However, the search-based method may lead to inconsistency for the same obstacle path and may lead to high computational cost, and the optimization-based algorithm needs to manually design the cost function and constraints. Especially in the multi-obstacle scenario, the search-based or optimization-based method may require a large amount of computation, time, and labor cost, and thus the efficiency of the mechanical arm obstacle avoidance planning is not high, that is, the search-based or optimization-based algorithm does not have an ideal effect in the mechanical arm obstacle avoidance in the multi-obstacle scenario. SUMMARY

[0004] The present application provides an obstacle avoidance method and device for a mechanical arm, an electronic device and a storage medium to solve the defects of high computational cost and low performance of the mechanical arm obstacle avoidance in the prior art.

[0005] The present application provides an obstacle avoidance method for a mechanical arm, comprising:

[0006] obtaining an initial running trajectory of the mechanical arm and a to-be-detected image;

[0007] determining an obstacle avoidance in the initial running trajectory based on an attention mechanism, positions of each arm end of the mechanical arm in the initial running trajectory and obstacle positions of each obstacle in the to-be-detected image.

[0008] adjusting the initial running trajectory based on the obstacle avoidance to obtain a to-be-executed running trajectory.

[0009] According to the obstacle avoidance method for a mechanical arm provided by the present application, the determination of the obstacle avoidance in the initial running trajectory based on the attention mechanism, the positions of each arm end of the mechanical arm in the initial running trajectory and the obstacle positions of each obstacle in the to-be-detected image comprises:

[0010] determining the relative positions of each obstacle and each arm end position based on the positions of each arm end of the mechanical arm in the initial running trajectory and the obstacle positions of each obstacle in the to-be-detected image.

[0011] The positional features of the robotic arm and the relative positional features of each obstacle in the initial running trajectory are extracted respectively.

[0012] Based on the self-attention mechanism, the similarity between each positional feature and each relative positional feature is determined, and the obstacles to avoid in the initial running trajectory are determined.

[0013] According to the obstacle avoidance method for a robotic arm provided by the present invention, the step of determining the relative position of each obstacle to each arm end position based on the arm end positions in the initial running trajectory and the obstacle positions of each obstacle in the image to be detected includes:

[0014] The positions of the obstacles are converted to the same coordinate system as the position of the robotic arm's end, thus obtaining the converted positions of the obstacles.

[0015] Based on the positions of each end of the robotic arm and the transformed positions of each obstacle, the relative positions of each obstacle with respect to each end of the robotic arm are obtained.

[0016] According to the obstacle avoidance method for a robotic arm provided by the present invention, the step of obtaining the relative positions of each obstacle with respect to each arm end position based on the positions of each arm end of the robotic arm and the transformed positions of each obstacle includes:

[0017] When the changed position of the obstacle is within the preset perception threshold range of the robotic arm, the relative position of each obstacle to each arm end position is obtained based on the position of each arm end of the robotic arm and the changed position of the obstacle.

[0018] According to the obstacle avoidance method for a robotic arm provided by the present invention, the step of obtaining the relative positions of each obstacle with respect to each arm end position based on the positions of each arm end of the robotic arm and the transformed positions of each obstacle further includes:

[0019] If the change position of an obstacle exceeds the preset perception threshold range of the robotic arm, the relative position of each obstacle to each arm end position is determined based on the position of each arm end of the robotic arm and the preset perception threshold range of the robotic arm.

[0020] According to the present invention, an obstacle avoidance method for a robotic arm, wherein adjusting the initial trajectory based on the obstacle avoidance to obtain the trajectory to be executed includes:

[0021] Based on dynamic motion primitives, the initial running trajectory is described to obtain the trajectory code of the initial running trajectory;

[0022] The obstacle avoidance is incorporated into the trajectory encoding, and the initial running trajectory is adjusted to obtain the running trajectory to be executed.

[0023] According to the obstacle avoidance method for a robotic arm provided by the present invention, the step of acquiring each obstacle in the image to be detected includes:

[0024] Based on the obstacle detection model and the image to be detected, each obstacle is obtained;

[0025] The obstacle detection model is obtained by fine-tuning a pre-trained target detection network based on the YOLOv8 algorithm, using sample images.

[0026] The present invention also provides an obstacle avoidance device for a robotic arm, comprising:

[0027] The acquisition unit acquires the initial running trajectory of the robotic arm and the image to be detected;

[0028] The obstacle prediction unit, based on an attention mechanism and the positions of each arm end of the robotic arm in the initial running trajectory and the obstacle positions of each obstacle in the image to be detected, determines the obstacles to avoid in the initial running trajectory.

[0029] The obstacle avoidance unit adjusts the initial running trajectory based on the obstacle to obtain the running trajectory to be executed.

[0030] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the obstacle avoidance method of the robotic arm as described above.

[0031] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the obstacle avoidance method of the robotic arm as described above.

[0032] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the obstacle avoidance method of the robotic arm as described above.

[0033] The obstacle avoidance method, device, electronic device, and storage medium for robotic arms provided by this invention replan the robotic arm's trajectory online in multi-obstacle scenarios through an attention mechanism. This enables fast and accurate obstacle avoidance planning even when multiple obstacles suddenly appear, thus improving the versatility of obstacle avoidance planning for robotic arms. Attached Figure Description

[0034] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0035] Figure 1 This is one of the flowcharts illustrating the obstacle avoidance method for the robotic arm provided by the present invention;

[0036] Figure 2 This is a flowchart illustrating the method for determining and avoiding obstacles provided by the present invention;

[0037] Figure 3 This is the second flowchart illustrating the obstacle avoidance method for the robotic arm provided by the present invention;

[0038] Figure 4 This is a schematic diagram of the obstacle avoidance device for the robotic arm provided by the present invention;

[0039] Figure 5 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0040] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0041] Search-based trajectory planning methods typically use random search to rapidly explore paths within the robot's configuration space, finding feasible solutions for the robot manipulator to reach its target. However, the randomness in the search process can lead to inconsistent paths for the same obstacle and can also result in high computational costs. Optimization-based trajectory planning methods integrate the robot's kinematic model with a cost function to optimize and find the optimal trajectory. These methods can find obstacle avoidance paths given a known robot workspace, but still require manual design of the cost function and constraints. Especially in multi-obstacle scenarios, search-based or optimization-based algorithms do not perform ideally for obstacle avoidance by robotic arms.

[0042] To address the aforementioned problems, this invention provides an obstacle avoidance method for robotic arms, enabling rapid and accurate obstacle avoidance planning in multi-obstacle scenarios. Figure 1 This is one of the flowcharts illustrating the obstacle avoidance method for the robotic arm provided by the present invention, such as... Figure 1As shown, the method includes:

[0043] Step 110: Obtain the initial running trajectory of the robotic arm and the image to be detected;

[0044] Here, the robotic arm can be a robotic arm or other types of robotic arms, such as those used for tasks like handling, assembly, welding, and painting. The initial trajectory of the robotic arm refers to its path from the starting position to the target position, reflecting its route and operational parameters, such as speed and acceleration. The image to be detected refers to the image of the current operating environment acquired by the robotic arm at its starting position, reflecting information about obstacles within that environment. Understandably, obstacles may exist in the initial trajectory, requiring the robotic arm to adjust its initial trajectory to successfully reach the target position. These obstacles can be detected using the acquired image to be detected.

[0045] Specifically, the initial trajectory of the robotic arm can be determined based on the task it needs to perform. For example, if the robotic arm needs to place object A onto object B, its initial trajectory could be a path based on the position of object A as the initial position and object B as the target position. Additionally, a camera or other image acquisition device can be mounted on the robotic arm to obtain images of the operating environment to be inspected.

[0046] Step 120: Based on the attention mechanism, the position of each arm end of the robotic arm in the initial running trajectory and the obstacle position of each obstacle in the image to be detected, determine the obstacles to avoid in the initial running trajectory;

[0047] Here, the arm-end positions in the initial trajectory refer to the arm-end positions at various points along the initial trajectory. These arm-end positions can be the positions of the arm's extremities. It's understood that the arm-end positions change as the robotic arm moves along the initial trajectory. Similarly, the obstacle positions in the detection image can remain constant within the same image. Furthermore, obstacle avoidance here refers to obstacles that might obstruct the robotic arm's movement along the initial trajectory; the robotic arm must avoid these obstacles according to the initial trajectory.

[0048] Specifically, the positional features of each end-point of the robotic arm in the initial trajectory, and the positional features of each obstacle in the image to be detected, can be analyzed using an attention mechanism to determine the potential relationships between the positions of each end-point of the robotic arm and the positions of each obstacle in the initial trajectory. For example, the distance relationships between each end-point of the robotic arm and each obstacle in the initial trajectory can be analyzed, and obstacles with distance relationships less than a certain threshold can be considered as obstacles for the robotic arm to avoid in the initial trajectory. It is understandable that fusing local robotic arm end-point position information with global obstacle position information enhances the versatility of obstacle avoidance prediction.

[0049] It should be noted that in the obstacle avoidance planning task of the robotic arm, there may be multiple obstacles in the detected image. Existing technologies attempt to learn the obstacle avoidance function directly from the demonstration through end-to-end learning methods, but due to the influence of the input dimension of the mapping model, it is difficult to directly apply it to the case of multiple obstacles.

[0050] This invention utilizes an attention mechanism to process multi-dimensional features. This attention mechanism can handle the multi-dimensional features of multiple obstacles, and can then be applied to obstacle avoidance planning tasks for robotic arms in multi-obstacle scenarios. This allows the obstacle avoidance planning to be generalized to multiple obstacle scenarios. It should also be noted that the position, shape, and size of obstacles in the detected image may differ. Some obstacles may be located on the initial trajectory, but due to their shape or size, they are insufficient to obstruct or otherwise affect the robotic arm's initial trajectory. Therefore, the attention mechanism can focus on key information and ignore unimportant details, thus identifying the obstacles that truly require avoidance and providing optimal obstacle avoidance planning.

[0051] Step 130: Based on the obstacle avoidance, adjust the initial running trajectory to obtain the running trajectory to be executed.

[0052] Specifically, the initial trajectory can be represented by dynamic motion primitives. By incorporating obstacle avoidance into the dynamic equations representing the initial trajectory, the initial trajectory can be adjusted. Once the planned new trajectory to be executed for obstacle avoidance is planned, it can be sent to the robotic arm control box so that the robotic arm can be controlled to move to the new position.

[0053] The method provided in this invention re-plans the robotic arm's trajectory online in multi-obstacle scenarios through an attention mechanism, enabling fast and accurate obstacle avoidance planning even when multiple obstacles suddenly appear, thus improving the versatility of the robotic arm in obstacle avoidance planning.

[0054] Based on any of the above embodiments, step 120 includes:

[0055] Based on the positions of each arm end of the robotic arm in the initial running trajectory and the positions of each obstacle in the image to be detected, the relative positions of each obstacle and each arm end are determined.

[0056] The positional features of the robotic arm and the relative positional features of each obstacle in the initial running trajectory are extracted respectively.

[0057] Based on the self-attention mechanism, the similarity between each positional feature and each relative positional feature is determined, and the obstacles to avoid in the initial running trajectory are determined.

[0058] Here, the positional features of the robotic arm can reflect its two-dimensional or three-dimensional positional information in the initial running trajectory, such as the horizontal and vertical coordinates of the robotic arm's end effector. Similarly, the relative positional features of the obstacle can reflect its two-dimensional or three-dimensional positional information relative to the robotic arm's end effector, such as the horizontal and vertical coordinates of the obstacle with the robotic arm's end effector as the origin of the coordinate system.

[0059] Specifically, firstly, the obstacle positions in the image to be detected and the arm-end positions of the robotic arm can be transformed to the same coordinate system. For example, the obstacle positions can be transformed to the coordinate system of the robotic arm's arm-end positions. Next, the relative positions of each obstacle to its respective arm-end position can be obtained using the coordinate systems of the robotic arm's arm-end positions and the obstacle positions of each obstacle within those coordinate systems. For example, if the arm-end position can be denoted as (X0, Y0), then the relative position of obstacle N can be denoted as (ΔX...). N ,ΔY N ).

[0060] Furthermore, the encoder can be used to extract the positional features of each robotic arm in the initial running trajectory, as well as the relative positional features of each obstacle. Finally, for the end-effector position of a single robotic arm, a self-attention mechanism can be used to analyze the similarity between the positional features of the robotic arm and the relative positional features of each obstacle. Obstacles with similarity less than a certain threshold can be used as obstacles to avoid in the initial running trajectory.

[0061] In one embodiment, Figure 2 This is a flowchart illustrating the method for determining and avoiding obstacles provided by the present invention, as shown below. Figure 2 As shown, the method includes: First, inputting a tensor like Figure 2ManipΔX0,ΔY0 shown in represents the positions of the arm ends of the robotic arm in the initial running trajectory; the input tensor also includes Ob.1ΔX1,ΔY1 to Ob.NΔX N ,ΔY N , which represents the relative position of obstacle 1... the relative position of obstacle N. Then, each tensor can be encoded by a multi-layer perceptron with 2 input nodes, d e output nodes, and a single hidden layer with 64 nodes, that is, the information embedding process, to obtain the position features Embed.0 of the robotic arm in the initial running trajectory, and the relative position features Embed.1... Embed.N of each obstacle. Here, the arm end positions and each obstacle can be regarded as a single entity, so there are a total of N + 1 entities here, and the feature of each entity is a position feature with d e dimensions. Then, the position features of the robotic arm and the relative position features of each obstacle are input into an attention mechanism layer containing m heads. Through the self-attention mechanism, a corresponding query L q , key L k , and value L v linear transformation of the position features, and a corresponding key L k and value L v linear transformation of the relative position features. For example, where where j represents the j-th head (where 0 ≤ j < m), and at the same time let m×d k = d e , then for the first entity, that is, the arm end position of the robotic arm, a single query needs to be calculated, which can be calculated by the following formula:

[0062]

[0063] where, represents the linear transformation result of the query of the j-th attention head; Embed0 represents the position feature of the arm end position; represents the query of the j-th attention head.

[0064] For all entities, that is, including the arm end positions of the robotic arm and each obstacle, the transformation of the key and value needs to be calculated, which can be calculated by the following formula:

[0065]

[0066] In the formula, represents the linear transformation result of the key of the i-th entity; Embed i represents the position feature (relative position feature) of the i-th entity; represents the key of the j-th attention head; This represents the linear transformation result of the i-th entity value; This represents the value of the j-th attention head.

[0067] Therefore, we can obtain the linear transformation result of the query obtained by the first entity through j attention heads. The linear transformation result of the keys obtained by all entities through j attention heads The linear transformation result of the values ​​obtained by all entities through j attention heads The similarity between each positional feature and each relative positional feature can then be calculated using the following equation:

[0068]

[0069] In the formula, output j σ represents the relevance calculated by the j-th attention head; σ represents the softmax function; d k This represents the number of features queried by the manipulator entity after processing by each head. Therefore, the outputs of the m attention heads can ultimately be fused: Output = Concat([output0,…,output...)). m-1 ]).

[0070] Finally, the similarity can be decoded by a decoder to obtain the corresponding obstacle to avoid. Here, the decoder can consist of a multilayer perceptron to produce the final output C. x A multilayer perceptron consists of two hidden layers: an input layer and an output layer. The input layer has d... e The system has 64 nodes per hidden layer and one node per output layer. The ReLU function is used as the activation function for the entire architecture. For example, obstacle avoidance can be achieved using the following equation:

[0071]

[0072] In the formula, This represents the predicted obstacle to avoid; MLP stands for Multilayer Perceptron, and FC stands for Fully Connected. It should be noted that the mean squared error can be used as the loss function when training the encoder / decoder.

[0073] It should be noted that this approach extracts the positional features of the robotic arm's end effector and the relative positional features of each obstacle. This means that a single encoding is used to represent the features of the robotic arm's end effector and each obstacle, eliminating the need to consider the interaction between other parts of the robotic arm and the environment. This allows the self-attention mechanism to focus only on the positional relationship between the robotic arm's end effector and the remaining obstacles, significantly reducing computational load and making it better suited for obstacle avoidance scenarios. Furthermore, extracting the relative positional features of each obstacle relative to the robotic arm further reduces computational load.

[0074] Based on any of the above embodiments, determining the relative position of each obstacle to each arm end position based on the arm end positions of the robotic arm in the initial running trajectory and the obstacle positions of each obstacle in the image to be detected includes:

[0075] The positions of the obstacles are converted to the same coordinate system as the position of the robotic arm's end, thus obtaining the converted positions of the obstacles.

[0076] Based on the positions of each end of the robotic arm and the transformed positions of each obstacle, the relative positions of each obstacle with respect to each end of the robotic arm are obtained.

[0077] Specifically, the transformed positions of each obstacle are obtained by converting their positions to the same coordinate system as the robotic arm's end position. This can be achieved using the following equation:

[0078]

[0079] In the formula, (X i ,Y i Z i () represents the change position of the i-th obstacle, 1≤i≤N, where N represents the total number of obstacles; R c Represents the rotation matrix; t c Represents the translation vector; u i The pixel x-coordinate of the obstacle; v i The pixel ordinate of the obstacle; parameter c x and c y Let represent the principal point of the camera; focal represents the camera's focal length; and depth represents the distance from the camera to the obstacle. Here, obstacle avoidance can be considered only in two-dimensional space, and the set of obstacle transition positions can be represented as pos. real ={{X1,Y1},...,{X N ,Y N}}.

[0080] Next, using each arm-end position as the origin of the coordinate system, the relative position of each obstacle to the arm-end position can be represented by the positional difference between the transformed position of each obstacle and the arm-end position. For example, the relative position of obstacle N to the arm-end position can be represented by (ΔX). N ,ΔY N )express.

[0081] The method provided in this invention converts the obstacle positions of each obstacle into the same coordinate system as the arm end position of the robotic arm, thereby obtaining the converted positions of each obstacle. Based on the arm end positions of the robotic arm and the converted positions of each obstacle, the relative positions of each obstacle with respect to each arm end position are obtained. This achieves the representation of the position information of each obstacle in the same coordinate system, with the arm end position as the origin, by using the relative positions of each obstacle with respect to each arm end position. This further reduces the amount of calculation required to avoid obstacles and improves the obstacle avoidance performance of the robotic arm.

[0082] Based on any of the above embodiments, obtaining the relative positions of each obstacle with respect to each arm end position based on the arm end positions of the robotic arm and the transformed positions of each obstacle includes:

[0083] When the changed position of the obstacle is within the preset perception threshold range of the robotic arm, the relative position of each obstacle to each arm end position is obtained based on the position of each arm end of the robotic arm and the changed position of the obstacle.

[0084] If the change position of an obstacle exceeds the preset perception threshold range of the robotic arm, the relative position of each obstacle to each arm end position is determined based on the position of each arm end of the robotic arm and the preset perception threshold range of the robotic arm.

[0085] It should be noted that the robotic arm's preset perception range can be the size of the robot's overall perception range, while the detected image can be obtained from a camera mounted on the robot, and the area contained in its pixels is usually larger than the robotic arm's preset perception range. Therefore, some obstacles will inevitably be outside the robotic arm's preset perception range. To improve the robotic arm's obstacle avoidance generalization, when an obstacle's transition position exceeds the robotic arm's preset perception threshold, the horizontal and vertical coordinates of the obstacles exceeding that threshold can be projected onto the boundary, for example, using the following equation:

[0086]

[0087] In the formula, ΔX0 represents the x-coordinate of the end-effector position of the robotic arm; ΔX N The x-coordinate represents the relative position of the Nth obstacle; clip represents the adjustment function that limits the range of the coordinates; (-X thresh ,X thresh ) represents the width of the preset sensing threshold range; ΔY0 represents the ordinate of the robotic arm's end-effector position; ΔY N The ordinate of the relative position of the Nth obstacle; (-Y thresh ,Y thresh ) indicates the length of the preset perception threshold range.

[0088] The method provided in this invention greatly improves the generalization of obstacle avoidance by projecting obstacles that exceed the preset perception threshold range of the robotic arm onto the perception boundary.

[0089] Based on any of the above embodiments, step 130 includes:

[0090] Based on dynamic motion primitives, the initial running trajectory is described to obtain the trajectory code of the initial running trajectory;

[0091] The obstacle avoidance is incorporated into the trajectory encoding, and the initial running trajectory is adjusted to obtain the running trajectory to be executed.

[0092] Specifically, the initial trajectory here can be obtained through... It means that, among them, Indicates the initial state. This represents the target state of the initial trajectory. Therefore, the initial trajectory is encoded using dynamic motion primitives to obtain its trajectory code, which can be expressed by the following formula:

[0093]

[0094] In the formula, τ is the time scaling factor, used to change the velocity of the trajectory, thereby obtaining trajectories with different convergence velocities; y represents the position of the trajectory. Indicates speed, α represents acceleration; β and α are constant terms; g is the endpoint of the learned trajectory; f is the forcing function, a nonlinear term, and a trajectory shape learner.

[0095] Where f can be defined by the following formula:

[0096]

[0097] Ψ i (x)=exp(-0.5h i (xc i ) 2 )

[0098]

[0099] In the formula, f(x) represents the forcing function; the specific ω i The coefficients can be obtained using locally weighted regression; This is the initial state; x is the clock system variable, ensuring its trajectory is independent of time; N is the number of basis functions; the more basis functions, the smoother the generalized target trajectory; Ψ i (x) represents the Gaussian kernel function; hi and c i These are the bandwidth and center of the Gaussian kernel function, respectively; α x Represents a constant term; The first derivative in time.

[0100] Next, the predicted obstacle avoidance can be incorporated into the trajectory encoding to adjust the initial trajectory and obtain the trajectory to be executed. This can be achieved, for example, using the following equation:

[0101]

[0102] In the formula, The forced function encoding representing the initial running trajectory, This indicates the change in coupling terms caused by avoiding obstacles.

[0103] The method provided in this invention introduces an attention mechanism network into the dynamic motion primitive framework, which enables the robotic arm to have the advantage of anti-interference and better obstacle avoidance performance.

[0104] Based on any of the above embodiments, the step of acquiring each obstacle in the image to be detected includes:

[0105] Based on the obstacle detection model and the image to be detected, each obstacle is obtained;

[0106] The obstacle detection model is obtained by fine-tuning a pre-trained target detection network based on the YOLOv8 algorithm, using sample images.

[0107] It should be noted that fine-tuning a pre-trained object detection network based on the YOLOv8 algorithm requires only a small amount of training data to train an accurate obstacle detection model. Furthermore, obstacle detection based on the YOLOv8 algorithm yields better detection performance and more accurate results.

[0108] Based on any of the above embodiments Figure 3 This is the second flowchart illustrating the obstacle avoidance method for the robotic arm provided by this invention, as shown below. Figure 3 As shown, the method includes:

[0109] First, the image to be detected is acquired, and task parameters are estimated, i.e., the positions of obstacles in the image are obtained. Next, the position of the robotic arm's end effector in the initial trajectory is acquired. Then, using the obstacle positions and the robotic arm's end effector position, an input tensor is obtained, representing the relative positions of the robotic arm's end effector and obstacles to the end effector. Next, the input tensor is fed into the encoder in an attention-based coupling term prediction network to extract positional features, and the similarity between the positional features of the robotic arm's end effector and the relative positional features of each obstacle is analyzed using the attention mechanism. This similarity is then input into the decoder for decoding, predicting the obstacle to be avoided, i.e., obtaining the coupling term. Finally, the task-specific coupling is incorporated into the clock-signal-driven motion dynamics primitive (DMP) of the initial trajectory to obtain the trajectory to be executed. Understandably, after the robotic arm completes the trajectory to be executed, it can continue to acquire detection images to perform obstacle avoidance for the next trajectory, until obstacle avoidance under the specific task is completed. During the training phase of the attention-based coupling term prediction network, obstacle avoidance can be obtained by subtracting the trajectory encoding of the initial running trajectory from the trajectory encoding of the obstacle avoidance trajectory. For example, it can be calculated using the following formula:

[0110]

[0111] C x Indicates avoiding an obstacle; Forced function encoding representing obstacle avoidance trajectory; The forced function encoding represents the initial running trajectory.

[0112] Based on any of the above embodiments Figure 4 This is a schematic diagram of the obstacle avoidance device for the robotic arm provided by the present invention, as shown below. Figure 4 As shown, the device includes:

[0113] The acquisition unit 410 acquires the initial running trajectory of the robotic arm and the image to be detected;

[0114] The obstacle prediction unit 420, based on an attention mechanism and the positions of each arm end of the robotic arm in the initial running trajectory and the obstacle positions of each obstacle in the image to be detected, determines the obstacles to avoid in the initial running trajectory.

[0115] The obstacle avoidance unit 430 adjusts the initial running trajectory based on the obstacle avoidance to obtain the running trajectory to be executed.

[0116] The device provided in this invention re-plans the robotic arm's trajectory online in multi-obstacle scenarios through an attention mechanism, enabling fast and accurate obstacle avoidance planning even when multiple obstacles suddenly appear, thus improving the generalization of obstacle avoidance planning for the robotic arm.

[0117] Based on any of the above embodiments, the obstacle prediction unit is specifically used for:

[0118] Based on the positions of each arm end of the robotic arm in the initial running trajectory and the positions of each obstacle in the image to be detected, the relative positions of each obstacle with respect to each arm end are determined.

[0119] The positional features of the robotic arm and the relative positional features of each obstacle in the initial running trajectory are extracted respectively.

[0120] Based on the self-attention mechanism, the similarity between each positional feature and each relative positional feature is determined, and the obstacles to avoid in the initial running trajectory are determined.

[0121] Based on any of the above embodiments, the obstacle prediction unit is further specifically used for:

[0122] The positions of the obstacles are converted to the same coordinate system as the position of the robotic arm's end, thus obtaining the converted positions of the obstacles.

[0123] Based on the positions of each end of the robotic arm and the transformed positions of each obstacle, the relative positions of each obstacle with respect to each end of the robotic arm are obtained.

[0124] Based on any of the above embodiments, the obstacle prediction unit is further specifically used for:

[0125] When the changed position of the obstacle is within the preset perception threshold range of the robotic arm, the relative position of each obstacle to each arm end position is obtained based on the position of each arm end of the robotic arm and the changed position of the obstacle.

[0126] Based on any of the above embodiments, the obstacle prediction unit is further specifically used for:

[0127] If the change position of an obstacle exceeds the preset perception threshold range of the robotic arm, the relative position of each obstacle to each arm end position is determined based on the position of each arm end of the robotic arm and the preset perception threshold range of the robotic arm.

[0128] Based on any of the above embodiments, the obstacle avoidance unit is specifically used for:

[0129] Based on dynamic motion primitives, the initial running trajectory is described to obtain the trajectory code of the initial running trajectory;

[0130] The obstacle avoidance is incorporated into the trajectory encoding, and the initial running trajectory is adjusted to obtain the running trajectory to be executed.

[0131] Based on any of the above embodiments, the acquisition unit further includes a detection unit, which is specifically used for:

[0132] Based on the obstacle detection model and the image to be detected, each obstacle is obtained;

[0133] The obstacle detection model is obtained by fine-tuning a pre-trained target detection network based on the YOLOv8 algorithm, using sample images.

[0134] Figure 5 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 5 As shown, the electronic device may include a processor 510, a communication interface 520, a memory 530, and a communication bus 540, wherein the processor 510, the communication interface 520, and the memory 530 communicate with each other via the communication bus 540. The processor 510 can call logical instructions in the memory 530 to execute an obstacle avoidance method for the robotic arm. The method includes: acquiring the initial running trajectory of the robotic arm and an image to be detected; determining the obstacles to avoid in the initial running trajectory based on an attention mechanism, and the positions of each arm end of the robotic arm in the initial running trajectory and the obstacle positions of each obstacle in the image to be detected; and adjusting the initial running trajectory based on the obstacle avoidance to obtain the running trajectory to be executed.

[0135] Furthermore, the logical instructions in the aforementioned memory 530 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0136] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the obstacle avoidance method for the robotic arm provided by the above methods. The method includes: acquiring an initial running trajectory of the robotic arm and an image to be detected; determining obstacles to avoid in the initial running trajectory based on an attention mechanism, the positions of each end of the robotic arm in the initial running trajectory, and the obstacle positions of each obstacle in the image to be detected; and adjusting the initial running trajectory based on the obstacle avoidance to obtain a running trajectory to be executed.

[0137] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements an obstacle avoidance method for a robotic arm provided by the methods described above. The method includes: acquiring an initial running trajectory of the robotic arm and an image to be detected; determining obstacles to be avoided in the initial running trajectory based on an attention mechanism, and the positions of each end of the robotic arm in the initial running trajectory and the obstacle positions of each obstacle in the image to be detected; and adjusting the initial running trajectory based on the obstacle avoidance to obtain a running trajectory to be executed.

[0138] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0139] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0140] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for obstacle avoidance of a robot arm, the method comprising: The method comprises the following steps: obtaining an initial running trajectory of a robot arm and a to-be-detected image; determining an obstacle-avoiding position in the initial running trajectory based on an attention mechanism and positions of each arm end of the robot arm in the initial running trajectory and positions of each obstacle in the to-be-detected image; adjusting the initial running trajectory based on the obstacle-avoiding position to obtain a to-be-executed running trajectory; the step of determining the obstacle-avoiding position in the initial running trajectory based on the attention mechanism and the positions of each arm end of the robot arm in the initial running trajectory and the positions of each obstacle in the to-be-detected image comprises the following steps: determining relative positions of each obstacle and each arm end of the robot arm based on the positions of each arm end of the robot arm in the initial running trajectory and the positions of each obstacle in the to-be-detected image; respectively extracting position features of each arm end of the robot arm in the initial running trajectory and relative position features of each obstacle; determining similarities between the position features and the relative position features based on a self-attention mechanism to determine the obstacle-avoiding position in the initial running trajectory.

2. The method of claim 1, wherein, the step of determining the relative positions of each obstacle and each arm end of the robot arm based on the positions of each arm end of the robot arm in the initial running trajectory and the positions of each obstacle in the to-be-detected image comprises the following steps: converting the positions of each obstacle into converted positions of each obstacle in the same coordinate system as the positions of each arm end of the robot arm to obtain the converted positions of each obstacle; obtaining the relative positions of each obstacle and each arm end of the robot arm based on the positions of each arm end of the robot arm and the converted positions of each obstacle.

3. The method of claim 2, wherein, the step of obtaining the relative positions of each obstacle and each arm end of the robot arm based on the positions of each arm end of the robot arm and the converted positions of each obstacle comprises the following steps: when the converted position of an obstacle is within a preset perception threshold range of the robot arm, obtaining the relative positions of each obstacle and each arm end of the robot arm based on the positions of each arm end of the robot arm and the converted position of the obstacle.

4. The method of claim 2, wherein, when the converted position of an obstacle exceeds the preset perception threshold range of the robot arm, determining the relative positions of each obstacle and each arm end of the robot arm based on the positions of each arm end of the robot arm and the preset perception threshold range of the robot arm. the step of adjusting the initial running trajectory based on the obstacle-avoiding position to obtain the to-be-executed running trajectory comprises the following steps:

5. The method of claim 1-4, wherein, describing the initial running trajectory based on a dynamic motion primitive to obtain a trajectory code of the initial running trajectory; including the obstacle-avoiding position in the trajectory code to adjust the initial running trajectory and obtain the to-be-executed running trajectory. the step of obtaining each obstacle in the to-be-detected image comprises the following steps:

6. The method of claim 1-4, wherein, obtaining each obstacle based on an obstacle detection model and the to-be-detected image. ​ The obstacle detection model is obtained by fine-tuning a pre-trained target detection network based on a YOLOv8 algorithm based on sample images.

7. An obstacle avoidance device for a robotic arm, characterized in that, Comprise: An acquisition unit acquires an initial running trajectory of a robot arm and a to-be-detected image; An obstacle prediction unit determines an obstacle-avoiding position in the initial running trajectory based on an attention mechanism and positions of each arm end of the robot arm in the initial running trajectory and positions of each obstacle in the to-be-detected image; An obstacle avoidance unit adjusts the initial running trajectory based on the obstacle-avoiding position to obtain a to-be-executed running trajectory; The obstacle prediction unit: determines relative positions of each obstacle and each arm end in the initial running trajectory based on the positions of each arm end of the robot arm in the initial running trajectory and the positions of each obstacle in the to-be-detected image; extracts features of each position of the robot arm in the initial running trajectory and features of the relative positions of each obstacle, respectively; determines similarities between each position feature and each relative position feature based on a self-attention mechanism to determine an obstacle-avoiding position in the initial running trajectory.

8. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the program to implement the obstacle avoidance method of the robot arm of any one of claims 1 to 6. 9.A non-transitory computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the obstacle avoidance method of the robot arm of any one of claims 1 to 6.

Images