Arm path planning method based on attention mechanism and hierarchical reinforcement learning

Abstract

The application discloses a kind of mechanical arm path planning method based on attention mechanism and hierarchical reinforcement learning, belong to the cross technical field of robot motion planning and artificial intelligence.The application is by collecting the state of mechanical arm and the depth data detected by camera, and by improved ViT encoder, the state of mechanical arm and visual data are fused, and the observed state is assigned weight by attention mechanism;Secondly, a hierarchical reinforcement learning framework is used to generate a path, reducing the difficulty of training, constructing a composite reward function to update and adjust the network parameters, and achieving approximate solution of the optimal path.The application fully considers the approximation safety in dense and unstructured environment, improves the planning efficiency and real-time performance, meets the demand of online autonomous planning, and gets rid of the restriction that traditional path planning method can only solve convex optimization problems, and overcomes the shortcomings of high complexity and poor real-time performance in solving multi-constraint optimization problems.

Description

Technical Field

[0001] This invention belongs to the interdisciplinary field of robot motion planning and artificial intelligence, specifically relating to a robotic arm path planning method based on attention mechanism and hierarchical reinforcement learning. Background Technology

[0002] In mass production lines, robotic manufacturing and assembly require engineers to pre-plan the robot's path and guide it through teaching methods. However, in unstructured industrial scenarios, tasks involve densely packed structural components with varying geometries, sizes, and physical characteristics, severely limiting the reach of the end effector and arm. This makes collisions highly likely during approach, seriously threatening the safety of the manufacturing process. Traditional planning methods typically rely on manually designed rules and adjustments, resulting in low efficiency and susceptibility to environmental changes. The nonlinear strong coupling between robot dynamics and motion models, along with the nonlinearity and nonconvexity of constraints, leads to nonconvex optimization problems. Existing planning methods cannot handle these multi-constraint optimal planning problems or suffer from high algorithm complexity and poor real-time performance. In contrast, deep reinforcement learning algorithms do not require physical modeling of the environment. They can continuously learn from the environment to improve the performance of robotic arm path planning and adaptively respond to environmental changes, thereby improving the efficiency and accuracy of path planning.

[0003] Deep reinforcement learning can continuously adjust and update strategies based on environmental feedback during the learning process to adapt to the environment. It utilizes deep neural networks to learn the state-action mapping of robotic arms and can handle problems in high-dimensional state spaces and continuous action spaces. Therefore, as an emerging intelligent path planning algorithm for robotic arms, deep reinforcement learning has broad application prospects in the field of robotic arm path planning research and has become one of the current research hotspots. However, most current methods focus on the direct motion control of the robotic arm, rarely considering the presence of obstacles (e.g., patent publication number: CN 119897848 A), or only considering the obstacle avoidance of the end effector (e.g., patent publication number: CN 119610096 A), as well as obstacles that can be regularized (e.g., Zhang T, Zhang K, Lin J, et al, “Sim2real learning of obstacle avoidance forrobotic manipulators in uncertain environments,” IEEE Robotics and Automation Letters, vol. 7, no. 1, pp. 65-72, 2021.). Summary of the Invention

[0004] To address the aforementioned technical problems, this invention provides a robotic arm path planning method based on attention mechanisms and hierarchical reinforcement learning. This method overcomes the limitation of traditional path planning methods, which can only solve convex optimization problems, and also overcomes the shortcomings of high complexity and poor real-time performance in solving multi-constraint optimization problems. By utilizing the current and predicted information of the target's motion state, the network parameters are updated and adjusted to achieve an approximate solution for the optimal path.

[0005] This invention discloses a robotic arm path planning method based on attention mechanism and hierarchical reinforcement learning, the method comprising:

[0006] Collect the location information of obstacles, and determine the target location area for the robotic arm path planning task based on the provided location information;

[0007] Determine the target position of the robotic arm in the current training round within the target position region;

[0008] The current state vector of the robotic arm and the visual data detected by the camera are obtained, and the robotic arm state and visual data are fused through a preset ViT encoder to obtain a potential fused feature representation.

[0009] Based on the potential fusion feature representation, a hierarchical reinforcement learning model is used to generate the joint angle motion of the robotic arm. The hierarchical reinforcement learning model includes a low-level policy layer and a high-level policy layer connected in series. The low-level policy layer is used to obtain a first motion vector based on the potential fusion feature representation. The high-level policy layer is used to output the joint angle motion of the robotic arm based on the first motion vector and the robotic arm state vector. The first motion vector is the predicted position of the robotic arm end effector relative to the world coordinate system.

[0010] After the robotic arm performs the current joint angle action, the strategy is updated through a composite reward function until the robotic arm end effector reaches the target position.

[0011] As can be seen from the above technical solutions, the present invention has the following beneficial effects:

[0012] This invention collects robotic arm state data and camera-detected depth data, and fuses the robotic arm state and visual data using an improved ViT encoder. An attention mechanism is used to assign weights to the observed states and ignore redundant information. Secondly, a hierarchical reinforcement learning framework is employed to generate paths, reducing training difficulty. A reward function is constructed using "energy consumption-time-safety" as a performance indicator to update and adjust network parameters, achieving an approximate solution for the optimal path. This invention fully considers approximation safety in dense and unstructured environments, improves planning efficiency and real-time performance, meets the needs of online autonomous planning, and overcomes the limitations of traditional path planning methods that can only solve convex optimization problems. It also overcomes the drawbacks of high complexity and poor real-time performance in solving multi-constraint optimization problems, making it suitable for tasks such as shaft-hole assembly and dense obstacle avoidance in unstructured environments. Attached Figure Description

[0013] Figure 1 The present invention provides an overall flowchart of a robotic arm path planning method based on attention mechanism and hierarchical reinforcement learning. Detailed Implementation

[0014] To make the objectives, features, and advantages of the present invention more apparent and understandable, specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Several embodiments of the present invention are shown in the drawings. However, the present invention can be implemented in many different forms and is not limited to the embodiments described herein. Rather, these embodiments are provided so that the disclosure of the present invention will be thorough and complete.

[0015] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a particular order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0016] It should be noted that when an element is referred to as being "fixed to" another element, it can be directly on the other element or there may be an intervening element. When an element is considered to be "connected to" another element, it can be directly connected to the other element or there may be an intervening element. The terms "vertical," "horizontal," "left," "right," "up," "down," and similar expressions used herein are for illustrative purposes only and are not intended to indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be construed as limiting the invention.

[0017] In this invention, unless otherwise explicitly specified and limited, the terms "installation," "connection," "linking," "fixing," etc., should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal communication between two components. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances. The term "and / or" as used herein includes any and all combinations of one or more of the related listed items.

[0018] For obstacle avoidance path planning tasks in unstructured spaces, this invention provides a robotic arm path planning method based on attention mechanisms and hierarchical reinforcement learning. The aim is to overcome the limitation of traditional path planning methods, which can only solve convex optimization problems, by incorporating depth image features and considering multiple complex constraints such as obstacle avoidance and arm collision avoidance.

[0019] In one embodiment, the present invention discloses a robotic arm path planning method based on attention mechanisms and hierarchical reinforcement learning, such as... Figure 1 As shown, the specific steps include:

[0020] S101. Collect the location information of obstacles and determine the target location area of ​​the robotic arm path planning task based on the location information shown.

[0021] In one embodiment, the process of determining the target location region includes:

[0022] The obstacle and the robotic arm are enveloped by a group of line segments to approximate their shape. The shortest distance between the robotic arm and the obstacle is obtained by solving the minimum geometric spacing between the envelopes. The workspace of the robotic arm is determined based on the shortest distance between the robotic arm and the obstacle.

[0023] Within the workspace of the robotic arm, the target location area is set according to the path planning task.

[0024] Specifically, the location information of obstacles is first collected. Irregular obstacles and the robotic arm are both enveloped by line segment groups to flexibly approximate their shapes. The shortest distance between the robotic arm and the obstacle is then determined. The workspace of the robotic arm is obtained by solving for the minimum geometric distance between the envelopes, ensuring that the robotic arm will not collide with obstacles within the workspace. Within the workspace, a target location region is set according to the path planning task requirements, where the target location region is the area reachable by the robotic arm's end effector.

[0025] S102. Determine the target position of the robotic arm in the current training round within the target position area;

[0026] Specifically, a point can be sampled in the target location area as the target position of the robotic arm in the current training round.

[0027] S103. Obtain the current state vector of the robotic arm and the visual data detected by the camera, and fuse the robotic arm state and visual data through a preset ViT encoder to obtain a potential fusion feature representation.

[0028] By collecting the current state of the robotic arm and the depth data detected by the camera, and fusing the robotic arm state and visual data through the ViT encoder, the observed state is assigned weights and redundant information is ignored through the attention mechanism.

[0029] In one embodiment, the step of fusing the robotic arm state and visual data using a preset ViT encoder to obtain a potential fused feature representation specifically includes:

[0030] The visual data detected by the camera is divided into several non-overlapping image blocks according to a fixed size, and each image block is represented by a vector to obtain an image block sequence;

[0031] The state vector of the robotic arm is processed by a state encoder to obtain a state encoded vector. The state encoder is used to project the state vector of the robotic arm onto a feature space that is the same as the image features. The state vector of the robotic arm includes the joint angle of the robotic arm at any given time, the position of the target and the end effector relative to the world coordinate system, and the shortest distance between the robotic arm and the obstacle.

[0032] The initial input sequence is obtained by concatenating the state encoding vector, the image block vector sequence, and the position encoding matrix in sequence.

[0033] The initial input sequence is input into a preset ViT encoder for feature extraction, and a potential fusion feature representation is output. The preset ViT encoder models the global correlation between different image patches through a multi-head self-attention mechanism.

[0034] Specifically, let the input image, i.e., the depth data detected by the camera, be... ,in, For the input image Given the resolution and channel dimensions, the image is divided into several non-overlapping image blocks of a fixed size, with each image block having a spatial resolution of . Then it can be divided into Image patches, of which Indicates the number of image blocks.

[0035] For the After flattening the image blocks The index of the image patch can be used to obtain a length of The vector, all image patches can be represented as .in, This represents the image patch embedding sequence obtained by dividing and flattening the input image, the nth... The image blocks are denoted as To meet the ViT (Vision Transformer) encoder's requirement for input with a uniform feature dimension, each image patch needs to be projected onto a linear mapping. 3D feature space. Here, This represents the embedding dimension of the token, which is also the dimension of the hidden layer features of the Transformer encoder.

[0036] In one implementation case, , , , , , .

[0037] Furthermore, a learnable token is added before the image patch embedding sequence: And introduce a learnable position encoding matrix This preserves the spatial location information of each image patch within the original image. Therefore, the input sequence of the standard ViT encoder can be represented as:

[0038] (1)

[0039] in, Represents a linear projection operation, symbol This means that the classification labels and image embeddings are concatenated along the sequence dimension in sequence order. This is the initial classification tag sequence input into the standard ViT (Vision Transformer) encoder.

[0040] In order to process the multimodal input including the state of the robotic arm and the image, the ViT encoder is further adjusted to obtain the ViT encoder preset by the present invention.

[0041] Specifically, a special Token is defined, denoted as... From the state of the robotic arm It is obtained through mapping via a multilayer perceptron (MLP) network, specifically represented as follows:

[0042] (2)

[0043] in, This represents the multilayer perceptron mapping function, which projects the original robotic arm state onto the same feature space as the visual token, so that it can be uniformly encoded with image features in the future.

[0044] The state vector of the robotic arm is defined as follows:

[0045] (3)

[0046] in, for The joint angles of the robotic arm are constantly monitored. and These represent the positions of the target and the end effector relative to the world coordinate system, respectively. This represents the shortest distance between the robotic arm and the obstacle.

[0047] Based on this, to meet the requirement of joint modeling of state and visual information in robotic arm trajectory planning tasks, the robotic arm state is encoded into a state token using a state encoder. Together with the image token, they constitute the multimodal initial input sequence. Let the input embedding be... Position embedding is Enter the preset initial input sequence of the ViT encoder. It can be represented as:

[0048] (4)

[0049] (5)

[0050] Through this design, the ViT encoder can not only model the spatial structural relationships within a visual image, but also explicitly fuse the robotic arm's motion state with task target information, thereby enhancing decision-making and perception capabilities in complex obstacle environments.

[0051] In one embodiment, the preset ViT encoder includes two identical coding layers connected in series, with multiple attention heads set in parallel for each layer. The initial input sequence is input into the preset ViT encoder for feature extraction, and the output is a latent fusion feature representation, including:

[0052] The initial input sequence is input into the first coding layer to obtain the coding result, and then the output coding result of the first coding layer is input into the second coding layer to obtain the latent fusion feature representation.

[0053] Specifically, the above multimodal initial input sequence is input into a preset ViT encoder, and the first... The update process of each coded block can be represented as:

[0054] (6)

[0055] in, Presentation layer normalization operation, This represents the mapping function of a multilayer perceptron. This indicates a fully connected mapping. This represents the function for the multi-head self-attention mechanism. , and They represent the first The input of the encoded block, the ... The output of the attention sublayer of the first coded block and the first The final output consists of coded blocks, where... .

[0056] In one embodiment, the execution steps of the multi-head self-attention mechanism function include:

[0057] The initial input sequence is transformed linearly to obtain the query vector, key vector, and value vector.

[0058] Based on the query vector, key vector, and value vector, the output of each attention head is calculated using a scaled dot product attention method.

[0059] The outputs of each attention head are concatenated and then linear projection is performed to obtain the output of the multi-head self-attention mechanism.

[0060] Specifically, after obtaining the initial input sequence, it is input into a pre-defined ViT encoder for feature extraction. During the encoding process, the ViT encoder models the global correlation between different image patches through a multi-head self-attention (MSA) mechanism, the expression of which is:

[0061] (7)

[0062] in, , and These represent the query vector, key vector, and value vector matrices, respectively. Indicates the number of attention heads; Indicates the first The self-attention computation process corresponding to each attention head is a structure that enables the model to characterize the correlation between image patches from different feature subspaces by setting multiple attention heads in parallel, thereby enhancing the ability to represent the global geometric structure of complex obstacle environments.

[0063] The calculation process for a single attention head uses the scaled dot product attention form, and its expression is:

[0064] (8)

[0065] in, This represents the dimension of the key vector and is also a scaling factor used to prevent the softmax function from entering regions with excessively small gradients due to excessively large inner product values. (Matrix) , and It is obtained from the initial input sequence through a linear transformation, specifically as follows:

[0066] (9)

[0067] Considering the requirements of computational efficiency and lightweight model in trajectory planning tasks, this invention employs two ViT encoding layers for feature extraction, using the initial input sequence... The first encoding layer is used to obtain the encoding result. Then, the output encoding result of the first encoding layer is input into the second encoding layer, and the latent fusion feature is obtained from the final encoding result as follows:

[0068] (10)

[0069] in, These are the parameters of the default ViT encoder. This represents the preset ViT encoder. This invention employs two encoding layers, each with four attention heads, to control network complexity and online inference overhead while ensuring environmental representation capabilities.

[0070] S104. Based on the potential fusion feature representation, a hierarchical reinforcement learning model is used to generate the joint angle motion of the robotic arm. The hierarchical reinforcement learning model includes a low-level policy layer and a high-level policy layer connected in series. The low-level policy layer is used to obtain a first motion vector based on the potential fusion feature representation. The high-level policy layer is used to output the joint angle motion of the robotic arm based on the first motion vector and the robotic arm state vector. The first motion vector is the predicted position of the robotic arm end effector relative to the world coordinate system.

[0071] In one embodiment, both the high-level policy layer and the low-level policy layer are constructed using an Actor-Critic network. The Actor-Critic network includes an Actor network and a Critic network. The Actor network adopts a multilayer perceptron architecture and consists of an input layer, two hidden layers, and an output layer. The Actor network generates an action probability distribution based on input features and obtains the action to be executed at the current time by sampling. The Critic network is used to evaluate the action vector output by the Actor network and update the network parameters based on the evaluation value.

[0072] Specifically, this invention introduces a hierarchical architecture, dividing the strategy into a high-level strategy and a low-level strategy. The high-level strategy focuses on the global objective task. After receiving the state observation at each time step, the high-level strategy predicts the path to the intermediate target point to obtain the reference path for the end effector. The low-level strategy is responsible for the specific action execution based on this intermediate target point, i.e., selecting the joint angle at each step, thereby reducing the complexity of optimization and enhancing generalization performance. Furthermore, since the high-level strategy focuses on long-term goals while the low-level strategy focuses on short-term goals, both are portable. Through pre-training, data utilization efficiency in new scenarios can be improved and training time reduced.

[0073] Both the high-level and low-level policy layers are built based on the SAC algorithm, differing only in their inputs and outputs. An Actor-Critic network, or Actor-Critic network, is constructed using the SAC algorithm. This requires two action-value functions and one policy function. (Parameters are) Modeling. The policy function, i.e., the Actor network, adopts a multilayer perceptron architecture, consisting of an input layer, two hidden layers, and an output layer. The network output contains two parts: the action mean and the log standard deviation. The action to be executed at the current time is obtained by sampling through the reparameterization technique.

[0074] Specifically, in obtaining Potential fusion features of time Subsequently, the Actor strategy generates an action probability distribution based on this feature, and obtains the action to be executed at the current time by sampling. This process can be represented as:

[0075] (11)

[0076] in, express Momentary action output express Motion of the robotic arm in real time Obeying network parameters Definition, based on potential fusion features The policy probability distribution used as input conditions. Indicates the inclusion of network parameters The strategy function.

[0077] The actions generated by the high-level strategy and the low-level strategy are respectively and , This means predicting the position of the end effector relative to the world coordinate system. These are the angle commands for each joint of the generated robotic arm. The input states are as follows: and Each is defined as:

[0078] (12)

[0079] (13)

[0080] in, The current joint angle of the robotic arm. and The positions of the target and the end effector relative to the world coordinate system. This represents the shortest distance between the robotic arm and the obstacle.

[0081] Furthermore, the Critic network comprises two Q-networks with identical structures, each consisting of an input layer, two hidden layers, and an output layer. The network takes the state vector and action vector concatenated along the feature dimension as input, processes them through the two hidden layers, and outputs a scalar Q-value. The two Q-networks use different randomly initialized weights to provide diverse value estimates, effectively mitigating the overestimation problem caused by bootstrapping. To make training more stable, its parameters are not updated directly through gradient descent but instead employ a soft update mechanism, effectively suppressing drastic fluctuations in the value function during training.

[0082] Specifically, state transition data is stored in an experience pool, and Q-value estimates are obtained from the input state-action pairs using two Q-networks. The target value is calculated using the smaller value in the two Q-networks, and the Critic network parameters are updated by minimizing the mean squared error loss. The Q-value output by the Critic serves as the evaluation criterion for Actor policy optimization, driving the policy to adjust towards higher returns.

[0083] This invention first inputs the physical state of the robotic arm into a multilayer perceptron and maps it into state feature tokens. Simultaneously, the visually observed image is divided into several image blocks, linearly projected, and then superimposed with position codes to obtain a visual token sequence. Subsequently, the state tokens and visual tokens are input into a pre-defined ViT encoder. Cross-modal feature fusion is achieved through a multi-head self-attention mechanism, a feedforward network, layer normalization, and residual connections, ultimately extracting a potential fusion feature representation for subsequent decision-making. Based on this potential fusion representation, the Actor network outputs the action distribution and samples it to obtain the next control command, further improving the strategy's adaptability to virtual-real differences and environmental disturbances.

[0084] S105. After the robotic arm performs the current joint angle action, the strategy is updated through a composite reward function until the robotic arm end effector reaches the target position.

[0085] Specifically, a comprehensive performance index function for "energy consumption-efficiency-safety" is designed, mainly including dense rewards for obstacle avoidance, training steps, and position error, as well as sparse rewards for reaching the target range and collisions, namely:

[0086] (14)

[0087] in, , , , , These are the coefficients of each reward function; in one implementation case, , , , , ;

[0088] The obstacle avoidance reward function is a simulated artificial potential field. Considering the relative positions of the robotic arm and the obstacle, the repulsive force property ensures that within the obstacle's influence range, the closer the robotic arm is to the obstacle, the greater the penalty. The traditional repulsive force field formula is:

[0089] (15)

[0090] in, As a scale factor, The shortest distance between the robotic arm and the obstacle. Represents the radius of influence of the obstacle.

[0091] Therefore, the obstacle avoidance reward function is set according to the gradient of the repulsive field as follows:

[0092] (16)

[0093] In one implementation case... , .

[0094] For practical applications of robotic arms, energy optimization is often a key consideration, aiming to complete the same task with minimal energy consumption and thus reduce costs. In deep reinforcement learning training, reducing the number of steps required to complete the same task can also significantly reduce the energy consumption of the robotic arm. Therefore, designing a reward function related to the number of training steps is crucial. This allows the robotic arm to reach the target area in the fewest possible steps, as shown in the following formula:

[0095] (17)

[0096] in, This is the number of steps in the current round. The number of steps increases by one each time the robotic arm makes a movement adjustment. This represents the maximum number of moves per round. In one implementation case, If the robotic arm fails to complete its task in the current round, then The cumulative value for the entire round is 0; if the robotic arm completes the task in fewer steps, then... A large cumulative value throughout the round prompts the robotic arm to learn strategies that allow it to complete the task in fewer steps.

[0097] Design a reward function that reflects the current progress of the robotic arm's task. Defined as:

[0098] (18)

[0099] in, It can provide the progress status of the robotic arm throughout the training process, making the rewards more frequent and beneficial to the robotic arm's strategy learning.

[0100] In addition to the above-mentioned intensive rewards, there are also rewards for reaching the target range. Sparse rewards for collisions .

[0101] In one embodiment, the invention further includes:

[0102] Determine whether the training results satisfy the condition that the average composite reward function change within a preset round is less than a preset threshold.

[0103] If so, then stop training;

[0104] If not, continue to determine whether the current end effector of the robotic arm has reached the target position and whether the robotic arm has collided with the obstacle. If yes, proceed to the next control cycle step of sampling a point in the target position area as the target position of the robotic arm in the current training round.

[0105] Otherwise, repeat the steps of obtaining the current state of the robotic arm and the depth information detected by the camera, and fusing the robotic arm state and visual data through a preset ViT encoder to obtain a potential fused feature representation.

[0106] Specifically, the composite reward function is obtained through the above formula. After the robotic arm performs the current joint angle action and updates the strategy according to the obtained composite reward function, it first determines whether the training result satisfies the condition that the average reward function change fluctuation of the last 100 rounds is less than 5%. If so, training stops; otherwise, it continues to the next step: determining whether the current end of the robotic arm has reached the target position and whether the robotic arm has collided with the obstacle. If so, it enters the next control cycle, that is, after performing S102, it repeats S103, S104, and S105; otherwise, it continues to repeat S103, S104, and S105.

[0107] This invention collects robotic arm state data and camera-detected depth data, and fuses the robotic arm state and visual data using an improved ViT encoder. An attention mechanism is used to assign weights to the observed states and ignore redundant information. Secondly, a hierarchical reinforcement learning framework is employed to generate paths, reducing training difficulty. A reward function is constructed using "energy consumption-time-safety" as a performance indicator to update and adjust network parameters, achieving an approximate solution for the optimal path. This invention fully considers approximation safety in dense and unstructured environments, improves planning efficiency and real-time performance, meets the needs of online autonomous planning, and overcomes the limitations of traditional path planning methods that can only solve convex optimization problems. It also overcomes the drawbacks of high complexity and poor real-time performance in solving multi-constraint optimization problems, making it suitable for tasks such as shaft-hole assembly and dense obstacle avoidance in unstructured environments.

[0108] This invention also provides an electronic device, which includes a processor and a memory. The memory stores at least one instruction or at least one program, which is loaded and executed by the processor. The method described above provides a robotic arm path planning method based on attention mechanism and hierarchical reinforcement learning.

[0109] Furthermore, the electronic device may participate in or include the apparatus or system provided in the embodiments of the present invention. The electronic device may include one or more processors (processors may include, but are not limited to, processing devices such as microprocessors (MCUs) or programmable logic devices (FPGAs), memory for storing data, and transmission devices for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the I / O interface), a network interface, a power supply, and / or a camera.

[0110] It should be noted that the aforementioned one or more processors and / or other data processing circuits are generally referred to herein as "data processing circuits". These data processing circuits may be embodied, in whole or in part, in software, hardware, firmware, or any other combination thereof. Furthermore, the data processing circuit may be a single, independent processing module, or may be integrated, in whole or in part, into any other element within the device (or mobile device). As involved in the embodiments of the present invention, the data processing circuit serves as a processor control mechanism (e.g., selection of a variable resistor termination path connected to an interface).

[0111] The memory can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the method described in the embodiments of the present invention. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, thereby realizing the above-mentioned data processing method. The memory may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include memory remotely located relative to the processor, and these remote memories can be connected to electronic devices via a network. Examples of the above-mentioned networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0112] The transmission device is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the device's communication provider. In one example, the transmission device includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device may be a Radio Frequency (RF) module, used for wireless communication with the Internet.

[0113] The display can be, for example, a touchscreen liquid crystal display (LCD), which allows users to interact with the user interface of an electronic device (or mobile device).

[0114] This invention also provides a computer storage medium storing at least one instruction or at least one program, which is loaded and executed by a processor to implement the robotic arm path planning method based on attention mechanism and hierarchical reinforcement learning provided in the above-described method embodiments.

[0115] Optionally, in this embodiment, the aforementioned computer storage medium may be located at at least one of the multiple network servers in a computer network. Optionally, in this embodiment, the aforementioned storage medium may include, but is not limited to, various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0116] This invention also provides a computer program product or computer program, which includes computer instructions stored in a computer storage medium. The processor of an electronic device reads the computer instructions from the computer storage medium and executes the computer instructions, causing the electronic device to perform the robotic arm path planning method based on attention mechanism and hierarchical reinforcement learning provided in the above-described method embodiments.

[0117] It should be noted that the order of the above embodiments of the present invention is merely for descriptive purposes and does not represent the superiority or inferiority of the embodiments. Furthermore, specific embodiments have been described above. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps described in the claims can be performed in a different order than that shown in the embodiments and still achieve the desired result. Additionally, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0118] It should be understood that the above description of the preferred embodiments is quite detailed, but it should not be considered as a limitation on the scope of protection of this invention. Those skilled in the art, under the guidance of this invention, can make substitutions or modifications without departing from the scope of protection of the claims of this invention, and all such substitutions or modifications fall within the scope of protection of this invention. The scope of protection of this invention should be determined by the appended claims.

Claims

1. A robotic arm path planning method based on attention mechanism and hierarchical reinforcement learning, characterized in that, The method includes: Collect the location information of obstacles, and determine the target location area for the robotic arm path planning task based on the provided location information; Determine the target position of the robotic arm in the current training round within the target position region; The current state vector of the robotic arm and the visual data detected by the camera are obtained, and the robotic arm state and visual data are fused through a preset ViT encoder to obtain a potential fused feature representation. Based on the potential fusion feature representation, a hierarchical reinforcement learning model is used to generate the joint angle motion of the robotic arm. The hierarchical reinforcement learning model includes a low-level policy layer and a high-level policy layer connected in series. The low-level policy layer is used to obtain a first motion vector based on the potential fusion feature representation. The high-level policy layer is used to output the joint angle motion of the robotic arm based on the first motion vector and the robotic arm state vector. The first motion vector is the predicted position of the robotic arm end effector relative to the world coordinate system. After the robotic arm performs the current joint angle action, the strategy is updated through a composite reward function until the robotic arm end effector reaches the target position.

2. The robotic arm path planning method based on attention mechanism and hierarchical reinforcement learning according to claim 1, characterized in that, Determining the target location area for the robotic arm path planning task based on the provided location information includes: The obstacle and the robotic arm are enveloped by a group of line segments to approximate their shape. The shortest distance between the robotic arm and the obstacle is obtained by solving the minimum geometric spacing between the envelopes. The workspace of the robotic arm is determined based on the shortest distance between the robotic arm and the obstacle. Within the workspace of the robotic arm, the target location area is set according to the path planning task.

3. The robotic arm path planning method based on attention mechanism and hierarchical reinforcement learning according to claim 1, characterized in that, The process of fusing robotic arm state and visual data using a preset ViT encoder to obtain potential fusion feature representations includes: The visual data detected by the camera is divided into several non-overlapping image blocks according to a fixed size, and each image block is represented by a vector to obtain the image block embedding sequence; The state vector of the robotic arm is processed by a state encoder to obtain a state encoded vector. The state encoder is used to project the state vector of the robotic arm onto a feature space that is the same as the image features. The state vector of the robotic arm includes the joint angle of the robotic arm at any given time, the position of the target and the end effector relative to the world coordinate system, and the shortest distance between the robotic arm and the obstacle. The initial input sequence is obtained by concatenating the state encoding vector, the image block vector sequence, and the position encoding matrix in sequence. The initial input sequence is input into a preset ViT encoder for feature extraction, and a potential fusion feature representation is output. The preset ViT encoder models the global correlation between different image patches through a multi-head self-attention mechanism.

4. The robotic arm path planning method based on attention mechanism and hierarchical reinforcement learning according to claim 3, characterized in that, The preset ViT encoder includes two identical coding layers connected in series, with multiple attention heads set in parallel for each layer. The process of inputting the initial input sequence into the preset ViT encoder for feature extraction and outputting a latent fusion feature representation includes: The initial input sequence is input into the first coding layer to obtain the coding result, and then the output coding result of the first coding layer is input into the second coding layer to obtain the latent fusion feature representation.

5. The robotic arm path planning method based on attention mechanism and hierarchical reinforcement learning according to claim 4, characterized in that, The execution process of each coding layer is shown in the following formula: ； in, Presentation layer normalization operation, This represents the mapping function of a multilayer perceptron. This indicates a fully connected mapping. This represents the function for the multi-head self-attention mechanism. Indicates the first The input of each coding layer, Indicates the first The output of the attention sublayer of the coding layer Indicates the first The final output of each coding layer Indicates the number of coding layers, and .

6. The robotic arm path planning method based on attention mechanism and hierarchical reinforcement learning according to claim 5, characterized in that, The execution steps of the multi-head self-attention mechanism function include: The initial input sequence is transformed linearly to obtain the query vector, key vector, and value vector. Based on the query vector, key vector, and value vector, the output of each attention head is calculated using a scaled dot product attention method. The outputs of each attention head are concatenated and then linear projection is performed to obtain the output of the multi-head self-attention mechanism.

7. The robotic arm path planning method based on attention mechanism and hierarchical reinforcement learning according to claim 1, characterized in that, Both the high-level and low-level policy layers are constructed using an Actor-Critic network. The Actor-Critic network includes an Actor network and a Critic network. The Actor network adopts a multilayer perceptron architecture, consisting of an input layer, two hidden layers, and an output layer. The Actor network generates an action probability distribution based on input features and obtains the action to be executed at the current time by sampling. The Critic network is used to evaluate the action vector output by the Actor network and update the network parameters based on the evaluation value.

8. The robotic arm path planning method based on attention mechanism and hierarchical reinforcement learning according to claim 7, characterized in that, The Critic network comprises two Q-networks, and the two Q-networks use different randomly initialized weights. The method further includes: The Bellman objective is calculated based on the smaller scalar Q value in the two Q-networks, and the objective scalar Q value is obtained. The Critic network parameters are updated by minimizing the mean square error loss for the target scalar Q value.

9. The robotic arm path planning method based on attention mechanism and hierarchical reinforcement learning according to claim 1, characterized in that, The composite reward function is obtained by weighted summation of the obstacle avoidance reward function, the training step reward function, the dense reward of the position error reward, and the sparse reward of reaching the target range and collision.

10. The robotic arm path planning method based on attention mechanism and hierarchical reinforcement learning according to claim 1, characterized in that, The method further includes: Determine whether the training results satisfy the condition that the average composite reward function change within a preset round is less than a preset threshold. If so, then stop training; If not, continue to determine whether the current end effector of the robotic arm has reached the target position and whether the robotic arm has collided with an obstacle. If yes, proceed to the next control cycle step of determining the target position of the robotic arm in the target position area of ​​the current training round. Otherwise, repeat the steps of obtaining the current state of the robotic arm and the depth information detected by the camera, and fusing the robotic arm state and visual data through a preset ViT encoder to obtain a potential fused feature representation.

Images