A method and apparatus for target tracking in electric robots based on a self-attention mechanism

Abstract

This invention discloses a target tracking method and device for electric robots based on a self-attention mechanism. First, a ViT model is used for feature extraction. Features from the template image and the search image are input into a Transformer for multi-head self-attention modeling to capture the correlation between the target and the background. Second, inter-frame features are extracted using ResNet, and LSTM is combined to model the inter-frame temporal relationship, dynamically updating the target trajectory. The features output by LSTM are fused with the target trajectory information for target position prediction. Finally, the decoder completes target classification and position regression through multiple branches. This method effectively integrates spatiotemporal information, solving the problem of tracking accuracy and stability of electric robots in dynamic environments, and has strong practical value and application potential. By combining temporal information, trajectory information, and a dynamic feature update mechanism, this method improves the accuracy and robustness of target tracking.

Description

Technical Field

[0001] This invention belongs to the field of computer vision technology, specifically relating to a method and apparatus for target tracking of electric robots based on a self-attention mechanism. Background Technology

[0002] While significant progress has been made in the research and application of single-target tracking algorithms, many challenges remain when applied to the field of power robots. Power robots, as crucial tools for power grid operation and maintenance, need to perform high-precision, high-efficiency tasks in complex and ever-changing environments, such as fault inspection and equipment testing. These tasks require robots to continuously and stably track specific targets, such as conductors, substation equipment, and fault points. However, most current single-target tracking algorithms cannot fully meet the needs of power robots in practical applications. Traditional convolutional neural network (CNN)-based tracking algorithms, while achieving single-target tracking to some extent, suffer from low tracking accuracy and poor real-time performance when dealing with complex environmental changes. While Transformer-based tracking methods, which have emerged in recent years, have made progress in some aspects, they still face limitations in power robot applications.

[0003] Research on single-object tracking algorithms can be broadly categorized into two types: traditional CNN-based methods and modern Transformer-based methods. Traditional single-object tracking algorithms mostly employ CNNs for feature extraction of both the target and background, then track the target through a specific matching mechanism. These methods typically use a Siamese network structure, where two identical CNN branches extract features from the target template and the search region respectively, and then perform matching through simple correlation operations, such as SiamFC and SiamRPN. These methods were widely used in the tracking field due to their simple structure and high speed. However, due to their local convolutional characteristics, CNNs struggle to capture the global correlation between the target and the background, and are prone to tracking instability or target loss in scenarios with changing lighting, occlusion, and deformation. Furthermore, CNN algorithms are limited in real-time performance, making it difficult to meet the high-efficiency requirements of electric robots.

[0004] In recent years, the Transformer, a deep learning model that has achieved success in natural language processing, has been introduced into computer vision tasks to capture long-range dependencies in images and videos. Transformer-based tracking algorithms have gradually become a research hotspot, mainly divided into two architectures: CNN-Transformer hybrid methods and methods based entirely on Transformer. The CNN-Transformer method improves tracking robustness by combining the local feature extraction capabilities of CNNs with the global feature capture capabilities of Transformers. For example, STARK (Siamese Transformer Architecture for Real-time Visual Tracking) introduces a two-stage Transformer architecture on top of CNNs to capture the global relationship between the target and the background, thereby enhancing the feature fusion effect in complex backgrounds. The HiFT (Hierarchical Feature Transformer) algorithm utilizes multi-layer convolutional features and performs layer-by-layer feature fusion through the Transformer's self-attention mechanism, effectively improving tracking accuracy. These methods leverage the Transformer's powerful global information capture capabilities, enabling them to cope with complex tracking scenarios to a certain extent. However, the dual-stream, dual-stage structure of such methods introduces significant computational overhead, which limits their application, especially in real-time-critical electric robot tasks.

[0005] While single-stream, single-stage algorithms based entirely on Transformers perform well in terms of tracking speed and accuracy, their practical application in power robots remains limited. The main problem is that most existing algorithms treat single-target tracking as a template matching problem, neglecting the effective use of temporal information. Most algorithms focus only on the best match within each frame, lacking the mining and integration of inter-frame temporal information. This leads to tracking instability or target loss under complex conditions such as prolonged occlusion or changes in lighting. Furthermore, Transformer algorithms typically require large-scale training data, which is difficult to achieve in power robot scenarios, and data scarcity limits the algorithm's generalization ability.

[0006] To address the tracking requirements of power robots, an improved single-stream, single-stage Transformer architecture is proposed. This method introduces a temporal information module, integrating tracking results from previous frames with features from the current frame to enhance tracking robustness and accuracy. Specifically, the algorithm adds a temporal feature extraction layer during the encoding stage, adjusting the target position estimate in real-time by comparing it with features from the current frame, and employing an adaptive mechanism to handle occlusion and illumination changes. Simultaneously, the algorithm utilizes few-shot learning and self-supervised techniques to improve its generalization ability in real-world power scenarios.

[0007] In summary, existing single-target tracking algorithms have several limitations in the application of electric robots, mainly in terms of structural complexity, insufficient real-time performance, and lack of utilization of temporal information. While the two-stream, two-stage Transformer architecture has advantages in feature fusion, its high computational cost makes it unsuitable for electric robots. Although a single-stream, single-stage architecture based entirely on Transformer improves tracking speed, it still fails to fully utilize temporal information. To address this issue, this study proposes a single-stream, single-stage Transformer architecture integrating temporal features, aiming to achieve more efficient and accurate single-target tracking in electric robot applications. Summary of the Invention

[0008] The purpose of this invention is to overcome the shortcomings of the existing technology by providing a method and apparatus for target tracking of electric robots based on a self-attention mechanism.

[0009] The objective of this invention can be achieved through the following technical solutions:

[0010] This invention provides a target tracking method for electric robots based on a self-attention mechanism, comprising the following steps:

[0011] Step S1: Obtain the template image by acquiring the current search image in real time through the electric robot's camera;

[0012] Step S2: Initialize inter-frame information and trajectory information;

[0013] Step S3: Process the template image and the current search image to obtain a joint feature vector. Input the joint feature vector into the pre-trained ViT model encoder for processing and output the joint feature matrix of the template image and the current search image.

[0014] Step S4: Determine whether the current search image is the first frame. If it is not the first frame, proceed to step S5. If it is the first frame, proceed to step S6.

[0015] Step S5: Using the current search image and the previous search image, model the inter-frame information through the ResNet network and LSTM network to obtain the inter-frame feature vector, and update the inter-frame information based on the inter-frame feature vector;

[0016] Step S6: Use the joint feature matrix as the query vector of the pre-trained ViT model decoder, and concatenate the inter-frame information and trajectory information as the key-value vector of the pre-trained ViT model decoder. The decoder outputs the decoded feature matrix. The precise position and pose estimation of the target in the current search image are obtained through the decoded feature matrix. The pose adjustment of the electric robot is performed based on the precise position and pose estimation of the target in the current search image.

[0017] Step S7: Input the decoded feature matrix into the pre-trained ViT model regression network, output the bounding box information of the target in the current search image, and update the trajectory information based on the bounding box information of the target in the current search image.

[0018] Furthermore, the initialization of inter-frame information includes the following steps: modeling the information between the template image and the current search image using a ResNet network and an LSTM network to obtain the feature vector between the template image and the current search image, and using the feature vector between the template image and the current search image as inter-frame information.

[0019] Furthermore, the initialization of trajectory information includes the following steps: obtaining the bounding box information of the target in the template image, copying the bounding box information of the target in the template image n times, and using the copied n bounding box information as trajectory information, where n is a constant.

[0020] Furthermore, the process of combining the template image with the current search image to obtain a joint feature vector includes the following steps:

[0021] The template image and the current search image are divided into multiple image blocks. Each image block is flattened into a vector and converted into an embedding vector through a linear transformation to obtain template features and search features. Position encoding is added to each feature, and the template features and search features are concatenated into a joint feature vector.

[0022] Furthermore, when the template image and the current search image undergo joint feature vector processing, they employ the same image segmentation method and positional encoding strategy.

[0023] Furthermore, the step of using the current search image and the previous search image to model inter-frame information through a ResNet network and an LSTM network to obtain inter-frame feature vectors specifically includes the following steps:

[0024] The current search image and the previous search image are input into a pre-trained ResNet network for feature extraction, and a fused feature map is output. The fused feature map is flattened into a one-dimensional vector, and the one-dimensional vector is used as a time series input into a pre-trained LSTM network to output an inter-frame feature vector.

[0025] Furthermore, the ViT model encoder includes multiple Transformer encoder layers, each of which includes a self-attention mechanism and a feedforward neural network for extracting global features of the input image and generating a joint feature matrix.

[0026] Furthermore, the trajectory information is a target bounding box information queue of size n.

[0027] Furthermore, updating the trajectory information based on the bounding box information of the target in the current search image includes: dequeuing the bounding box information at the head of the target bounding box information queue and enqueuing the bounding box information of the target in the current search image.

[0028] In another aspect, the present invention provides a computer storage medium storing executable program code; the executable program code is used to execute a target tracking method for an electric robot based on a self-attention mechanism as described above.

[0029] Compared with the prior art, the present invention has the following advantages:

[0030] (1) To achieve accurate target tracking for electric robots, this invention constructs a model structure centered on a single-stream, single-stage spatiotemporal Transformer, utilizing inter-frame information and target trajectory information in real time during the tracking process. This model effectively integrates historical features and temporal information of the target by combining encoder and decoder structures, thereby enhancing the model's stability in dynamic scenarios. Through reasonable spatiotemporal feature integration, this invention achieves the first application of the spatiotemporal Transformer in target tracking for electric robots, providing an innovative solution to overcome the bottlenecks of traditional algorithms in real-time performance and accuracy, and demonstrating excellent performance in complex environments.

[0031] (2) This invention makes a series of improvements to the spatiotemporal Transformer to better adapt to the complex needs of power scenarios. Specifically, these improvements include: introducing a dynamic feature update mechanism to optimize tracking accuracy through the target's historical trajectory information; employing a candidate elimination module to reduce the computational burden on the model and improve real-time tracking speed; and introducing a combination of ResNet and LSTM in the inter-frame information module to achieve relationship modeling of consecutive frames. These improvements effectively enhance the robustness of the algorithm, enabling it to remain stable under complex conditions such as occlusion, changes in illumination, and rapid target movement.

[0032] (3) For different power scenarios, this invention has built and verified application scenarios in various environments. By setting up typical complex power environments (such as changes in lighting, occlusion obstacles, etc.), the adaptability and versatility of the algorithm are verified, and the target tracking performance of the spatiotemporal Transformer in different situations is evaluated. This process demonstrates the applicability and advantages of this invention in various complex power environments, providing strong support for the practical application of power robots.

[0033] (4) To address the long-term tracking requirements of electric robot tasks, this invention employs a dynamic feature update mechanism. This mechanism updates the target representation in real time based on historical frame trajectory information and the prediction results of the latest frame, enabling the electric robot to maintain stable tracking in complex scenarios. By introducing a target trajectory feature update strategy, the robot can adaptively adjust its tracking strategy when dealing with fast-moving targets or nonlinear paths, effectively improving the accuracy and reliability of target localization.

[0034] (5) This invention extracts global features from a template image and a search image by inputting them into a pre-trained ViT (Visual Transformer) model encoder and combining this with a self-attention mechanism, generating a joint feature matrix. This technique can effectively handle complex background and target features in images and maintain high-precision target tracking performance even when the target position changes.

[0035] (6) This invention utilizes ResNet and LSTM networks to model inter-frame information, combining the features of the current frame and the previous frame to generate inter-frame feature vectors for continuous target tracking. This inter-frame feature update method can effectively capture temporal information and improve the accuracy of target tracking, especially when the target is moving rapidly or changing significantly, it can maintain stable tracking performance.

[0036] (7) This invention introduces the ViT model regression network to output the bounding box information of the target and update the trajectory information, which can reflect the boundary changes of the target in real time and ensure that the electric robot can continuously adjust and optimize its motion trajectory during the tracking process. This bounding box information update mechanism effectively improves the robustness of target tracking and has strong adaptability.

[0037] (8) This invention converts the template image and the search image into a unified embedding space through image segmentation and positional encoding, and generates a joint feature vector by combining global features. This feature processing method can improve computational efficiency while maintaining adaptability to complex scenes, especially in the case of multiple targets or rapidly changing targets, it can still accurately track targets.

[0038] (9) By using the temporal modeling of inter-frame features and the dynamic updating of the target bounding box, combined with the input of LSTM network and trajectory information, this invention can cope with complex situations such as target occlusion, background changes and viewpoint changes, thereby ensuring the stable tracking capability of the electric robot in complex environments.

[0039] (10) This invention integrates a ResNet network for feature extraction and an LSTM network for temporal modeling, enabling full transmission and utilization of inter-frame information, thereby improving the long-term stability and adaptability of target tracking. Especially when the target spans a long time interval, this invention can effectively avoid target tracking loss. Attached Figure Description

[0040] Figure 1 This is a flowchart of the method of the present invention. Detailed Implementation

[0041] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0042] Example 1:

[0043] This embodiment provides a Transformer-based target tracking method for electric robots, such as... Figure 1 As shown, it includes the following steps:

[0044] Step S1: Obtain the template image by acquiring the current search image in real time through the electric robot's camera;

[0045] Step S2: Initialize inter-frame information and trajectory information;

[0046] Step S3: Process the template image and the current search image to obtain a joint feature vector. Input the joint feature vector into the pre-trained ViT model encoder for processing and output the joint feature matrix of the template image and the current search image.

[0047] Step S4: Determine whether the current search image is the first frame. If it is not the first frame, proceed to step S5. If it is the first frame, proceed to step S6.

[0048] Step S5: Using the current search image and the previous search image, model the inter-frame information through the ResNet network and LSTM network to obtain the inter-frame feature vector, and update the inter-frame information based on the inter-frame feature vector;

[0049] Step S6: Use the joint feature matrix as the query vector of the pre-trained ViT model decoder, and concatenate the inter-frame information and trajectory information as the key-value vector of the pre-trained ViT model decoder. The decoder outputs the decoded feature matrix. The precise position and pose estimation of the target in the current search image are obtained through the decoded feature matrix. The pose adjustment of the electric robot is performed based on the precise position and pose estimation of the target in the current search image.

[0050] Step S7: Input the decoded feature matrix into the pre-trained ViT model regression network, output the bounding box information of the target in the current search image, and update the trajectory information based on the bounding box information of the target in the current search image.

[0051] The initialization of inter-frame information includes the following steps: the template image and the current search image are modeled using a ResNet network and an LSTM network to obtain the feature vector between the template image and the current search image, and the feature vector between the template image and the current search image is used as the inter-frame information.

[0052] The initialization of trajectory information includes the following steps: obtaining the bounding box information of the target in the template image, copying the bounding box information of the target in the template image n times, and using the n copied bounding box information as trajectory information, where n is a constant.

[0053] The process of combining the template image and the current search image to obtain a joint feature vector includes the following steps:

[0054] The template image and the current search image are divided into multiple image blocks. Each image block is flattened into a vector and converted into an embedding vector through a linear transformation to obtain template features and search features. Position encoding is added to each feature, and the template features and search features are concatenated into a joint feature vector.

[0055] When the template image and the current search image are processed together for joint feature vectors, the same image segmentation method and position encoding strategy are used.

[0056] This involves using the current search image and the previous search image to model inter-frame information through a ResNet network and an LSTM network, obtaining inter-frame feature vectors. The specific steps include:

[0057] The current search image and the previous search image are input into a pre-trained ResNet network for feature extraction, and a fused feature map is output. The fused feature map is flattened into a one-dimensional vector, and the one-dimensional vector is used as a time series input into a pre-trained LSTM network to output an inter-frame feature vector.

[0058] The ViT model encoder consists of multiple Transformer encoder layers, each of which includes a self-attention mechanism and a feedforward neural network to extract global features from the input image and generate a joint feature matrix.

[0059] The trajectory information is a target bounding box information queue of size n.

[0060] The process of updating trajectory information based on the bounding box information of the target in the current search image includes: dequeuing the bounding box information at the head of the target bounding box information queue and enqueuing the bounding box information of the target in the current search image.

[0061] Example 2:

[0062] The parts not mentioned in this embodiment are the same as in Embodiment 1.

[0063] This embodiment delves into the challenges faced by electric robots in single-target tracking within complex power environments. Due to the complex and ever-changing nature of power scenarios, traditional tracking algorithms often struggle to maintain tracking stability under conditions such as prolonged occlusion and changes in lighting, making it difficult to balance real-time performance and accuracy. To address this issue, this embodiment proposes an improved algorithm based on a single-stream, single-stage spatiotemporal Transformer, leveraging temporal information to enhance tracking robustness and accuracy. By introducing a temporal feature extraction layer, information from previous frames is integrated with features from the current frame, improving the algorithm's adaptability to occlusion and environmental changes, potentially overcoming the bottlenecks of traditional methods in electric robot target tracking.

[0064] This embodiment proposes a spatiotemporal Transformer-based single-target tracking algorithm for electric robots, aiming to solve the challenge of single-target tracking in complex power environments, particularly addressing robustness and accuracy issues under conditions of occlusion, changing lighting, and rapid target movement. The algorithm combines temporal information, trajectory information, and a dynamic feature update mechanism to improve tracking accuracy and stability.

[0065] During the initialization phase, the algorithm uses the Frame Token from the inter-frame information fusion module and the Trajectory Token from the trajectory information module to initialize the temporal information. The Frame Token models the inter-frame relationship by extracting features from the template frame and the first frame of the search area, and is then initialized. The Trajectory Token establishes a preliminary target trajectory by copying the bounding boxes of the template frame n times, and improves tracking accuracy and stability by updating historical trajectory information in real time during subsequent tracking.

[0066] For feature extraction, a pre-trained ViT (Visual Transformer) model was used, which includes a 12-layer encoder. The input image is divided into multiple patches, and each patch undergoes an embedding operation to obtain an embedding vector. These vectors are then fed into the Transformer for processing. Specifically, the image is encoded by flattening each patch into a one-dimensional vector, assuming the input is... Where P is the size of each block, and C is the number of channels per block. To preserve the spatial location information of the image, a positional encoding PE is added to the embedding vector of each block. i The feature vector after position embedding is obtained. The embedding vectors of the template image and the search image are then concatenated into a joint feature sequence. in and These represent the embedding vectors of the template image and the search image, respectively. The concatenated feature matrix... Where N is the sequence length and D is the dimension of the embedding vector.

[0067] Next, the Transformer models the input features using a multi-head self-attention mechanism. Given a query vector... key vector Sum value vector Attention is calculated as follows:

[0068]

[0069] Where, d k The dimension of the key vector is used for scaling to ensure numerical stability. The softmax operation is used to normalize the attention weights, thereby calculating a weighted average of the output features. This self-attention mechanism can capture the correlation between the target and the background in the image, enhance the ability to extract target features, and maintain the stability of target information.

[0070] In the inter-frame information fusion stage, a framework combining ResNet and LSTM is used to model the relationships between consecutive frames. The ResNet network is responsible for extracting and fusing features from the current frame and the previous frame. Assume the extracted and fused features are F. t Next, these features are fed into an LSTM network for sequence modeling. LSTM can effectively capture temporal information between frames. The computation of LSTM involves the computation of four gates: input gate i... t Forgotten Gate t Output gate o t and candidate unit status The calculation formulas are as follows:

[0071] i t =σ(W i [h t-1 ,F t ]+b i )

[0072] f t =σ(W f [h t-1 ,F t ]+b f )

[0073] o t =σ(W o [h t-1 ,F t ]+b o )

[0074]

[0075] h t =o t ·tanh(C t )

[0076] Among them, i t f t o t It is the activation function of the input gate, forget gate, and output gate, F. t It is a feature of the current frame, C t and h t These are the cell state and the hidden state, respectively. Through these calculations, LSTM can effectively capture the temporal information between frames, and the output state h t It contains relevant information about historical frames and ultimately generates a Frame Token.

[0077] This embodiment introduces an innovative dynamic feature update strategy. Besides using LSTM to encode the target's historical information and update the target's representation in real time, it also incorporates the target's trajectory information from the previous five frames. This allows the model to optimize tracking results by analyzing the target's motion trends, rather than relying solely on single-frame images. This strategy significantly enhances the model's performance when handling occlusion and rapid target movement.

[0078] The main process is as follows:

[0079] I) Initialize the first n frames. The initialization method proposed in this embodiment is relatively simple, which is to copy the anno of the template frame n times.

[0080] II) Coordinate updates are implemented during the tracking process. When the model predicts the coordinates of the current frame each time, it saves the coordinates to the bbox memory bank and maps them to a Trajectory Token.

[0081] III) In this embodiment, after multiple experiments, n is set to 5, that is, the length of the bbox memory bank is always 5.

[0082] This embodiment uses an encoder-decoder structure, and the two inputs of the decoder are:

[0083] I) Query vector q: Composed of features x output by the encoder, representing target feature information in the current frame, containing high-level abstract information of the input data.

[0084] II) Key-value vectors k and v: kv = {kv1, kv2, ..., kv} n The data contains temporal information from the previous frames, including inter-frame features and target trajectory information, which is used to enhance the tracking stability and continuity of the current frame.

[0085] During decoding, the decoder uses a multi-head self-attention mechanism to capture the association between q and kv. Specifically, the query vector q, key vector k, and value vector v are linearly mapped to different subspace representations, and the attention is represented as follows:

[0086]

[0087] Here, the query vector q represents the target features of the current frame, while the key vectors k and v represent the features and target trajectory information of historical frames. The decoder uses this temporal information to adjust the prediction of the target position, thereby achieving accurate target localization. k is the dimension of the key vector, used for scaling, and the softmax operation ensures the normalization of the weights. Through this mechanism, the decoder can dynamically adjust the target position estimate using temporal information, improving robustness to complex situations such as occlusion and illumination changes. Finally, the output of the multi-head attention layer is passed through a feedforward network to generate the final output of the decoder, which is the precise position and pose estimate of the target in the current frame. The output feature matrix can be represented as:

[0088]

[0089] This output is used for further target classification and position regression tasks, ensuring real-time, high-precision single-target tracking in electric robot tracking applications.

[0090] Finally, the decoded feature matrix output by the decoder is processed through three independent branches in the regression network.

[0091] I) Central Prediction Branch: Responsible for predicting the center location of the target;

[0092] II) Size regression branch: responsible for predicting the size, width, and height of the target;

[0093] III) Offset Regression Branch: Responsible for predicting the offset of the target position.

[0094] The location of the maximum response (i.e., the center coordinates) is selected based on score_map_ctr. Then, the size and offset of the target are obtained from size_map and offset_map, and finally, a precise predicted bounding box for the target's location in the current frame is generated. The trajectory information is updated based on the generated precise predicted bounding box for the target's location in the current frame.

[0095] In power robot applications, due to the complexity and dynamism of the environment, single-target tracking algorithms need to be able to handle occlusion, changes in lighting, and prediction of the target's movement path. This invention, based on the Transformer's single-flow, single-stage model, introduces temporal and trajectory information, enabling power robots to efficiently and accurately identify and track specific power facilities or fault points, thereby achieving intelligent and automated inspection operations.

[0096] This invention employs a spatiotemporal Transformer model for target tracking in power robots. First, temporal information, including inter-frame and trajectory information, is initialized during model initialization to ensure stable tracking in the initial state. During tracking, the model fuses template features with search features to capture the correlation between the current frame and historical frames, thereby enhancing target localization capabilities. Through this model, target tracking in complex power environments is significantly improved, enhancing not only real-time performance but also robustness under occlusion conditions.

[0097] In this invention, through inter-frame information and trajectory information modules, the power robot can adjust the target's position estimation in real time to adapt to dynamic changes in different power scenarios. Inter-frame information is captured by introducing continuous inter-frame relationship modeling and using ResNet and LSTM network structures to continuously track the target, while trajectory information records the target's position changes in previous frames, helping the model determine the target's motion trend. In scenarios with occlusion or changing lighting, this strategy effectively reduces the risk of tracking loss.

[0098] To address the long-term tracking requirements of electric robot tasks, this invention employs a dynamic feature update mechanism. This mechanism updates the target representation in real time based on historical frame trajectory information and the prediction results of the latest frame, enabling the electric robot to maintain stable tracking in complex scenarios. By introducing a target trajectory feature update strategy, the robot can adaptively adjust its tracking strategy when dealing with fast-moving targets or non-linear paths, effectively improving the accuracy and reliability of target localization.

[0099] The decoder receives target feature information from the encoder for the current frame, along with a key-value vector containing temporal information from historical frames. A multi-head self-attention mechanism calculates the correlation between current frame features and historical trajectories, dynamically adjusting the target's position prediction. This method enables the decoder to accurately estimate the target's position and orientation in complex power environments, facilitating efficient target tracking for power robots under dynamic conditions. The target position and orientation information output by the decoder can also be used for subsequent classification and regression tasks, further enhancing the stability and intelligence of power robots in inspection tasks.

[0100] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0101] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A power robot target tracking method based on a self-attention mechanism, characterized in that, The method comprises the following steps: Step S1: acquiring a template image, acquiring a current search image in real time through a power robot camera; Step S2: initializing inter-frame information and initializing track information; Step S3: processing the template image and the current search image to obtain a joint feature vector, inputting the joint feature vector into a pre-trained ViT model encoder for processing, and outputting a joint feature matrix of the template image and the current search image; Step S4: determining whether the current search image is the first frame, if not, executing step S5, and if yes, executing step S6; Step S5: modeling inter-frame information by using the current search image and the previous search image through a ResNet network and an LSTM network to obtain an inter-frame feature vector, and updating the inter-frame information according to the inter-frame feature vector; Step S6: taking the joint feature matrix as a query vector of a pre-trained ViT model decoder, taking the inter-frame information and the track information as key-value vectors of the pre-trained ViT model decoder, and outputting a decoding feature matrix by the decoder, so as to obtain accurate position and pose estimation of a target in the current search image, and to adjust the pose of the power robot according to the accurate position and pose estimation of the target in the current search image; Step S7: inputting the decoding feature matrix into a pre-trained ViT model regression network to output bounding box information of the target in the current search image, and updating the track information according to the bounding box information of the target in the current search image; The modeling of the inter-frame information by using the current search image and the previous search image through the ResNet network and the LSTM network to obtain the inter-frame feature vector comprises the following steps: inputting the current search image and the previous search image into a pre-trained ResNet network to extract features, outputting a fusion feature map, flattening the fusion feature map into a one-dimensional vector, inputting the one-dimensional vector as a time sequence into a pre-trained LSTM network, and outputting the inter-frame feature vector.

2. The power robot target tracking method based on self-attention mechanism according to claim 1, characterized in that, The initialization of the inter-frame information comprises the following steps: modeling information of the template image and the current search image through a ResNet network and an LSTM network to obtain a feature vector between the template image and the current search image, and taking the feature vector between the template image and the current search image as the inter-frame information.

3. The power robot target tracking method based on self-attention mechanism according to claim 1, characterized in that, The initialization of the track information comprises the following steps: acquiring bounding box information of a target in the template image, copying the bounding box information of the target in the template image n times, taking the n copied bounding box information as the track information, and n is a constant.

4. The power robot target tracking method based on self-attention mechanism according to claim 1, characterized in that, The processing of the template image and the current search image to obtain the joint feature vector comprises the following steps: dividing the template image and the current search image into a plurality of image blocks, flattening each image block into a vector and converting it into an embedding vector through linear transformation to obtain template features and search features, adding position encoding to each feature, and splicing the template features and the search features into a joint feature vector.

5. The power robot target tracking method based on self-attention mechanism according to claim 4, characterized in that, The template image and the current search image adopt the same image blocking mode and position encoding strategy when processing the joint feature vector.

6. The power robot target tracking method based on self-attention mechanism according to claim 1, characterized in that, The ViT model encoder includes a plurality of Transformer encoder layers, each encoder layer including a self-attention mechanism and a feedforward neural network, for extracting global features of an input image and generating a joint feature matrix.

7. The power robot target tracking method based on self-attention mechanism according to claim 1, characterized in that, The trajectory information is a target bounding box information queue with a size of n.

8. The power robot target tracking method based on self-attention mechanism according to claim 1 or 7, characterized in that, The updating of the trajectory information according to the bounding box information of the target in the current search image comprises: dequeuing the head bounding box information of the target bounding box information queue and enqueuing the bounding box information of the target in the current search image.

9. A computer storage medium, characterized in that Executable program code is stored; the executable program code is used for executing the power robot target tracking method based on the self-attention mechanism as claimed in any one of claims 1-8.

Images