Power equipment target tracking method based on vision-language multi-mode perception

Abstract

The invention discloses a power equipment target tracking method based on vision-language multi-modal perception, which belongs to the technical field of power equipment target tracking, and comprises the following steps: firstly, constructing a language-guided target perception multi-modal appearance modeling model, and introducing language and vision information to jointly carry out target representation; more effective visual features are selected by using stable language features, and depth alignment and enhancement are performed on two types of multi-modal information in combination with a self-attention mechanism, so that the adaptability of a target model is improved. Secondly, for better classification and regression, a vision-language multi-mode joint perception module is provided, the module removes non-target marks in a search area in advance through self-attention strong similarity prior calculation by using stable language features and effective visual features perceived by languages, and the tracking speed and accuracy are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of power equipment target tracking, and specifically to a power equipment target tracking method based on visual-linguistic multi-modal perception. Background Art

[0002] As the world's largest energy system, the safe and stable operation of the State Grid architecture is the lifeblood of people's livelihood and the economy. As the main component of the power grid architecture, most power equipment is located in relatively harsh and complex natural environments, characterized by a large number of points, a wide area, and long-term exposure in the wild. It is prone to aging, corrosion, and damage, which directly determines the continuity of power supply. Once a power equipment fails, it is bound to affect the safe and stable operation of the power system and cause significant losses to the social economy. Machine vision, as the most popular application of artificial intelligence technology, is the basis and breakthrough bottleneck of many innovative key technologies. It is used to perform intelligent detection and target tracking on power equipment, so as to accurately and real-time monitor whether there are abnormalities in power equipment.

[0003] In recent years, due to the improvement of the accuracy and stability of target detection and tracking tasks by vision-based tracking algorithms. However, due to the variety of power equipment types and the complex and changeable power tracking scenarios, the tracking accuracy is still affected, which in turn affects the effective monitoring of power equipment. Therefore, there is an urgent need to propose a precise and robust tracking method that can significantly improve the positioning accuracy and has high robustness to ensure stable and reliable operation in various complex tracking environments, so as to improve the monitoring and management efficiency of power equipment, master the operating status of equipment, discover potential problems in time, and ensure the stable operation of the power system. Summary of the Invention

[0004] In view of this, the present invention provides a power equipment target tracking method based on visual-linguistic multi-modal perception in view of the deficiencies of the prior art.

[0005] To solve the above technical problems, the technical solution adopted by the present invention is: a power equipment target tracking method based on visual-linguistic multi-modal perception, including the following steps:

[0006] Step 1: According to the target language text description of the power equipment to be tracked, input it into the language encoding module to extract the language features of the power equipment to be tracked as the target language template features for subsequent tracking;

[0007] Step 2: According to the initial frame picture of the power equipment to be tracked and the position coordinates of the target, intercept the target visual template, cut and flatten it into a sequence as the target visual template features for subsequent tracking;

[0008] Step 3: Read the next frame of the image as the current frame to be tracked. Crop the image to be tracked in the current frame according to the target position information in the previous frame of the image, and cut and flatten it into a sequence as the visual feature of the search area for the current frame.

[0009] Step 4: Concatenate the target language template feature, the target visual template feature, and the visual feature of the search area for the current frame, and send them into the multi-modal interaction model designed based on Visual Transformer for self-attention feature extraction and interaction among the target multi-modal information.

[0010] Step 5: Use the stable target language feature to guide the selection of the reliable target visual template feature, and remove the non-target information in the target template to obtain the target multi-modal appearance model.

[0011] Step 6: Through the visual-language multi-modal joint perception module, use the target multi-modal appearance model obtained in Step 5 to gradually remove the redundant information in the search area in advance while modeling the target and the search area, and obtain the relationship feature between the target and the search area.

[0012] Step 7: Take out the search area feature in the relationship feature and send it into the prediction head network to obtain the center position, center deviation, and bounding box size of the predicted target respectively, and obtain the position of the power equipment tracked in the current frame.

[0013] Step 8: Loop through Steps 3 to 7 until the video sequence is traversed to complete the target tracking of the power equipment.

[0014] Further, in Step 1, the language feature of the power equipment to be tracked is extracted using a preset Roberta-base network.

[0015] Further, in Step 4, the multi-modal interaction model designed based on Visual Transformer uses the ViT network pre-trained by a preset MAE for feature extraction and interaction among multi-modalities.

[0016] Further, in Step 4, the self-attention calculation is described as:

[0017]

[0018] where Q, K, and v represent the query vector, key vector, and value vector generated from the total feature concatenated respectively. d k represents the dimension of K.

[0019] Further, in Step 5, the method of using the stable target language feature to guide the selection of the reliable target visual template feature and removing the non-target information in the target visual template to obtain the target multi-modal appearance model is as follows:

[0020] Step 51. Calculate the attention weights of the target visual template, and the formula is:

[0021]

[0022] where represents the average similarity between the i-th target visual template and all language Tokens. Q LP represents the query vector of the language feature, represents the value vector of the target visual template feature, and M represents the number of attention heads;

[0023] Step 52. Optimize the target visual template feature according to the obtained attention weight values, and the rule is shown in the following formula:

[0024]

[0025] where the function means to retain the first m in j the maximum values (top-m j ) corresponding target visual template tokens, and the remaining visual tokens are regarded as background information and discarded; η represents the retention rate of the visual template Tokens, which is a hyperparameter and takes 0.8; n j is the total number of tokens input to the j-th layer of the visual template of the network; m j is the number of tokens to be retained in the output of the k-th layer of the visual template of the network.

[0026] Furthermore, in step 6, the method of using the target multi-modal appearance model to gradually remove redundant information in the search area in advance while modeling the target and the search area, and obtaining more accurate and concise relationship features between the target and the search area is:

[0027] Step 61. Calculate the prior similarity between the visual-linguistic multi-modal feature and the search area Tokens, and the formula is:

[0028]

[0029] where represents the average attention weight matrix of the search area Tokens in the i-th ViT layer, and Q' P represents the query vector of the remaining effective visual templates after language guidance, represents the value vector of the search area feature;

[0030] Step 62. The higher the similarity value, the greater the possibility that the search area Token belongs to the target; then the formula for removing non-target tokens in the search area is:

[0031]

[0032] Among them, the function means to retain the first p maximum values (top-p j ) in the current frame search area tokens corresponding to the maximum values, and the remaining visual tokens are regarded as background information and discarded; ξ represents the retention rate of visual template Tokens, which is a hyperparameter; n j is the total number of tokens of the input features of the current frame search area in the j-th layer of the network; p j is the number of tokens to be retained in the output of the current frame search area in the j-th layer of the network. j is the number of tokens to be retained in the output of the current frame search area in the j-th layer of the network.

[0033] Furthermore, in step 7, the search area features in the relationship features are taken out and sent into the prediction head network to obtain the center position, center deviation, and bounding box size of the prediction target respectively. The method for obtaining the position of the power equipment tracked in the current frame is as follows:

[0034] Step 71: Take out the features corresponding to the current frame search area in the final output of the network and send them into the prediction head network;

[0035] Step 72: Obtain the center position of the prediction target from the response map output by the center branch of the prediction head network;

[0036] Step 73: Obtain the center deviation of the prediction target from the response map output by the offset branch of the prediction head network;

[0037] Step 74: Obtain the specific size of the bounding box of the prediction target from the response map output by the size branch of the prediction head network;

[0038] Step 75: Finally, obtain the specific position of the prediction target in the current frame.

[0039] Furthermore, in step 7, the prediction head network consists of a center branch, an offset branch, and a size branch; all three branches contain 5 stacked Conv-BN-ReLU layers; the center position, center deviation, and bounding box size of the prediction target of the power equipment can be obtained by inputting the search area features.

[0040] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0041] 1. The present invention designs a language-guided target-aware appearance modeling strategy, which compensates for the deficiency of visual information by introducing language information and uses stable language semantic information to guide the selection of target visual features, so as to obtain a more robust target visual-language multi-modal appearance model.

[0042] 2. The present invention designs a visual - language multimodal joint perception module. This module utilizes the stable visual - language multimodal target model obtained in the first step. According to the prior similarity between the target multimodal information and the markers of each part in the search area, non - target markers in the search area are eliminated in advance during the target tracking inference stage, thereby improving the speed and accuracy of the tracking task.

[0043] 3. The present invention proposes a visual - language single - stream tracking framework. By flexibly interacting feature learning and relationship modeling, and making full use of the advantages of multimodal information, the positioning and tracking accuracy of the target is improved.

[0044] The power equipment target tracking method based on visual - language multimodal perception of the present invention is as follows. First, a language - guided target - aware multimodal appearance modeling model is constructed. Language and visual information are introduced to jointly perform target characterization. Stable language features are used to select more effective visual features, and the self - attention mechanism is combined to deeply align and enhance the two types of multimodal information, improving the adaptability of the target model. Second, for better classification and regression, a visual - language multimodal joint perception module is proposed. This module uses stable language features and effective visual features perceived by language. Through self - attention strong similarity prior calculation, non - target markers in the search area are removed in advance, improving the tracking speed and accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 is the flowchart of the present invention;

[0046] Figure 2 is the network architecture diagram of the present invention in visual - language joint perception;

[0047] Figure 3 is the comparison diagram of the tracking effects of the present invention and other algorithms on the TNL2K dataset. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0048] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are some, but not all, of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the described embodiments of the present invention fall within the scope of protection of the present invention.

[0049] Refer to Figure 1 , the embodiments of the present invention provide a power equipment target tracking method based on visual - language multimodal perception, and the steps are as follows:

[0050] Step S1: According to the target language text description of the power equipment to be tracked, input it into the language encoding module to extract the language features of the power equipment to be tracked as the target language template features for subsequent tracking. The specific steps are as follows:

[0051] Step S11: According to the target language text description of the power equipment, first preprocess the text, perform text cleaning (remove irrelevant information such as HTML tags, special characters, URLs, etc. from the text), and then perform word segmentation. Use a tokenizer to split the text into words or sub-word units, and convert it into a series of language tokens;

[0052] Step S12: Further add special identifiers at the beginning and end of the text, and add position encoding to each token to capture the sequence information in the text;

[0053] The language description of the tracking target will not be too long, and considering the real-time nature of tracking, we set the maximum sequence length of the language to 20. Therefore, it is necessary to pad or truncate the input text. This ensures that all inputs have the same length so that they can be processed in batches. At this time, the dimension of the language features is 1×20×768.

[0054] In step S1, the language features of the power equipment to be tracked are extracted using a preset Roberta-base network.

[0055] Step S2: According to the initial frame image of the power equipment to be tracked and the position coordinates of the target, intercept the target visual template, with dimensions of 3×128×128 (3×192×192), and perform cutting and flattening into a sequence, adding position encoding, as the target visual template features for subsequent tracking. At this time, the dimension size of the visual template features is 1×64×768 (1×144×768).

[0056] Step S3: Read the next frame image as the current frame to be tracked, and crop the image to be tracked in the current frame according to the target position information in the previous frame image, with dimensions of 3×256×256 (3×384×384), and perform cutting and flattening into a sequence, adding position encoding, as the visual features of the search area in the current frame. At this time, the dimension size of the visual features of the search area is 1×256×768 (1×576×768).

[0057] Step S4: Concatenate the target language template features, target visual template features, and visual features of the search area in the current frame together, and send them into a multi-modal interaction model designed based on Visual Transformer for self-attention feature extraction and interaction among target multi-modal information. The specific steps are as follows:

[0058] Step S41: Concatenate the target language template features (1×20×768), visual template features (1×64×768), and visual features of the current frame search area (1×256×768) together to form the total features with a dimension of 1×340×768;

[0059] Step S42: Feed the total features into a pre-trained Visual Transformer (ViT) model to perform self-attention feature extraction and interaction among target multi-modal information. The self-attention operation can be described as:

[0060]

[0061] where Q, K, and V respectively represent the query vector, key vector, and value vector generated from the concatenated total features. d k represents the dimension of K.

[0062] In step S4, the multi-modal interaction model designed based on Visual Transformer uses the pre-trained ViT network with preset MAE for feature extraction and interaction among multi-modalities.

[0063] Step S5: Through the language-guided target-aware multi-modal appearance modeling model, use stable target language features to guide the selection of reliable target visual template features, remove non-target information in the target template, and obtain a robust target multi-modal appearance model; the specific steps are as follows:

[0064] Step S51: Perceive visual template tokens with language information, and calculate the attention weights of target visual template tokens. The formula is:

[0065]

[0066] where, represents the average similarity between the i-th target visual template and all language Tokens. Q LP represents the query vector of language features, represents the value vector of target visual template features, and M represents the number of attention heads;

[0067] Step S52: Optimize the target visual template features according to the obtained attention weight values. The rule is shown in the following formula:

[0068]

[0069] where the function means to retain the first m in j the maximum values (top-m j)The corresponding target visual template tokens, and the remaining visual tokens are identified as background information and discarded; η represents the retention rate of visual template tokens, which is a hyperparameter and takes 0.8; n j is the total number of tokens in the input of the j-th layer visual template of the network; m j is the number of tokens to be retained in the output of the j-th layer visual template of the network. After three layers of language-guided target perception modeling, the visual template is optimized from the initial 1×64×768 to 1×32×768.

[0070] Step S6: Through the visual-linguistic multi-modal joint perception module, using the target multi-modal appearance model obtained in Step 5, while modeling the target and the search area, gradually remove the redundant information in the search area in advance to obtain the relationship features between the target and the search area; the specific steps are as follows:

[0071] Step S61: The prior similarity between the selected high-quality visual-linguistic multi-modal features and the search area tokens, the formula is:

[0072]

[0073] Among them, represents the average attention weight matrix of the search area tokens in the i-th ViT layer, Q′ P represents the query vector of the remaining effective visual template after language guidance, represents the value vector of the search area features;

[0074] Step S62: The higher the similarity value, the greater the possibility that the search area token belongs to the target; then the formula for removing non-target tokens in the search area is:

[0075]

[0076] Among them, the function means to retain the first p in j the maximum values (top-p j ) corresponding to the current frame search area tokens, and the remaining visual tokens are identified as background information and discarded; ξ represents the retention rate of visual template tokens, which is a hyperparameter and takes 0.8; n j is the total number of tokens in the input features of the j-th layer current frame search area of the network; p j is the number of tokens to be retained in the output of the j-th layer current frame search area of the network. After three layers of visual-linguistic joint perception, the current search area visual template changes from the initial 1×256×768 to 1×131×768.

[0077] Step S7: Extract the search area feature in the relationship feature and send it into the prediction head network to obtain the center position, center deviation, and bounding box size of the prediction target, thereby obtaining the position of the power equipment being tracked in the current frame. The specific steps are as follows:

[0078] Step S71: Extract the search area feature corresponding to the current frame from the final output in the network and send it into the prediction head network.

[0079] Step S72: Obtain the center position of the prediction target from the response map output by the center branch of the prediction head network.

[0080] Step S73: Obtain the center deviation of the prediction target from the response map output by the offset branch of the prediction head network.

[0081] Step S74: Obtain the specific size of the bounding box of the prediction target, that is, the size of the tracking target at this time, from the response map output by the size branch of the prediction head network.

[0082] Step S75: Finally, obtain the specific position and size of the prediction target in the current frame.

[0083] In Step S7, the prediction head network consists of a center branch, an offset branch, and a size branch; all three branches contain 5 stacked Conv-BN-ReLU layers; by inputting the search area feature, the center position, center deviation, and bounding box size of the power equipment prediction target can be obtained.

[0084] Step S8: Loop and execute Steps S3 to S7 until the video sequence is traversed to complete the target tracking of the power equipment.

[0085] The experiment of the present invention was implemented on a computer with an Intel i7-10700 CPU (2.90 GHZ), 16 GB of RAM, and NVDIA TITAN RTX GPUs using the Python 3.7 programming language on the PyTorch framework. Figure 2 It is the internal structure diagram of the vision-language joint perception network. Figure 3 It is the accuracy comparison result graph of the method of the present invention (VLMP-256) and its variant (VLMP-384) with other advanced object tracking algorithms (SeqTrack, OSTrack, CiteTrack, SwinTrack, VLT, AutoMatch) on the TNL2K dataset. It has achieved excellent performance in terms of success rate. It can be seen that the present invention has more accurate positioning accuracy and better robustness, and can efficiently complete the tracking task of power equipment in the power scenario.

[0086] The above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those of ordinary skill in the art can still modify or equivalently replace the specific implementation manners of the present invention. Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention is within the scope of the claims of the present invention pending approval.

Claims

1. A power equipment target tracking method based on vision-language multi-modal perception, characterized in that Including the following steps: Step 1: According to the target language text description of the power equipment to be tracked, input it into the language encoding module to extract the language features of the power equipment to be tracked as the target language template features for subsequent tracking; Step 2: According to the initial frame image of the power equipment to be tracked and the position coordinates of the target, intercept the target visual template, cut and flatten it into a sequence as the target visual template features for subsequent tracking; Step 3: Read the next frame image as the current frame to be tracked, crop the image to be tracked in the current frame according to the target position information in the previous frame image, cut and flatten it into a sequence as the visual features of the search area in the current frame; Step 4: Concatenate the target language template features, target visual template features and visual features of the current frame search area together, and send them into the multi-modal interaction model designed based on Visual Transformer for self-attention feature extraction and interaction among target multi-modal information; Step 5: Use stable target language features to guide the selection of reliable target visual template features, remove non-target information in the target template to obtain the target multi-modal appearance model; Step 6: Through the visual-language multi-modal joint perception module, use the target multi-modal appearance model obtained in Step 5 to gradually remove redundant information in the search area in advance while modeling the target and the search area to obtain the relationship features between the target and the search area; Step 7: Extract the search area features in the relationship features and send them into the prediction head network to obtain the center position, center deviation and bounding box size of the predicted target respectively, and get the position of the power equipment to be tracked in the current frame; Step 8: Loop through Steps 3 to 7 until the video sequence is traversed to complete the target tracking of the power equipment.

2. The method for tracking power equipment targets based on visual-linguistic multi-modal perception according to claim 1, wherein: In Step 1, the language features of the power equipment to be tracked are extracted using a preset Roberta-base network.

3. The method for tracking power equipment targets based on vision-language multi-modal perception according to claim 1, wherein: In Step 4, the multi-modal interaction model designed based on Visual Transformer uses a ViT network pre-trained with a preset MAE for feature extraction and interaction among multi-modalities.

4. The method for tracking power equipment targets based on visual-linguistic multi-modal perception according to claim 1, characterized in that: In Step 4, the self-attention calculation is described as: Among them, Q, K, and V respectively represent the query vector, key vector, and value vector generated from the total features formed by splicing. d k represents the dimension of K.

5. The method for tracking power equipment targets based on vision-language multi-modal perception according to claim 1, wherein: In Step 5, the method of using stable target language features to guide the selection of reliable target visual template features and removing non-target information in the target visual template to obtain the target multi-modal appearance model is as follows: Step 51: Calculate the attention weight of the target visual template, and the formula is: Among them, represents the average similarity between the i-th target visual template and all language Tokens. Q LP represents the query vector of the language feature, represents the value vector of the target visual template feature, and M represents the number of attention heads; Step 52: Optimize the target visual template features according to the obtained attention weight value, and the rule is shown in the following formula: Among them, the function means to retain the first m maximum values (top-m j ) corresponding target visual template tokens in j , and the remaining visual tokens are regarded as background information and discarded; η represents the retention rate of visual template Tokens, which is a hyperparameter and takes 0.8; n j is the total number of tokens input to the j-th layer visual template of the network; m j is the number of tokens to be retained in the output of the j-th layer visual template of the network.

6. The method for tracking power equipment targets based on vision-language multi-modal perception according to claim 1, wherein: In Step 6, the method of using the target multi-modal appearance model to gradually remove redundant information in the search area in advance while modeling the target and the search area to obtain more accurate and concise relationship features between the target and the search area is as follows: Step 61: Calculate the prior similarity between the visual-language multi-modal features and the search area Tokens, and the formula is: Among them, represents the average attention weight matrix of the Tokens in the search region in the i-th ViT layer, and Q' P represents the query vector of the remaining effective visual templates after language guidance, represents the value vector of the search region features; Step 62: The higher the similarity value, the greater the possibility that the search area Token belongs to the target; then the formula for removing non-target tokens in the search area is: Among them, the function means to retain the first p in j the maximum values (top-p j ) corresponding to the tokens in the current frame search area, and the remaining visual tokens are regarded as background information and discarded; ξ represents the retention rate of visual template Tokens, which is a hyperparameter; n j is the total number of tokens of the input feature of the current frame search area in the j-th layer of the network; p j is the number of tokens to be retained in the output of the current frame search area in the j-th layer of the network.

7. The method for tracking power equipment targets based on vision-language multi-modal perception according to claim 1, characterized in that: In step 7, the search area feature in the relational feature is taken out and sent into the prediction head network to obtain the center position, center deviation, and bounding box size of the prediction target respectively. The method for obtaining the position of the power equipment tracked in the current frame is as follows: Step 71: Take out the search area feature corresponding to the current frame in the final output of the network and send it into the prediction head network; Step 72: Obtain the center position of the prediction target from the response map output by the center branch of the prediction head network; Step 73: Obtain the center deviation of the prediction target from the response map output by the offset branch of the prediction head network; Step 74: Obtain the specific size of the bounding box of the prediction target from the response map output by the size branch of the prediction head network; Step 75: Finally, obtain the specific position of the prediction target in the current frame.

8. The method for tracking power equipment targets based on vision-language multi-modal perception according to claim 1, wherein: In step 7, the prediction head network consists of a center branch, an offset branch, and a size branch; all three branches contain 5 stacked Conv-BN-ReLU layers; inputting the search area feature can obtain the center position, center deviation, and bounding box size of the prediction target of the power equipment.

Images