Modality missing RGBT tracking method and system based on missing perception prompts
By introducing a missing sense prompt module in RGBT tracking, combined with multimodal Transformer for information compensation, the problem of modal information is solved, and the robustness and performance of RGBT tracking is significantly improved.
Patent Information
- Application Number
- CN202411381944.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-30
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2044-09-30
AI Technical Summary
The existing RGBT tracking method fails to effectively deal with the missing visible light and thermal infrared mode information in actual scenarios, resulting in a significant decline in performance.
The modal missing RGBT tracking method based on missing perception prompts is adopted. The modal prototype missing perception prompt and context missing perception prompt module is combined with the multimodal Transformer for information compensation, and the training model is optimized to alleviate the accuracy reduction caused by modal missing.
The two-level efficient parameter prompts significantly alleviate the accuracy reduction problem caused by modal missing, improve the robustness of RGBT tracking, effectively compensate for missing information, and improve the tracking performance in actual scenarios.
Smart Images

Figure CN118887592B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of computer vision technology, and in particular to a modality missing RGBT tracking method and system based on missing perception prompts. Background Art
[0002] RGBT (RGB-Thermal) target tracking aims to track the target object in subsequent frames using the initial bounding box of the target object in the first frame of the tracking video. It is an important research direction in the field of target tracking and has extremely wide applications. Compared with single-modality RGB tracking, RGBT tracking has more advantages. Its ability to utilize enhanced and complementary information of visible light and thermal infrared modalities greatly improves the robustness of the model in challenging scenarios, such as low light, bad weather, and foggy conditions.
[0003] Early RGBT tracking methods directly concatenate the representations of visible and thermal encoders, which is effective to some extent while avoiding additional fusion modules, but they often introduce too much noise. In addition, some other methods select candidate boxes from the search frame and adopt various attention mechanisms for modal fusion and mix the representations from each candidate box pair. However, since these candidate boxes only contain local features in the search frame, the fusion paradigm does not fully utilize the global information and fails to exploit complementary multimodal information.
[0004] Recently, mainstream tracking methods usually use the powerful modeling ability of Transformer networks to achieve global and adaptive modal fusion. They combine multimodal template information with search area information during the fusion process, which can gradually reduce background noise and thus significantly enhance target information.
[0005] Although existing RGBT tracking methods have made significant progress, they are basically based on pre-processed noise-free datasets in ideal experimental scenarios. However, RGBT tracking in real-world scenarios will inevitably encounter serious modal missing issues caused by various environmental factors, such as overheating of acquisition equipment, frame skipping in multi-mode cameras, and packet loss during transmission. Directly applying existing off-the-shelf methods to noisy datasets without specifically considering the missing issues will lead to significant performance degradation. Therefore, it is very important to consider the missing links in the design process of RGBT tracking methods in real scenes. Summary of the invention
[0006] The technical problem to be solved by the present invention is to solve the problem that the missing information of visible light and thermal infrared modalities is not considered in the design process of the RGBT tracking method in actual scenes.
[0007] In order to solve the above technical problems, the present invention provides the following technical solutions:
[0008] A modality missing RGBT tracking method based on missing perception cues, comprising:
[0009] Randomly sample the video sequence to obtain training frame image pairs, and label the training frame image pairs, including missing types and non-missing types; and for images with missing types, set a corresponding missing compensation filling set;
[0010] Determine whether there are any missing images in the current image pair. If one of the images is missing, select the corresponding supplementary frame image from the corresponding missing compensation filling set according to the missing type, and combine it with the other image without missing to form an image pair as input. If there is no missing image in the current image pair, the current image pair is directly used as input;
[0011] Set the image template search frame bounding box;
[0012] The input image pair is segmented by searching the frame bounding box according to the image template, and then the dimension is converted to obtain the image semantic information;
[0013] Establish a missing-aware prompt modality fusion model, including a modality prototype missing-aware prompt module, a contextual missing-aware prompt module, and a multimodal Transformer module;
[0014] The image semantic information is used as the input of the modality prototype missing perception prompt module and the context missing perception prompt module, and the modality prototype compensation semantic information and the specific instance compensation semantic information are output respectively;
[0015] The multimodal Transformer module concatenates the modal prototype compensation semantic information, the specific instance compensation semantic information, and the key and value of the image template search frame bounding box in a cascade manner, and performs attention calculation with the image search query;
[0016] The established missing-aware cueing modal fusion model is optimized and trained using an optimization algorithm, and the target bounding box of the current frame image pair is obtained using the optimized missing-aware cueing modal fusion model.
[0017] Advantages: The present invention compensates for the basic missing information of RGBT tracking through two levels of efficient parameter hints, rather than restoring all the details in the missing frames. First, a series of modality prototype missing-aware hints are used to approximately compensate for the modality-specific but instance-independent modality information, which narrows the gap between the filled value and the modality prototype. Then, a series of contextual missing-aware hints are applied to perform instance-specific missing compensation through nearby contextual information, which further narrows the information gap between the modality prototype and the missing instance. With these two levels of hints, the problem of decreased accuracy caused by modality missing is significantly alleviated.
[0018] In one embodiment of the present invention, the missing types include visible light missing and infrared missing; when it is judged as visible light missing, the first N frames of visible light images at the current moment are selected as the missing compensation filling set of visible light; if there is no missing infrared frame image, the image of the current frame is selected; during training, a frame of visible light image and the infrared image of the current frame are randomly selected from the missing compensation filling set of visible light as an image pair, and input into the missing perception prompt modal fusion model for training; similarly, when it is judged as infrared missing, the processing method is the same as that when it is judged as visible light missing.
[0019] In one embodiment of the present invention, the image template search frame bounding box separates the image pair into a plurality of image blocks; the image blocks are converted from a two-dimensional image format to one-dimensional semantic information before being input into the modal prototype missing perception prompt module; and the image blocks maintain a two-dimensional image format when input into the context missing perception prompt module; wherein the image template search frame bounding box includes a visible light template frame area , visible light search frame area Infrared template frame area and infrared search frame area ; The image template search frame bounding box is denoted as .
[0020] In one embodiment of the present invention, the modal prototype missing perception prompt module adopts the form of multi-head attention, including a modal prototype compensation visible light missing prompt network, a modal prototype compensation infrared missing prompt network and a modal prototype adaptation non-missing prompt network; setting a learnable prompt parameter , and is divided into learnable key prompt parameters and learnable value hint parameters , and the learnable key hint parameter and learnable value hint parameters Inserted into the key and value in the attention respectively.
[0021] In one embodiment of the present invention, the modality prototype missing perception prompt module is expressed by the following formula:
[0022]
[0023] In the formula, Expressed as The attention operation of the layer, Expressed as The image features of the layer, Represented as the first Layer attention query, Represented as the dimension of semantic information, Represented as the first The key of the layer attention, Represented as the first The value of layer attention, Denoted as activation function, Represented as transposed.
[0024] In one embodiment of the present invention, the context missing awareness prompt module includes a selectable context missing awareness convolution layer and a double-layer bottleneck structure layer; and the selectable context missing awareness convolution layer includes a visible light missing learning convolution branch, an infrared missing learning convolution branch and a modality complete learning convolution branch;
[0025] According to the missing parts of the image, the corresponding learning convolution branch is selected to extract the key information of the segmented image pairs. The double-layer bottleneck structure layer restores the missing mode of the extracted key information pairs and outputs the specific instance compensation semantic information.
[0026] In one embodiment of the present invention, the context-deficient awareness prompt module is used It indicates that, Represented as a parameter set consisting of an optional context-missing convolutional layer and a double-layer bottleneck structure layer; a specific instance of the context-missing prompt module in each Transformer layer in the multimodal Transformer module compensates for the semantic information , obtained in the following way:
[0027] ;
[0028] ;
[0029] In the formula, is represented as an intermediate variable, Expressed as The key of the layer image feature, Expressed as The value of the layer image feature.
[0030] In one embodiment of the present invention, the missing perception prompt modality fusion model is expressed by the following formula: When the visible light image is used as the query feature,
[0031] ;
[0032] ;
[0033] ;
[0034] In the formula, It is represented as an attention operation with visible light images as query features, Expressed as a concatenation function; Represented as the key of the visible light template frame region, represents the key for the visible light search frame region, Indicates the key for the infrared search frame area. Represented as the value of the visible light template frame area, Represents the value of the visible light search frame area, Represents the value of the infrared search frame area, Denoted as activation function, Represented as transposed, Represented as the dimension of semantic information, It is represented as a key that concatenates two kinds of compensatory semantic information. Represented as the value of the concatenated two kinds of compensatory semantic information, A query representing a visible light search frame region; It is represented as the concatenation of the visible light template frame area, the visible light search frame area, the infrared search frame area, and the key of the two compensation semantic information; It is represented by the concatenation of the visible light template frame area, the visible light search frame area, the infrared search frame area, and the values of the two types of compensation semantic information.
[0035] In one embodiment of the present invention, when infrared images are used as query features,
[0036] ;
[0037] ;
[0038] ;
[0039] In the formula, It is represented as an attention operation with infrared images as query features, Represents the key of the infrared template frame area, Represented as the value of the infrared template frame area, It is represented as the concatenation of the infrared template frame area, the visible light search frame area, the infrared search frame area, and the key of the two compensation semantic information. It is represented as the concatenation of the infrared template frame area, the visible light search frame area, the infrared search frame area, and the values of the two types of compensation semantic information.
[0040] The present invention also provides a modality missing RGBT tracking system based on missing perception prompts, which uses the above-mentioned modality missing RGBT tracking method based on missing perception prompts, including:
[0041] The data set module is used to randomly sample the video sequence, obtain training frame image pairs, and label the training frame image pairs, including missing types and non-missing types; and for images with missing types, a corresponding missing compensation filling set is set;
[0042] Judgment module: judge whether there are any missing images in the current image pair. If one of the images is missing, select the corresponding supplementary frame image from the corresponding missing compensation filling set according to the missing type, and combine it with the other image without missing to form an image pair as input. If there is no missing in the current image pair, the current image pair is directly used as input;
[0043] Bounding box module, used to set the image template search frame bounding box;
[0044] The segmentation and dimension module is used to segment the input image pair according to the image template search frame bounding box, and then perform dimension conversion to obtain image semantic information;
[0045] The missing compensation module is used to establish a missing perception prompt modality fusion model, including a modality prototype missing perception prompt module, a context missing perception prompt module and a multimodal Transformer module; the image semantic information is used as the input of the modality prototype missing perception prompt module and the context missing perception prompt module, and the modality prototype compensation semantic information and the specific instance compensation semantic information are output respectively; the multimodal Transformer module splices the modality prototype compensation semantic information, the specific instance compensation semantic information and the key and value of the image template search frame bounding box in a cascade manner, and performs attention calculation with the image search query;
[0046] The optimization training prediction module is used to optimize the training of the established missing perception prompt modal fusion model using an optimization algorithm, and use the optimized missing perception prompt modal fusion model to obtain the target bounding box of the current frame image pair.
[0047] Compared with the prior art, the present invention has the following beneficial effects:
[0048] Different from the previous RGBT tracking, in order to solve the missing problem faced in actual scenes, the present invention proposes an RGBT tracking framework embedded with missing perception cues, which effectively compensates for the missing information and alleviates the degradation of robustness caused by the missing information.
[0049] The missing information is compensated by designing two levels of prompts. A set of learnable modality prototype missing-aware prompts is used to compensate for the prototype missing information that is specific to the modality but independent of the instance, narrowing the gap between the filled value and the modality prototype. A contextual missing-aware prompt network is designed to compensate for the detailed missing information of a specific instance, narrowing the information gap between the modality prototype and the missing instance. With these two levels of prompts, the accuracy degradation caused by modality missing is significantly alleviated. BRIEF DESCRIPTION OF THE DRAWINGS
[0050] Figure 1 The present invention is a flow chart of a method for modality missing RGBT tracking based on missing perception prompts according to an embodiment of the present invention.
[0051] Figure 2 This is a network structure diagram of a modality missing RGBT tracking method based on missing perception cues according to an embodiment of the present invention.
[0052] Figure 3 This is a network structure diagram of the loss-aware prompting modality fusion model according to an embodiment of the present invention.
[0053] Figure 4 A block diagram of a modality missing RGBT tracking system based on missing perception prompts according to an embodiment of the present invention. DETAILED DESCRIPTION
[0054] In order to facilitate those skilled in the art to understand the technical solution of the present invention, the technical solution of the present invention is further described in conjunction with the accompanying drawings of the specification.
[0055] The terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of this application, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.
[0056] See also Figure 1 and Figure 2 As shown, the present invention provides a modality missing RGBT tracking method based on missing perception prompts, comprising:
[0057] S10, randomly sampling the video sequence to obtain training frame image pairs, and labeling the training frame image pairs, including missing types and non-missing types; and for images of the missing type, a corresponding missing compensation filling set is set.
[0058] S20, determine whether there are any missing images in the current image pair. When one of the images is missing, select the corresponding supplementary frame image from the corresponding missing compensation filling set according to the missing type, and combine it with another non-missing image to form an image pair as input. If there is no missing image in the current image pair, the current image pair is directly used as input.
[0059] In one embodiment of the present invention, missing types include visible light missing and infrared missing. When it is judged that visible light is missing, the first N frames of visible light images at the current moment are selected as the missing compensation filling set of visible light. If there is no missing infrared frame image, the image of the current frame is selected. During training, a frame of visible light image and an infrared image of the current frame are randomly selected from the missing compensation filling set of visible light as an image pair, and input into the missing perception prompt modal fusion model for training. Similarly, when it is judged that infrared is missing, the processing method is the same as that when it is judged that visible light is missing.
[0060] S30, setting an image template search frame boundary box.
[0061] In one embodiment of the present invention, the template frame area and the search frame area are cropped according to the initialized bounding box to obtain the image template search frame bounding box. .in, Represented as the visible light template frame area, Represented as the infrared template frame area, Represented as the visible light search frame area, Indicates the infrared search frame area.
[0062] S40, searching for a frame bounding box according to the image template to segment the input image pair, and then performing dimensionality conversion to obtain image semantic information.
[0063] S50, establishing a missing-aware prompt modality fusion model, including a modality prototype missing-aware prompt module, a context missing-aware prompt module and a multimodal Transformer module.
[0064] S60, the image semantic information is used as the input of the modal prototype missing perception prompt module and the context missing perception prompt module, and the modal prototype compensation semantic information and the specific instance compensation semantic information are output respectively.
[0065] S70, the multimodal Transformer module concatenates the modal prototype compensation semantic information, the specific instance compensation semantic information, and the keys and values of the image template search frame bounding box in a cascade manner, and performs attention calculation with the image search query.
[0066] See also Figures 1 to 3As shown, in one embodiment of the present invention, the image template search frame bounding box separates the image pair into multiple image blocks, and the image blocks are converted from a two-dimensional image format to one-dimensional semantic information before being input into the modal prototype missing perception prompt module, and the image blocks maintain a two-dimensional image format when input into the context missing perception prompt module.
[0067] In this embodiment, the modal prototype missing perception prompt module and the context missing perception prompt module include three missing prompt networks according to the missing type and the non-missing type. According to different missing situations, they are sent to the prompt networks in the two missing perception prompt modules at the same time for compensation.
[0068] like Figure 3 Block diagram (a) in FIG. 1 , in this embodiment, the modality prototype missing-aware hint module aims to approximate the modality-specific but instance-independent missing information, which is information related to a specific modality but does not depend on a specific instance, thereby alleviating the instability caused by random modality missing. The key motivation of the modality prototype missing-aware hint module is to gradually learn a continuous hint vector and then adaptively compensate for the missing information with instance-independent modality prototypes, or hint vectors.
[0069] In this embodiment, the modality prototype missing perception prompt module adopts the form of multi-head attention, including the modality prototype compensation visible light missing prompt network, the modality prototype compensation infrared missing prompt network and the modality prototype adaptation non-missing prompt network. The notation for queries, keys, and values for layer multi-head attention is as follows:
[0070]
[0071] in, , respectively The linear weight of the corresponding query, the linear weight of the corresponding key, and the linear weight of the corresponding value of the layer multi-head attention. Expressed as a real number, Represents the dimension of semantic information. Set the learnable hint parameters , indicating the The modal prototype of the layer lacks perceptual cues and the learnable cue parameters Divided into learnable key prompt parameters and learnable value hint parameters , and the learnable key hint parameter and learnable value hint parameters Inserted into the key and value of the attention respectively to realize our modal prototype missing perception prompt. Specifically, the modal prototype missing perception prompt module is expressed by the following formula:
[0072]
[0073] In the formula, Expressed as The attention operation of the layer, Expressed as The image features of the layer, Represented as the first Layer attention query, Represented as the dimension of semantic information, Represented as the first The key of the layer attention, Represented as the first The value of layer attention, Denoted as activation function, is represented as transposed. Since all hints are only inserted into the key and value as prefixes, but not into the query as prefixes, the dimension of the output remains the same as the dimension of the input sequence. Therefore, no additional post-processing operations are required, and the compensated representation can be directly fed from the last transformer layer to the prediction head for bounding box prediction.
[0074] like Figure 3 In the block diagram (b) in FIG. 1 , in this embodiment, although the modality prototype missing perception prompt module takes effective measures to alleviate the performance degradation caused by random modality missing, the missing information of a specific instance cannot be compensated by relying solely on the prototype modality information. The missing information of a specific instance is information specially restored for the current missing frame. Therefore, the present invention proposes a context missing perception prompt module to generate context prompts to further compensate for instance-specific information.
[0075] In this embodiment, contextual cues are generated from current and historical video frames, which contain rich contextual information about missing instances, wherein the missing instance is the currently missing frame of image. Specifically, the contextual missingness-aware cue module includes an optional contextual missingness-aware convolution layer and a double-layer bottleneck structure layer. And the optional contextual missingness-aware convolution layer includes a visible light missingness learning convolution branch, an infrared missingness learning convolution branch, and a modality complete learning convolution branch, wherein the number of cores of each convolution branch is 3×3 and the stride is 2. The network architecture of the double-layer bottleneck structure includes a first linear transformation layer, a normalization layer, a ReLU activation layer, and a tail linear transformation layer in sequence to generate the final contextual cue compensation. That is, according to the missingness of the image, the corresponding learning convolution branch is selected to extract key information from the segmented image pair, and the double-layer bottleneck structure layer restores the missing modality of the extracted key information pair, and outputs the specific instance compensation semantic information. Formally speaking, the contextual missingness-aware cue module is represented as ,in represents its parameter set, which consists of an optional context-missing convolutional layer and a double-layer bottleneck structure layer. And the specific instance of the context-missing cue module in each Transformer layer in the multimodal Transformer module compensates for the semantic information , obtained in the following way:
[0076] ;
[0077] ;
[0078] in, is represented as an intermediate variable, Expressed as The key of the layer image feature, Expressed as The value of the layer image feature.
[0079] In this embodiment, the multimodal Transformer module concatenates the modal prototype compensation semantic information, the specific instance compensation semantic information, and the keys and values of the image template search frame bounding box, and performs attention calculation with the image search query to realize the feature interaction between the template and the search. The modal interaction between visible light and infrared has lost the compensation of the modality.
[0080] See also Figure 3 Block diagram (c) in, in this embodiment, the multimodal Tranformer module is the basic unit of the feature interaction and missing perception prompt module. The interaction between the template and the search feature is achieved by splicing the keys and values of the template and the search. Modal interaction in RGBT tracking can be implemented similarly, that is, further performing corresponding cascade operations on the keys and values from other modalities. In addition, since this article deals with more complex RGBT tracking scenarios involving modal missing, the present invention also needs to incorporate two types of missing perception prompt modules into the multimodal attention operation. Finally, with the visible light branch or the infrared branch, the interaction and compensation are presented by equations. Specifically, when the visible light image is used as the query feature:
[0081] ;
[0082] ;
[0083] ;
[0084] In the formula, It is represented as an attention operation with visible light images as query features, Expressed as a concatenation function; Represented as the key of the visible light template frame region, represents the key for the visible light search frame region, Indicates the key for the infrared search frame area. Represented as the value of the visible light template frame area, Represents the value of the visible light search frame area, Represents the value of the infrared search frame area, Denoted as activation function, Represented as transposed, Represented as the dimension of semantic information, It is represented as a key that concatenates two kinds of compensatory semantic information. Represented as the value of the concatenated two kinds of compensatory semantic information, A query representing a visible light search frame region; It is represented as the concatenation of the visible light template frame area, the visible light search frame area, the infrared search frame area, and the key of the two compensation semantic information; It is represented by the concatenation of the values of the visible light template frame area, the visible light search frame area, the infrared search frame area, and the two types of compensation semantic information. In this embodiment, the two types of compensation semantic information are modal prototype compensation semantic information and specific instance compensation semantic information.
[0085] When infrared images are used as query features:
[0086] ;
[0087] ;
[0088] ;
[0089] In the formula, It is represented as an attention operation with infrared images as query features, Represents the key of the infrared template frame area, Represented as the value of the infrared template frame area, It is represented as the concatenation of the infrared template frame area, the visible light search frame area, the infrared search frame area, and the key of the two compensation semantic information. It is represented as the concatenation of the infrared template frame area, the visible light search frame area, the infrared search frame area, and the values of the two types of compensation semantic information.
[0090] S80, using an optimization algorithm to optimize and train the established missing perception prompting modal fusion model, and using the optimized missing perception prompting modal fusion model to obtain a target bounding box of the current frame image pair.
[0091] In one embodiment of the present invention, an adaptive learning rate optimization algorithm ADAMW is applied to optimize and train the missing perception prompt modality fusion model. The adaptive learning rate optimization algorithm ADAMW is not improved by the present invention.
[0092] In one embodiment of the present invention, in order to realize a noisy image pair dataset in a real scene and evaluate the tracking performance of the missing perception prompt modality fusion model, the present invention simulates the corresponding missing variants of the three datasets RGBT210, RGBT234 and LasHeR respectively. Specifically, the missing rate is defined as the ratio of the missing duration to the duration of the entire sequence. Unless otherwise specified, the overall missing rate is set to 60%, with an average of 30% for the visible light and thermal infrared modalities respectively. In detail, we focus on challenging missing scenarios, in which random continuous frame missing occurs within a period of time in any modality. By setting different missing rates, the effectiveness and robustness of the present invention are fully verified.
[0093] See also Figures 1 to 3 As shown, unlike previous RGBT tracking, in order to solve the missing problem faced in actual scenarios, the present invention proposes an RGBT tracking framework embedded with missing-aware cues, which effectively compensates for the missing information and alleviates the degradation of robustness caused by missing information. The present invention also simulates the missing situation in real scenarios for existing data sets. The present invention inserts the proposed missing-aware cues into the existing mainstream tracking framework, and achieves effective compensation on different data sets, alleviating the performance degradation caused by missing information. In addition, different missing rates are set to fully verify the effectiveness and robustness of the present invention. The two missing-aware cues proposed in the present invention have a great effect on improving robustness, and the effect is most obvious when they are used in combination.
[0094] See also Figures 1 to 4 As shown, the present invention also provides a modality missing RGBT tracking system based on missing perception prompts, applying the above-mentioned modality missing RGBT tracking method based on missing perception prompts, including:
[0095] The data set module 10 is used to randomly sample the video sequence, obtain training frame image pairs, and label the training frame image pairs, including missing types and non-missing types; and for images of the missing type, a corresponding missing compensation filling set is set.
[0096] The judging module 20 is used to judge whether there are any missing images in the current image pair. When one of the images is missing, a corresponding supplementary frame image is selected from the corresponding missing compensation filling set according to the missing type, and the image is combined with another image without missing images to form an image pair as input. If there is no missing image in the current image pair, the current image pair is directly used as input;
[0097] The bounding box module 30 is used to set the image template search frame bounding box.
[0098] The segmentation and dimension module 40 is used to segment the input image pair according to the image template search frame bounding box, and then perform dimension conversion to obtain image semantic information.
[0099] The missing compensation module 50 is used to establish a missing perception prompt modal fusion model, including a modal prototype missing perception prompt module, a context missing perception prompt module and a multimodal Transformer module; the image semantic information is used as the input of the modal prototype missing perception prompt module and the context missing perception prompt module, and the modal prototype compensation semantic information and the specific instance compensation semantic information are output respectively; the multimodal Transformer module splices the modal prototype compensation semantic information, the specific instance compensation semantic information and the key and value of the image template search frame bounding box in a cascade manner, and performs attention calculation with the image search query.
[0100] The optimization training prediction module 60 is used to optimize the training of the established missing perception prompting modal fusion model using an optimization algorithm, and use the optimized missing perception prompting modal fusion model to obtain the target bounding box of the current frame image pair.
[0101] It is obvious to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential features of the present invention. Therefore, the embodiments should be regarded as exemplary and non-limiting from any point of view, and the scope of the present invention is defined by the appended claims rather than the above description, and it is intended that all changes falling within the meaning and scope of the equivalent elements of the claims are included in the present invention, and any reference numerals in the claims should not be regarded as limiting the claims involved.
[0102] The above-described embodiments merely represent implementation methods of the invention. The protection scope of the present invention is not limited to the above-described embodiments. For those skilled in the art, several modifications and improvements may be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention.
Claims
1. A modality missing RGBT tracking method based on missing perception cues, characterized in that: include: Randomly sample the video sequence to obtain training frame image pairs, and label the training frame image pairs, including missing types and non-missing types; and for images with missing types, set a corresponding missing compensation filling set; Determine whether there are any missing images in the current image pair. If one of the images is missing, select the corresponding supplementary frame image from the corresponding missing compensation filling set according to the missing type, and combine it with the other image without missing to form an image pair as input. If there is no missing image in the current image pair, the current image pair is directly used as input; Set the image template search frame bounding box; The input image pair is segmented by searching the frame bounding box according to the image template, and then the dimension is converted to obtain the image semantic information; Establish a missing-aware prompt modality fusion model, including a modality prototype missing-aware prompt module, a contextual missing-aware prompt module, and a multimodal Transformer module; The image semantic information is used as the input of the modality prototype missing perception prompt module and the context missing perception prompt module, and the modality prototype compensation semantic information and the specific instance compensation semantic information are output respectively; The multimodal Transformer module concatenates the modal prototype compensation semantic information, the specific instance compensation semantic information, and the key and value of the image template search frame bounding box in a cascade manner, and performs attention calculation with the image search query; The established missing-aware cueing modal fusion model is optimized and trained using an optimization algorithm, and the target bounding box of the current frame image pair is obtained using the optimized missing-aware cueing modal fusion model.
2. The method for modality missing RGBT tracking based on missing perception prompts according to claim 1, characterized in that: The missing types include visible light missing and infrared missing. When it is judged as visible light missing, the previous N frames of visible light images at the current moment are selected as the missing compensation filling set of visible light. If there is no missing infrared frame image, the image of the current frame is selected. During training, a frame of visible light image and the infrared image of the current frame are randomly selected from the missing compensation filling set of visible light as an image pair, and input into the missing perception prompt modal fusion model for training. Similarly, when it is judged as infrared missing, the processing method is the same as that when it is judged as visible light missing.
3. The modality missing RGBT tracking method based on missing perception prompts according to claim 1 is characterized in that: The image template search frame bounding box separates the image pair into multiple image blocks; the image blocks are first converted from two-dimensional image format to one-dimensional semantic information before being input into the modality prototype missing perception prompt module; And the image block maintains a two-dimensional image format when inputting the context-missing perception prompt module; wherein the image template search frame bounding box includes the visible light template frame area , visible light search frame area Infrared template frame area and infrared search frame area ; The image template search frame bounding box is denoted as .
4. The method for modality missing RGBT tracking based on missing perception prompts according to claim 1, characterized in that: The modality prototype missing perception prompt module adopts the form of multi-head attention, including the modality prototype compensation visible light missing prompt network, the modality prototype compensation infrared missing prompt network and the modality prototype adaptation non-missing prompt network; Setting learnable hint parameters , and is divided into learnable key prompt parameters and learnable value hint parameters , and the learnable key hint parameter and learnable value hint parameters Inserted into the key and value in the attention respectively.
5. The method for modality missing RGBT tracking based on missing perception prompts according to claim 4, characterized in that: The modal prototype missing perception prompt module is expressed by the following formula: In the formula, Expressed as The attention operation of the layer, Expressed as The image features of the layer, Represented as the first Layer attention query, Represented as the dimension of semantic information, Represented as the first The key of the layer attention, Represented as the first The value of layer attention, Denoted as activation function, Represented as transposed.
6. The method for modality missing RGBT tracking based on missing perception prompts according to claim 1, characterized in that: The context-missing aware hint module includes an optional context-missing aware convolutional layer and a double-layer bottleneck structure layer; And the optional contextual missingness-aware convolutional layers include a visible light missingness learning convolutional branch, an infrared missingness learning convolutional branch, and a modality complete learning convolutional branch; According to the missing parts of the image, the corresponding learning convolution branch is selected to extract the key information of the segmented image pairs. The double-layer bottleneck structure layer restores the missing mode of the extracted key information pairs and outputs the specific instance compensation semantic information.
7. The method for modality missing RGBT tracking based on missing perception prompts according to claim 6, characterized in that: The context-less awareness prompt module is used It indicates that, Represented as a parameter set consisting of an optional context-missing convolutional layer and a double-layer bottleneck structure layer; a specific instance of the context-missing prompt module in each Transformer layer in the multimodal Transformer module compensates for the semantic information , obtained in the following way: ; ; In the formula, Represented as an intermediate variable, Expressed as The key of the layer image feature, Expressed as The value of the layer image feature.
8. The method for modality missing RGBT tracking based on missing perception prompts according to claim 1, characterized in that: The missing perception prompt modality fusion model is expressed by the following formula. When the visible light image is used as the query feature, ; ; ; In the formula, It is represented as an attention operation with visible light images as query features, Expressed as a concatenation function; Represented as the key of the visible light template frame region, represents the key for the visible light search frame region, Indicates the key for the infrared search frame area. Represented as the value of the visible light template frame area, Represents the value of the visible light search frame area, Represents the value of the infrared search frame area, Denoted as activation function, Represented as transposed, Represented as the dimension of semantic information, It is represented as a key that concatenates two kinds of compensatory semantic information. Represented as the value of the concatenated two kinds of compensatory semantic information, A query representing a visible light search frame region; It is represented as the concatenation of the visible light template frame area, the visible light search frame area, the infrared search frame area, and the key of the two compensation semantic information; It is represented by the concatenation of the visible light template frame area, the visible light search frame area, the infrared search frame area, and the values of the two types of compensation semantic information.
9. The method for modality missing RGBT tracking based on missing perception prompts according to claim 8, characterized in that: When infrared images are used as query features, ; ; ; In the formula, It is represented as an attention operation with infrared images as query features, Represents the key of the infrared template frame area, Represented as the value of the infrared template frame area, It is represented as the concatenation of the infrared template frame area, the visible light search frame area, the infrared search frame area, and the key of the two compensation semantic information. It is represented as the concatenation of the infrared template frame area, the visible light search frame area, the infrared search frame area, and the values of the two types of compensation semantic information.
10. A modality missing RGBT tracking system based on missing perception cues, characterized in that: The method for tracking modality missing RGBT based on missing perception prompts according to any one of claims 1 to 9 is applied, comprising: The data set module is used to randomly sample the video sequence, obtain training frame image pairs, and label the training frame image pairs, including missing types and non-missing types; and for images with missing types, a corresponding missing compensation filling set is set; Judgment module: used to judge whether there are any missing images in the current image pair. When one of the images is missing, select the corresponding supplementary frame image from the corresponding missing compensation filling set according to the missing type, and combine it with the other image without missing to form an image pair as input. If there is no missing in the current image pair, the current image pair is directly used as input; Bounding box module, used to set the image template search frame bounding box; The segmentation and dimension module is used to segment the input image pair according to the image template search frame bounding box, and then perform dimension conversion to obtain image semantic information; The missing compensation module is used to establish a missing perception prompt modality fusion model, including a modality prototype missing perception prompt module, a context missing perception prompt module and a multimodal Transformer module; the image semantic information is used as the input of the modality prototype missing perception prompt module and the context missing perception prompt module, and the modality prototype compensation semantic information and the specific instance compensation semantic information are output respectively; the multimodal Transformer module splices the modality prototype compensation semantic information, the specific instance compensation semantic information and the key and value of the image template search frame bounding box in a cascade manner, and performs attention calculation with the image search query; The optimization training prediction module is used to optimize the training of the established missing perception prompt modal fusion model using an optimization algorithm, and use the optimized missing perception prompt modal fusion model to obtain the target bounding box of the current frame image pair.
Citation Information
Patent Citations
Real-time RGBT tracking method and device based on multi-challenge interactive learning
CN111476823A
RGBT visual tracking method and system based on two-stage fusion structure search
CN113837296A
RGBT target tracking method based on target perception enhancement fusion structure
CN117474957A
Universal multi-modal target tracking model based on adaptive perception, training method and application
CN118674945A
