A method and system for occluded vehicle re-identification based on multi-task joint pre-training

Abstract

The application belongs to the field of computer vision, and discloses a kind of based on multi-task joint pre-training's occlusion vehicle re-identification method and system, the method includes: obtaining vehicle image to be processed;Based on vehicle image, generate a pre-training model through a unified multi-task joint pre-training framework, and the pre-training framework executes the following steps in parallel: execute foreground enhancement knowledge distillation, learn perspective invariance feature by rotating image block to foreground feature and distilling its knowledge to global feature;Execute visual-linguistic contrast learning, generate foreground perception text prompt aligned with vehicle foreground area, and carry out image-text contrast learning to enhance the representation ability of visible foreground area;And execute mask visual modeling, reconstruct foreground feature and semantic reconstruction of vehicle perspective mask to image feature after mask to learn local feature;Finally, based on pre-training model, occlusion vehicle is re-identified, and occlusion interference is effectively inhibited.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, in particular to a method and system for occluded vehicle re-identification based on multi-task joint pre-training. BACKGROUND

[0002] Vehicle Re-identification (ReID) technology aims to retrieve vehicle images with the same identity as the query vehicle in a monitoring network across cameras, perspectives, and scenes. With the development of deep learning technology, existing vehicle ReID methods have achieved remarkable results in handling complete and clear vehicle images. However, in real traffic monitoring scenarios, vehicles are often occluded by various obstacles such as pedestrians, other vehicles, roadside trees, and traffic facilities, making the vehicle body region partially invisible.

[0003] This occlusion introduces a lot of noise interference, severely damaging the integrity of the vehicle's visual features, making feature extraction and alignment extremely difficult, and thus leading to a sharp decline in ReID performance. Currently, most vehicle ReID research focuses on matching complete vehicles, while research on occluded vehicle ReID is relatively scarce. Some works draw on occluded pedestrian ReID, trying to align the visible parts using human part parsing or pose estimation, but these methods are not ideal for vehicles with small appearance differences and more drastic perspective changes, and are highly dependent on additional high-precision parsing tools.

[0004] In addition, another bottleneck of existing methods is the lack of a general pre-training model tailored for downstream occlusion tasks. Most works directly use models pre-trained on large-scale general image classification datasets such as ImageNet as the backbone network. However, there is a significant domain difference between the data distribution of ImageNet and the specific domain of vehicle ReID tasks (especially in scenes full of occlusions), which makes the initial feature representation learned by the model not optimal, limiting its fine-tuning potential in downstream tasks. Although recent works have begun to explore general pre-training paradigms for ReID tasks, visual-linguistic pre-training methods specifically for occluded vehicles are still in the blank.

[0005] In summary, existing technologies face the problem of low recognition accuracy when dealing with occluded vehicle ReID tasks, due to the difficulty of feature alignment caused by noise interference from occlusions, and the lack of a general pre-training model that can learn robust foreground representations. SUMMARY

[0006] The purpose of this invention is to provide an occluded vehicle re-identification method and system based on multi-task joint pre-training, aiming to solve the problems of performance degradation in vehicle re-identification due to occlusion noise and viewpoint changes, as well as the lack of a general pre-trained model in existing technologies. This invention designs a multi-task pre-training framework that integrates foreground enhancement knowledge distillation, visual-language contrastive learning, and masked visual modeling. This enables the model to effectively focus on visible vehicle foreground regions and learn general feature representations robust to occlusion and viewpoint changes, thereby improving performance in various vehicle re-identification tasks.

[0007] In a first aspect, the present invention provides an occluded vehicle re-identification method based on multi-task joint pre-training, comprising the following steps:

[0008] Acquire the vehicle image to be processed;

[0009] Based on the vehicle images, a pre-trained model is generated through a unified multi-task joint pre-training framework. The pre-trained model consists of a unimodal image encoder, a unimodal text encoder, and a multimodal image decoder. The pre-training framework executes the following steps in parallel:

[0010] The foreground enhancement knowledge distillation step includes: extracting global visual features and foreground subject features of the vehicle image using the single-modal image encoder; performing an image block-level rotation operation on the foreground subject features to generate rotated foreground subject features; and transferring the viewpoint invariant information contained in the rotated foreground subject features to the global visual features through knowledge distillation.

[0011] Performing a visual-language contrastive learning step includes: generating a foreground-aware text prompt aligned with the visible foreground region of the vehicle based on foreground information of the vehicle image; extracting text features of the foreground-aware text prompt using the unimodal text encoder; and performing contrastive learning on the global visual features and the text features to enhance the representation of key foreground information of the vehicle by the global visual features.

[0012] The masking visual modeling step includes: performing a masking operation on the image blocks of the vehicle image to obtain masked image features; inputting the masked image features and the text features into the multimodal image decoder to reconstruct the masked foreground subject features and the vehicle view mask corresponding to the vehicle image in parallel.

[0013] Based on the optimized single-modal image encoder in the pre-trained model, the target occluded vehicle is re-identified.

[0014] As an optional implementation of the first aspect of this application, before acquiring the vehicle image to be processed, a pre-training data preparation step is further included. This step specifically involves: constructing an obstacle instance library, which contains obstacle images and corresponding instance segmentation masks for various traffic monitoring scenarios, and labeling each obstacle instance with a priori position index and a priori view index; based on a complete vehicle dataset, according to the view information of the vehicle image and the position and view prior indexes of the obstacle instances, pasting obstacles at reasonable positions in the vehicle image to synthesize an occluded vehicle image; generating foreground annotations, including: extracting an initial foreground region mask of the occluded vehicle image using a vehicle parsing model; obtaining the precise position mask of the obstacle during synthesis; and removing regions overlapping with the precise position mask of the obstacle from the initial foreground region mask to obtain a clean foreground mask containing only the visible portion of the vehicle, the clean foreground mask being used to generate the foreground main features and the vehicle view mask.

[0015] As an optional implementation of the first aspect of this application, the step of generating the foreground-aware text prompt in the visual-language contrastive learning step specifically includes: generating a learnable background-normalized foreground image based on the clean foreground mask, wherein the background region of the vehicle image is filled with a learnable background vector; inputting the learnable background-normalized foreground image to the unimodal image encoder to extract its global foreground features; inputting the global foreground features to a multilayer perceptron to generate multiple foreground subject semantic vectors; and fusing the multiple foreground subject semantic vectors with learnable tags in an initial text prompt template containing multiple learnable tags to generate the final foreground-aware text prompt.

[0016] As an optional implementation of the first aspect of this application, the mask visual modeling step includes: performing random mask sampling on the image patch features of the vehicle image, replacing the masked image patches with learnable mask markers; inputting the mask image features into the multimodal image decoder, wherein each decoder block of the multimodal image decoder contains a cross-attention layer, wherein the cross-attention layer takes the text features as key and value input and performs information fusion with the mask image features; using the output of the multimodal image decoder, calculating a first mean square error loss between the reconstructed foreground subject features and the real foreground subject features, and a second mean square error loss between the reconstructed vehicle view mask and the real vehicle view mask, and updating the parameters of the multimodal image decoder and the single-modal image encoder according to the first and second mean square error losses.

[0017] As an optional implementation of the first aspect of this application, the foreground enhancement knowledge distillation step includes: reshaping multiple image block labels of the foreground subject features into a two-dimensional matrix; performing a rotation operation on the two-dimensional matrix to generate the rotated foreground subject features, wherein the classification labels retain their original position information during the rotation, and the rotation angle is within a preset range; calculating the feature distribution difference between the global visual features and the rotated foreground subject features as a knowledge distillation loss, and updating the parameters of the single-modal image encoder according to the knowledge distillation loss.

[0018] As an optional implementation of the first aspect of this application, the method further includes: constructing an overall loss function for the pre-trained framework, comprising a knowledge distillation loss for performing a foreground enhancement knowledge distillation step, a contrastive learning loss for performing a vision-language contrastive learning step, and a weighted sum of a first mean squared error loss and a second mean squared error loss for performing a masked visual modeling step; the parameters of the unimodal text encoder are kept fixed during pre-training; and the parameters of the unimodal image encoder and the multimodal image decoder are updated end-to-end according to the overall loss function.

[0019] As an optional implementation of the first aspect of this application, the steps for re-identifying a target occluded vehicle based on the optimized single-modal image encoder in the pre-trained model include: fine-tuning the optimized single-modal image encoder in the pre-trained model as a backbone network for downstream tasks; adding a linear projection layer after the backbone network to generate identity prediction probabilities; jointly optimizing the backbone network and the linear projection layer using cross-entropy loss and triplet loss; and extracting global visual features of the target vehicle image during the inference phase, and performing similarity matching and retrieval based on the global visual features to complete the re-identification.

[0020] Secondly, embodiments of this application provide an occluded vehicle re-identification system based on multi-task joint pre-training, comprising:

[0021] The data acquisition module is used to acquire images of the vehicles to be processed.

[0022] A joint pre-training module is used to generate a pre-trained model based on the vehicle image through a unified multi-task joint pre-training framework. The joint pre-training module includes:

[0023] The foreground enhancement knowledge distillation unit is used to extract the global visual features and foreground subject features of the vehicle image using the single-modal image encoder; perform image block-level rotation operation on the foreground subject features to generate rotated foreground subject features; and transfer the viewpoint invariance information contained in the rotated foreground subject features to the global visual features through knowledge distillation.

[0024] A visual-language contrastive learning unit is used to generate a foreground-aware text prompt aligned with the visible foreground region of the vehicle based on the foreground information of the vehicle image; extract the text features of the foreground-aware text prompt using the unimodal text encoder; and perform contrastive learning on the global visual features and the text features to enhance the representation of key information of the vehicle foreground by the global visual features.

[0025] A masking visual modeling unit is used to perform masking operations on image blocks of the vehicle image to obtain masked image features; the masked image features and the text features are input to the multimodal image decoder to reconstruct the masked foreground subject features and the vehicle view mask corresponding to the vehicle image in parallel.

[0026] The vehicle re-identification module is used to re-identify occluded vehicles based on the optimized single-modal image encoder in the pre-trained model.

[0027] Thirdly, embodiments of this application provide an electronic device, which includes a processor, a memory, and a program or instructions stored in the memory and executable on the processor. When the program or instructions are executed by the processor, they implement the steps of the method described in the first aspect.

[0028] Fourthly, embodiments of this application provide a readable storage medium on which a program or instructions are stored, which, when executed by a processor, implement the steps of the method described in the first aspect.

[0029] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0030] 1. Enhanced foreground perception capability: By designing text cues (FTP) and visual-language contrastive learning (VCL) for foreground perception, this invention guides the model's attention to the visible foreground area of ​​the vehicle, and enhances the cross-modal description capability of key identity information through image-text alignment, effectively suppressing the interference of occlusion noise.

[0031] 2. Enhanced viewpoint robustness: Through foreground augmentation knowledge distillation (FKD), this invention uses image block-level rotation operations to simulate vehicle viewpoint changes, and through the mutual distillation of foreground and global features, the model learns viewpoint-invariant feature representations, thereby better adapting to feature alignment under various viewpoints.

[0032] 3. Learning robust local representations: Through masked visual modeling (MVM), this invention not only reconstructs the masked foreground information at the feature level, but also reconstructs the vehicle's view mask at the semantic level. This dual reconstruction task enables the model to learn more complete and robust vehicle component-level information, which is beneficial for part-to-whole and part-to-part matching.

[0033] 4. Provides a general pre-trained model: The multi-task joint pre-training framework proposed in this invention can produce a general pre-trained model tailored for occluded vehicle re-identification tasks. This model, used as initialization parameters for downstream tasks, can significantly improve performance on occluded, partial, and complete vehicle re-identification tasks, demonstrating strong generalization ability. Attached Figure Description

[0034] Figure 1 This is a schematic diagram of the construction process of the Occ-Wild benchmark dataset used to generate occlusion data in an embodiment of the present invention;

[0035] Figure 2 This is a detailed illustration of the Occ-Wild dataset in an embodiment of the present invention;

[0036] Figure 3 This is the overall architecture diagram of the multi-task vision-language pre-training framework proposed in this embodiment of the invention;

[0037] Figure 4 This is a schematic diagram illustrating the construction of foreground-aware text prompts (FTP) in an embodiment of the present invention;

[0038] Figure 5 This is a detailed flowchart of the pre-training framework in an embodiment of the present invention;

[0039] Figure 6 This is a schematic diagram of the structure of an occluded vehicle re-identification system based on multi-task joint pre-training proposed in an embodiment of this application. Detailed Implementation

[0040] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0041] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0042] Example 1

[0043] This invention proposes a vehicle re-identification method based on multi-task vision-language pre-training. The core idea of ​​this method is to construct a pre-training framework that combines three tasks: foreground enhancement knowledge distillation (FKD), vision-language contrastive learning (VCL), and masked vision modeling (MVM), in order to learn a general vehicle feature representation that is robust to occlusion and viewpoint changes.

[0044] 1. Pre-training data preparation

[0045] To support pre-training, this embodiment first constructs a large-scale occluded vehicle re-identification benchmark dataset (named Occ-Wild). For example... Figure 1 As shown, its generation process is as follows:

[0046] (a) Obstacle Instance Library Construction: Collect images of common obstacles in traffic monitoring scenarios from the Internet, such as pedestrians, utility poles, trees, and traffic cones, and construct an instance library containing multiple categories of obstacles. Use an instance segmentation model (such as SAMv2) to generate an accurate mask for each obstacle instance.

[0047] For example, the present invention collects images of common obstacles in traffic monitoring from the Internet to construct a diverse obstacle instance library, which includes 12 categories of obstacles, with 15 samples in each category, for a total of 180 instances. Figure 2 (a) describes the specific names of 12 obstacles. For each obstacle instance, a powerful zero-shot segmentation capability is used to segment the obstacle image. Specifically, an anchor point is randomly set within the main region of the obstacle as a cue word to quickly infer and obtain segmentation results containing only the obstacle. These segmentation results can effectively remove foreground-irrelevant pixels to obtain images containing only the obstacle and its mask annotation.

[0048] After extracting images containing only obstacles, this study empirically concluded that it is unreasonable for certain obstacles to appear in specific locations and from specific viewpoints. For example, traffic lights typically appear at the top of the image and in the top-down view of vehicles, while road obstacles cannot appear in these areas. To ensure that obstacles can be composited into reasonable positions in the original image, this invention adds a priori position index and viewpoint index to each instance in the obstacle instance library. Specifically, the original image is divided horizontally into top, middle, and bottom regions, corresponding to the position indexes, respectively. Furthermore, it considers the prior view regions that obstacles may tend to (including front, top, back, and side regions), and introduces an additional view index for each instance in the obstacle instance library. .

[0049] (b) Occlusion Image Synthesis: Based on an existing complete vehicle dataset (such as VERI-Wild), occlusion is synthesized for each vehicle image. To ensure the realism of the synthesis, reasonable position and viewpoint prior indices are preset for each obstacle instance. During synthesis, based on the vehicle's viewpoint information (such as front and side views) and the obstacle's prior index, the obstacle is pasted into a reasonable position in the original image to generate an occluded vehicle image.

[0050] After collecting a library of obstacle examples, one of the most straightforward image synthesis methods is to randomly stitch the obstacles pixel-wise with the original image. However, obstacles appearing in inappropriate locations will introduce additional noise into the synthesized data. Figure 1 As shown in (b), when traversing the complete VERI-Wild dataset, obstacle elements are randomly selected from the obstacle instance library, and their corresponding orientation and view index values ​​are obtained. Then, an existing analytical model is used to extract the view mask image from the original image. Specifically, the view mask image labels the pixels of the four viewpoints of the vehicle body as {1, 2, 3, 4}, while the pixels of the remaining parts are labeled as 0. Finally, the position of the occluded object relative to the original image (i.e., the intersection of the viewpoint region and the position region) is limited based on the two index values, and the obstacle is then pasted into the specified position to synthesize the occlusion image. It is worth noting that, since the pixel scale of different types of obstacle instances is not aligned with the vehicle body pixel scale in the original image, a suitable pixel-level scaling ratio is preset before synthesis to ensure the reasonableness of the obstacle scale.

[0051] To more closely resemble real-world occlusion settings, Occ-Wild also includes composite images of multiple occlusion instances, such as... Figure 2As shown in (b). Specifically, different types of obstacle instances are selected and pasted at different locations in the original image, ensuring that there is no positional overlap between obstacles. In addition, the present invention further implements two data augmentation strategies to increase the diversity of Occ-Wild: (1) Each obstacle instance is not limited to any specific location and can be randomly offset within two index-restricted regions. (2) More occlusion samples under complex weather conditions are generated using readily available image rain and fog algorithms. Figure 2 Statistics for each obstacle in the synthetic data are provided.

[0052] (c) Foreground annotation generation: The initial foreground region of the synthesized occluded image is extracted using the vehicle parsing model, while the precise location mask of the obstacles during synthesis is obtained. By subtracting the obstacle region from the initial foreground region, a clean foreground mask containing only the visible vehicle portion is obtained.

[0053] In occlusion tasks, obtaining pure foreground annotations as auxiliary information can significantly improve the model's ability to distinguish visible regions. However, the high cost of prior foreground annotations makes it difficult for existing vehicle parsing models to accurately distinguish between visible and occluded regions in various complex scenes. This erroneous parsing further leads to multi-view misalignment issues for the vehicle.

[0054] To address the aforementioned issues, a foreground annotation generation method is introduced to automatically generate vehicle foreground masks for each sample in Occ-Wild. Assuming the use of... and Let represent the VeRiWILD and Occ-Wild datasets respectively, where and , and They represent and The number of samples. Extracting synthetic images from an existing vehicle analytical model. The pixel coordinates of the foreground region are At the same time, the synthetic method can obtain the results a priori. k An obstacle is located in the composite image. pixel coordinate set Subsequently, the foreground region was calculated. and k Obstacle areas The intersection of these intersections is used to represent the occlusion information contained in the foreground region. Finally, from the foreground region... Remove the intersecting regions to obtain pure foreground annotations As shown in formula (1).

[0055] =

[0056] in k This indicates the number of obstacles in a composite image, as set in this context. .

[0057] based on The coordinate position is first preserved in this invention. The middle belongs to the set The pixels of the target image are set to 0, and the outer pixels are set to 0 to obtain the composite foreground subject image. Subsequently, Set the inner pixels to 1 and the outer pixels to 0 to generate foreground annotation Finally, view-level foreground annotations. Foreground annotations are possible The corresponding derivation is as follows. Specifically, the pixel values ​​of the four view labels in the view mask image corresponding to the original image are mapped to... The foreground region with a value of 1 is thus obtained. .

[0058] Based on the above three parts, a benchmark dataset of large-scale complex scenes, namely Occ-Wild, can be created for VERI-Wild at a relatively low cost. It is worth emphasizing that the proposed occlusion synthesis scheme can also be flexibly applied to other holistic vehicle datasets. Table 1 shows a comparison of Occ-Wild with other large-scale benchmark vehicle datasets. In summary, the benchmark dataset of this invention not only possesses a large number of realistic occlusion scenes but also contains rich pure foreground annotation information.

[0059] Table 1. Statistical comparison of Occ-Wild with other vehicle re-identification datasets

[0060]

[0061] 2. Pre-training methods

[0062] The core of this invention lies in the proposed pre-training details, including: model architecture, foreground-aware text prompts, pre-training methods, and model fine-tuning.

[0063] 2.1 Model Architecture

[0064] To adapt to complex outdoor occlusion scenarios, this invention proposes a unified multi-task vision-language pre-training strategy. The model architecture of the proposed method is as follows: Figure 3 As shown. First, the composite image is... Learnable background-normalized foreground (LBN) images (See Figure 4And formula (2)), view-level foreground annotation (or view mask image) and FTP (See section 2.2 foreground-aware text prompts below for details) as multimodal input. Among them, the three image-based inputs are segmented into multiple image patches. The images are then tokenized into a token sequence; subsequently, the image patches and text token sequences are processed by independent embedding layers to generate their respective embedding sequences. Finally, these sequence embeddings are processed using three core components: a single-modal image encoder derived from CLIP-B / 16. With a single-modal text encoder And a cross-modal image encoder consisting of 12 Transformer blocks with integrated cross-attention layers. The detailed processing procedure is as follows:

[0065] (1) Single-modal image encoder: Process separately , and From the embedded sequences, we obtain different types of features and feature sequences: The global features are ; All feature sequences are ; All feature sequences are and from Extract its global features With image block-level feature sequences .in Indicates the number of image blocks, take =256, This indicates the dimension of the image features, which is 768 dimensions in this case. Specifically, Used for visual-language contrastive learning (VCL) tasks, and with Together they are used for the foreground-enhanced knowledge distillation (FKD) task; Participate in the construction of foreground-aware text cues (FTP); and and It is then applied to masked visual modeling (MVM) tasks. As the optimization target of the entire pre-training process, its parameters are continuously updated throughout the training process and are eventually deployed to downstream tasks.

[0066] (2) Single-modal text encoder: The input is The output is The EOS features are denoted as ,in This indicates the dimension of the text features, which is 512 in this case. Specifically, Used for visual-language contrastive learning (VCL) tasks, and as The K and V matrices of the cross-attention layer are input to support the execution of the masked visual modeling (MVM) task. The parameters remain constant throughout the pre-training process.

[0067] (3) Multimodal image decoder: and and different, Each Transformer block integrates a cross-attention layer between the self-attention layer and the feedforward network layer, and this cross-attention layer has the same structure as the self-attention layer. It is worth noting that... The original input is from the pair and Apply masking operation (See Figure 3 ) to obtain its masked version; for Subsequent cross-attention layers, and The input to the matrix is For detailed training procedures, please refer to the Masked Visual Modeling (MVM) task. Accordingly, The parameters are continuously updated throughout the pre-training process to match the masked visual modeling (MVM) task objectives.

[0068] 2.2 Foreground-Aware Text Hints (FTP)

[0069] Existing research has proposed several learnable text cues (such as CoOp and CoCoOp), which extract text features of specific identities through global image-text alignment. However, directly using such text cues as text descriptions for occluded vehicle images is not optimal, mainly because global cross-modal alignment inevitably incorporates a large amount of occlusion noise, thus hindering the text representation from expressing the semantic meaning of the visible region's identity. Considering that the Occ-Wild dataset only contains occluded visual data, this step attempts to introduce an additional general text template to enhance the cross-modal representation capability of the pre-trained model in occluded scenarios. To this end, this invention designs a foreground-aware text cues (FTP) to achieve image-text semantic alignment in the foreground subject region, specifically as follows: Figure 4 As shown.

[0070] To reduce the impact of background-independent noise on FTP training, this invention employs LBN-foreground image... Specifically, by utilizing learnable vectors right The background area is filled. Compared with the traditional method of setting the background to 0, this design optimizes the background variable value through end-to-end training. This avoids the feature discontinuity problem at the vehicle-background boundary caused by setting the background to 0, and can actively normalize the background features to eliminate noise interference, guiding FTP to focus on learning the discriminative features of the vehicle body (such as color, outline and details). The specific expression is shown in formula (2):

[0071]

[0072] in This represents the Hadamah. Subsequently, Enter to To obtain global features .

[0073] In obtaining Afterwards, we designed a basic text prompt: "A photo of a "vehicle body." This represents the learnable token, and m represents the number of tokens, which is set to 4. Subsequently, the extracted LBN foreground features... Input to a multilayer perceptron with a Linear-ReLU-Linear structure In order to derive its semantic vector We then generate The four dungeons will each be connected to The corresponding learnable vectors in the image are fused to form the FTP, and the FTP is then used to synthesize the image. Text prompts Defined as "A photo of a "vehicle body". Finally, input the FTP into the frozen text encoder. In this process, the encoder extracts identity-related textual features. .

[0074] 2.3 Pre-training methods

[0075] Based on the above discussion, the pre-training process of this invention includes four types of inputs: synthetic image, LBN-foreground image, view mask image, and FTP. The pre-training architecture simultaneously optimizes three objectives, which combine three self-supervised paradigms (i.e., knowledge distillation, image-text contrast learning, and mask reconstruction) to facilitate the learning of a general representation suitable for occluded vehicle re-identification (Re-ID) tasks. The implementation details of each objective are as follows:

[0076] (1) Foreground Enhancement Knowledge Distillation (FKD): Considering that changes in the vehicle's viewpoint directly affect model performance in occlusion tasks, FKD utilizes feature-level knowledge distillation to transfer foreground region information with an unchanged viewpoint to the global region. Image block-level rotation is used to simulate changes in viewpoint orientation (see...). Figure 5 ).Depend on Extracted foreground subject feature sequence Include Each image patch is labeled with a [CLS] tag, and the spatial location of each image patch is treated as two-dimensional coordinates (x, y). Therefore, it can be reshaped using a reshaping operation. Each marker is converted into a two-dimensional matrix. To ensure semantic consistency of the identity features, only the foreground subject region is rotated at the image block level to obtain the rotated foreground subject features. It is worth noting that the marker retains its original position information during rotation, including the rotation angle. The interval was experimentally set as The proposed rotation rule can be expressed as formula (4):

[0077]

[0078] Subsequently, in global visual features Features of the foreground subject after rotation Distillation learning is performed between the two methods to gradually learn the feature invariance of the global image under different viewpoint changes, ultimately enhancing the foreground subject region information of the occluded image under viewpoint changes. The optimization objective of FKD can be defined as Equation (5):

[0079]

[0080] in This represents the knowledge distillation loss. This indicates the number of samples in a batch during training.

[0081] (2) Visual-Language Contrastive Learning (VCL): The second goal of pre-training is to utilize FTP as textual information and leverage CLIP's powerful image-text alignment capabilities to improve the performance of the visually occluded vehicle re-identification (Re-ID) task. This invention first generates learnable textual cues FTP using the foreground subject image, and then inputs it into a frozen text encoder. To extract text features Subsequently, by using global visual features 768-dimensional projection onto text features The 512 dimensions are aligned between the image and text to enhance the model's cross-modal understanding capabilities. Specifically, for each occluded vehicle instance and its corresponding text prompt, it is considered a positive sample pair, and all other sample pairs are considered negative sample pairs. This invention does not require building a queue for all positive and negative sample pairs, but only focuses on the sample pairs built in the current batch, thereby accelerating the training speed of the model. For each text-image pair, the similarity between them is normalized, and the specific expressions are shown in formulas (6) and (7):

[0082]

[0083]

[0084] Where τ is the temperature parameter; The expression used to calculate the similarity between image and text pairs is: .

[0085] This invention uses This represents the true one-hot labels, where negative sample pairs are set to 0 and positive sample pairs to 1. VCL uses the cross-entropy function. Optimize as shown in formula (8):

[0086]

[0087] in This represents the learning loss compared to the comparison.

[0088] (3) Masked Visual Modeling (MVM): Existing masked visual modeling methods aim to learn the latent feature information of images. However, since the reconstructed target inevitably contains occlusion noise, these methods learn redundant information unrelated to identity during the reconstruction process, making it difficult to generalize to occluded vehicle re-identification (Re-ID) tasks. Furthermore, low-level pixel reconstruction also struggles to learn robust feature representations. To address these bottlenecks, the MVM task utilizes existing view masks in the Occ-Wild dataset. Simultaneously, feature reconstruction and semantic reconstruction are performed: feature reconstruction targets the features of the foreground subject. Feature-level reconstruction predicts the features of the mask input region; semantic reconstruction, on the other hand, borrows from the mask visual modeling approach to analyze the view mask. Pixel reconstruction is performed on the four gray levels.

[0089] It is worth noting that this invention employs a random masking sampling strategy, masking approximately 75% of image blocks in the image features, and replacing the masked feature blocks with learnable mask markers. This is relevant in multimodal decoders. In each decoder block, text features are processed through a cross-attention layer. The multimodal input representation is obtained by fusing the mask features. To match the 512-dimensional text embedding, the image patch embedding is converted from 768-dimensional to 512-dimensional through a linear layer. For feature reconstruction and semantic reconstruction, mean squared error (MSE) is used as the optimization loss. The optimization objective of MVM is shown in Equation (9):

[0090]

[0091] in Indicates the mean square error loss. and These represent the mask sets in the two tasks respectively. and Size. and These respectively represent the characteristics of the foreground subject. The predicted results of feature reconstruction are compared with the true values; similarly, and These represent the view masks respectively. The predicted results of semantic reconstruction are compared with the true values.

[0092] Similar to the MAE method, this invention only calculates the loss of the masked image patch. Therefore, the total loss during the pre-training stage can be defined by formula (10):

[0093]

[0094] 2.4 Model Fine-tuning

[0095] To verify the performance of the proposed pre-trained model, a downstream occluded vehicle re-identification (Re-ID) dataset is essential. Therefore, this invention adopts the image generation strategy of Occ-Wild and constructs downstream occluded datasets Occ-VeRi776 and Occ-VehicleID for the VeRi-776 and VehicleID datasets, respectively. Specifically, maintaining the same number of images as the original training set, 50% occluded data is synthesized, ensuring that each identity contains at least one occluded image. Furthermore, drawing on the general settings of occluded pedestrian datasets, this invention defines two task settings in the test set: "occluded" and "partial". In the occluded task setting, the query set consists of 100% occluded data, the image library contains 20% occluded data, and it is guaranteed that each ID in the image library has at least one occluded image. In the partial task setting, the query image is cropped from the synthesized occluded image and the complete image, retaining only the features of the unoccluded vehicle components, and a minimum threshold is set for the image size to ensure effective visual features.

[0096] This invention follows the standard Re-ID task protocol: image encoder From the image Extract global features Then add a linear layer Project the global feature to generate the ID prediction probability. For downstream tasks, this invention does not introduce any additional techniques, only retaining the cross-entropy loss. With triplet loss The Re-ID model is optimized. During the inference phase, global visual features of the image are used for matching and retrieval.

[0097] Example 2

[0098] like Figure 6 The diagram shown is a structural schematic of an occluded vehicle re-identification system based on multi-task joint pre-training, as proposed in the second embodiment of this application. The system includes the following key modules:

[0099] Data acquisition module 100 is used to acquire vehicle images to be processed;

[0100] The joint pre-training module 200 is used to generate a pre-trained model based on the vehicle image through a unified multi-task joint pre-training framework. The joint pre-training module includes:

[0101] The foreground enhancement knowledge distillation unit 210 is used to extract the global visual features and foreground subject features of the vehicle image using the single-modal image encoder; perform image block-level rotation operation on the foreground subject features to generate rotated foreground subject features; and transfer the viewpoint invariance information contained in the rotated foreground subject features to the global visual features through knowledge distillation.

[0102] The visual-language contrastive learning unit 220 is used to generate a foreground-aware text prompt aligned with the visible foreground region of the vehicle based on the foreground information of the vehicle image; extract the text features of the foreground-aware text prompt using the unimodal text encoder; and perform contrastive learning on the global visual features and the text features to enhance the representation of key information of the vehicle foreground by the global visual features.

[0103] The masking visual modeling unit 230 is used to perform masking operations on image blocks of the vehicle image to obtain masked image features; the masked image features and the text features are input to the multimodal image decoder to reconstruct the masked foreground subject features and the vehicle view mask corresponding to the vehicle image in parallel.

[0104] The vehicle re-identification module 300 is used to perform re-identification processing on target occluded vehicles based on the optimized single-modal image encoder in the pre-trained model.

[0105] An occluded vehicle re-identification system based on multi-task joint pre-training, as described in this application, can be a device, a component, integrated circuit, or chip in a terminal. The device can be a mobile electronic device or a non-mobile electronic device. For example, mobile electronic devices can be mobile phones, tablets, laptops, PDAs, in-vehicle electronic devices, wearable devices, ultra-mobile personal computers (UMPCs), netbooks, or personal digital assistants (PDAs), etc., while non-mobile electronic devices can be servers, network-attached storage (NAS), personal computers (PCs), etc. This application does not impose specific limitations.

[0106] The occluded vehicle re-identification system based on multi-task joint pre-training in this application embodiment can be a device with an operating system. This operating system can be Android, iOS, or other possible operating systems; this application embodiment does not specifically limit it.

[0107] The occluded vehicle re-identification system based on multi-task joint pre-training provided in this application embodiment can achieve... Figure 1 The various processes implemented in the method embodiment of the occluded vehicle re-identification method based on multi-task joint pre-training are not described in detail here to avoid repetition.

[0108] Optionally, embodiments of this application also provide an electronic device, including a processor, a memory, and a program or instructions stored in the memory and executable on the processor. When the program or instructions are executed by the processor, they implement the various processes of the above-described embodiment of a multi-task joint pre-training method for re-identifying occluded vehicles and achieve the same technical effect. To avoid repetition, they will not be described again here.

[0109] This application also provides a readable storage medium storing a program or instructions. When the program or instructions are executed by a processor, they implement the various processes of the above-described embodiment of a multi-task joint pre-training method for occluded vehicle re-identification and achieve the same technical effect. To avoid repetition, they will not be described again here.

[0110] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0111] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0112] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0113] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. An occluded vehicle re-identification method based on multi-task joint pre-training, characterized in that, Includes the following steps: Before acquiring the vehicle images to be processed, the process also includes pre-training data preparation: constructing an obstacle instance library, which contains obstacle images and corresponding instance segmentation masks in various traffic monitoring scenarios, and labeling each obstacle instance with a priori position index and a priori view index; based on a complete vehicle dataset, according to the view information of the vehicle images and the position and view prior index of the obstacle instances, obstacles are pasted at reasonable positions in the vehicle images to synthesize occluded vehicle images. The process of generating foreground annotations includes: extracting an initial foreground region mask from the occluded vehicle image using a vehicle parsing model; obtaining a precise location mask of the obstacle during synthesis; and removing regions that overlap with the precise location mask of the obstacle from the initial foreground region mask to obtain a clean foreground mask containing only the visible portion of the vehicle. The clean foreground mask is used to generate foreground subject features and a vehicle viewpoint mask. Based on the vehicle images, a pre-trained model is generated through a unified multi-task joint pre-training framework. The pre-trained model consists of a unimodal image encoder, a unimodal text encoder, and a multimodal image decoder. The pre-training framework executes the following steps in parallel: The foreground enhancement knowledge distillation step includes: extracting global visual features and foreground subject features of the vehicle image using the unimodal image encoder; performing an image block-level rotation operation on the foreground subject features to generate rotated foreground subject features; and transferring the viewpoint invariant information contained in the rotated foreground subject features to the global visual features through knowledge distillation. This includes: reshaping multiple image block labels of the foreground subject features into a two-dimensional matrix; performing a rotation operation on the two-dimensional matrix to generate the rotated foreground subject features, wherein the classification labels retain their original position information during rotation, and the rotation angle is within a preset range; calculating the feature distribution difference between the global visual features and the rotated foreground subject features as the knowledge distillation loss, and updating the parameters of the unimodal image encoder based on the knowledge distillation loss. The visual-language contrastive learning step includes: generating a foreground-aware text prompt aligned with the visible foreground region of the vehicle based on the foreground information of the vehicle image; specifically, this includes: generating a learnable background-normalized foreground image based on the clean foreground mask, wherein the background region of the vehicle image is filled with learnable background vectors; inputting the learnable background-normalized foreground image to the unimodal image encoder to extract its global foreground features; inputting the global foreground features to a multilayer perceptron to generate multiple foreground subject semantic vectors; fusing the multiple foreground subject semantic vectors with learnable tags in an initial text prompt template containing multiple learnable tags to generate the final foreground-aware text prompt; extracting the text features of the foreground-aware text prompt using the unimodal text encoder; and performing contrastive learning on the global visual features and the text features to enhance the representation of key foreground information of the vehicle by the global visual features. The masking visual modeling step includes: performing a masking operation on the image blocks of the vehicle image to obtain masked image features; inputting the masked image features and the text features into the multimodal image decoder to reconstruct the masked foreground subject features and the vehicle view mask corresponding to the vehicle image in parallel. Based on the optimized single-modal image encoder in the pre-trained model, the target occluded vehicle is re-identified.

2. The method according to claim 1, characterized in that, Perform the masked visual modeling steps, including: Random mask sampling is performed on the image patch features of the vehicle image, and the masked image patches are replaced with learnable mask markers; The mask image features are input to the multimodal image decoder. Each decoder block of the multimodal image decoder contains a cross-attention layer. The cross-attention layer takes the text features as key and value inputs and performs information fusion with the mask image features. Using the output of the multimodal image decoder, the first mean square error loss between the reconstructed foreground subject features and the real foreground subject features, and the second mean square error loss between the reconstructed vehicle view mask and the real vehicle view mask are calculated respectively. The parameters of the multimodal image decoder and the single-modal image encoder are updated according to the first and second mean square error losses.

3. The method according to claim 1, characterized in that, The method further includes: The overall loss function of the pre-trained framework is constructed as follows: knowledge distillation loss for performing the foreground enhancement knowledge distillation step, contrastive learning loss for performing the vision-language contrastive learning step, and a weighted sum of the first mean square error loss and the second mean square error loss for performing the mask visual modeling step. The parameters of the unimodal text encoder remain fixed during pre-training; the parameters of the unimodal image encoder and the multimodal image decoder are updated end-to-end according to the overall loss function.

4. The method according to claim 1, characterized in that, The steps for re-identifying occluded vehicles based on the optimized single-modal image encoder in the pre-trained model include: The optimized single-modal image encoder in the pre-trained model is used as the backbone network for fine-tuning in downstream tasks. A linear projection layer is added after the backbone network to generate identity prediction probabilities; The backbone network and the linear projection layer are jointly optimized using cross-entropy loss and triplet loss. During the inference phase, global visual features of the target vehicle image are extracted, and similarity matching and retrieval are performed based on the global visual features to complete the re-identification.

5. An occluded vehicle re-identification system based on multi-task joint pre-training, characterized in that, include: The data acquisition module is used to prepare pre-training data before acquiring the vehicle images to be processed: constructing an obstacle instance library, which contains obstacle images and corresponding instance segmentation masks in various traffic monitoring scenarios, and annotating the position prior index and view prior index for each obstacle instance; based on a complete vehicle dataset, according to the view information of the vehicle image and the position and view prior index of the obstacle instance, pasting obstacles at reasonable positions in the vehicle image to synthesize occluded vehicle images; The process of generating foreground annotations includes: extracting an initial foreground region mask from the occluded vehicle image using a vehicle parsing model; obtaining a precise location mask of the obstacle during synthesis; and removing regions that overlap with the precise location mask of the obstacle from the initial foreground region mask to obtain a clean foreground mask containing only the visible portion of the vehicle. The clean foreground mask is used to generate foreground subject features and a vehicle viewpoint mask. A joint pre-training module is used to generate a pre-trained model based on the vehicle image through a unified multi-task joint pre-training framework. The pre-trained model consists of a single-modal image encoder, a single-modal text encoder, and a multi-modal image decoder. The joint pre-training module includes: A foreground enhancement knowledge distillation unit is used to extract global visual features and foreground subject features of the vehicle image using the single-modal image encoder; perform image block-level rotation operation on the foreground subject features to generate rotated foreground subject features; and transfer the viewpoint invariant information contained in the rotated foreground subject features to the global visual features through knowledge distillation; including: reshaping multiple image block labels of the foreground subject features into a two-dimensional matrix; performing a rotation operation on the two-dimensional matrix to generate the rotated foreground subject features, wherein the classification labels retain their original position information during rotation, and the rotation angle is within a preset range; calculating the feature distribution difference between the global visual features and the rotated foreground subject features as the knowledge distillation loss, and updating the parameters of the single-modal image encoder according to the knowledge distillation loss; A visual-language contrastive learning unit is used to generate a foreground-aware text prompt aligned with the visible foreground region of the vehicle based on the foreground information of the vehicle image. Specifically, this includes: generating a learnable background-normalized foreground image based on the clean foreground mask, wherein the background region of the vehicle image is filled with learnable background vectors; inputting the learnable background-normalized foreground image to the unimodal image encoder to extract its global foreground features; inputting the global foreground features to a multilayer perceptron to generate multiple foreground subject semantic vectors; fusing the multiple foreground subject semantic vectors with learnable tags in an initial text prompt template containing multiple learnable tags to generate the final foreground-aware text prompt; extracting the text features of the foreground-aware text prompt using the unimodal text encoder; and performing contrastive learning on the global visual features and the text features to enhance the representation of key foreground information of the vehicle by the global visual features. A masking visual modeling unit is used to perform masking operations on image blocks of the vehicle image to obtain masked image features; the masked image features and the text features are input to the multimodal image decoder to reconstruct the masked foreground subject features and the vehicle view mask corresponding to the vehicle image in parallel. The vehicle re-identification module is used to re-identify occluded vehicles based on the optimized single-modal image encoder in the pre-trained model.

6. An electronic device, characterized in that, It includes a processor, a memory, and a program or instructions stored in the memory and executable on the processor. When the program or instructions are executed by the processor, they implement the steps of the occluded vehicle re-identification method based on multi-task joint pre-training as described in any one of claims 1-4.

7. A readable storage medium, characterized in that, The readable storage medium stores a program or instructions, which, when executed by a processor, implement the steps of the occluded vehicle re-identification method based on multi-task joint pre-training as described in any one of claims 1-4.

Images