Artificial intelligence visible light infrared mode pedestrian re-identification method

Abstract

The invention discloses an artificial intelligence visible light infrared modal pedestrian re-identification method, which belongs to the field of computer vision and deep learning, and constructs an end-to-end basic-detail feature learning framework, and losslessly captures modal exclusive detail features through a detail feature extraction module. And a basic embedding generation module is utilized to generate modal sharing basic features. According to the method, lossless feature extraction is realized by adopting the reversible neural network, feature interaction is promoted by combining a cross-modal joint processing strategy with a cross attention Transform module, and meanwhile, an exclusive-shared knowledge distillation loss function is designed to optimize cross-modal feature correlation, so that the cross-modal feature fusion effect and the model discrimination capability are remarkably improved, and the method has the advantages of being high in robustness and high in robustness. The calculation complexity is reduced while the precision is ensured, and the method has important application value for all-weather monitoring requirements in the fields of smart cities, public security and the like.

Description

Technical Field

[0001] The present invention relates to the field of computer vision and deep learning technologies, and specifically to an artificial intelligence visible light infrared modality pedestrian re-identification method. Background Art

[0002] Person re-identification (ReID) aims to retrieve the identity of a target from a library of images captured by different cameras. In recent years, it has made significant progress in the fields of security and public surveillance. As urban security systems evolve towards all-weather intelligent systems, traditional single-modal ReID systems based on visible light modalities have shown significant limitations in complex environments such as low illumination and backlighting. To address this challenge, the visible-infrared person re-identification (VI-ReID) method has emerged. This method can match IR (or RGB) images given a specific RGB (or IR) pedestrian image, adapting to 24-hour, all-weather scenarios that also include IR images captured by infrared (IR) cameras. Currently, cross-modal ReID technology has become a research hotspot in the field of computer vision.

[0003] Existing research on VI-ReID can be summarized into two main methods: The first is to extract discriminative modality shared features: Method Description: Reduce modality differences by aligning cross-modal features into a common semantic space.

[0004] Disadvantages: Failure to effectively utilize modality specificity and shared cues, resulting in performance bottlenecks.

[0005] The second is to compensate for modality-specific or modality-shared features: Method description: It can be divided into embedding-level and image-level methods, which respectively generate compensation knowledge at the embedding space and pixel levels through auxiliary models such as generative adversarial networks (GANs), segmentation networks, and part alignment networks.

[0006] shortcoming: Embedding-level methods: may introduce loss and noise to the generated features.

[0007] Image-level methods: They rely on additional data processing from other models, which limits their efficiency and practicality.

[0008] In addition, feature alignment methods aim to transform cross-modal features into a unified semantic space by using metric learning techniques or enhancing the feature extraction components of the network. However, these methods will eventually encounter performance bottlenecks due to the loss of modality-specific information.

[0009] Methods based on auxiliary information generate enhanced recognition knowledge through other models, such as: GAN series methods: Generate compensatory features at the image level or embedding level to simulate the features of another modality.

[0010] XIV method: Generate X modality through a lightweight auxiliary network to reduce modality differences.

[0011] LUPI method: constructs an intermediate domain between visible light and infrared, and generates images in this domain to guide the network to obtain more discriminative information.

[0012] SGIEL method: innovatively utilizes the identity shape knowledge generated by the segmentation model to enrich auxiliary information.

[0013] TMD method: Reduce image-level differences by generating style-aligned images, and then align feature distribution and instance features.

[0014] These methods either inevitably introduce information distortion during the generation process or fail to fully capture modality-specific and shared information. Summary of the Invention

[0015] In response to the shortcomings of the existing technology, the present invention provides an artificial intelligence visible light infrared modality pedestrian re-identification method, which solves the problems raised by the above background technology.

[0016] To achieve the above objectives, the present invention is implemented through the following technical solutions: an artificial intelligence visible light infrared modality pedestrian re-identification method, comprising: Build an end-to-end basic-detail feature learning framework, losslessly extract modality-specific detail features through the detail feature extraction module, and generate modality-shared basic features using the basic embedding generation module; Based on the exclusive-shared knowledge distillation loss function, the cross-modal feature correlation is optimized, and the detailed features and basic features are jointly trained to finally form a discriminative comprehensive feature for pedestrian re-identification.

[0017] Furthermore, the detail feature extraction module adopts a reversible neural network block group to realize lossless detail feature extraction, and embeds and splices the detail features of visible light and infrared modalities through a cross-modal joint processing strategy and inputs them into the cross-attention based Transformer module to promote cross-modal feature interaction and information fusion.

[0018] Furthermore, the basic embedding generation module projects the output features of the backbone network into the detail feature subspace and the basic feature subspace through a parameterizable projection matrix to generate independent basic embedding features; and fuses the basic features of the two modalities through a cross-modal feature fusion method to generate auxiliary features, ensuring that the basic features only contain modality shared knowledge.

[0019] Furthermore, the exclusive-shared knowledge distillation loss function enhances the discrimination between modality shared knowledge and modality exclusive knowledge by optimizing the correlation between basic features and detail features of visible light and infrared modalities. Its expression is:

[0020] in, represents the basic features generated by the proposed basic embedding generation module; represents the detail features extracted from the proposed detail feature extraction module; is the Pearson correlation coefficient operation; is a constant used to ensure that the denominator is non-zero; By optimizing the exclusive-shared knowledge distillation loss function, the correlation between the basic features and detail features of the visible light and infrared modalities is calculated as: in and are simultaneously enhanced and inhibited.

[0021] Furthermore, the base-detail feature learning framework imposes semantic space independence constraints on detail features and base features during the training phase, ensuring that detail features only carry modality-specific information and base features only contain modality-shared knowledge by minimizing mutual information loss.

[0022] Furthermore, the inference stage only uses the comprehensive features output by the backbone network for performance evaluation, and the comprehensive features enhance the discrimination capability by fusing the detail features extracted by the detail feature extraction module and the basic features generated by the basic embedding generation module.

[0023] Furthermore, the detail feature extraction module adopts channel dimension feature concatenation and cross-modal cross-attention operation after feature extraction to reduce computational complexity and focus on distinguishability information.

[0024] Furthermore, the basic embedding generation module generates basic features through feature space approximation, semantic representation alignment and cross-modal correlation constraints, wherein the feature space approximation is achieved by forcing the distance distribution of detailed features and basic features to converge.

[0025] Furthermore, in the SKD loss function, the basic feature correlation of visible light and infrared modalities is suppressed, and the detail feature correlation is enhanced, so as to strengthen the expression capabilities of modality shared knowledge and modality-specific knowledge respectively.

[0026] Furthermore, the backbone network adopts a single-stream ResNet-50 architecture, in which the intermediate features output by some layers are input into the DFE module, and the features after global pooling are input into the BEG module, and finally the prediction probability is output through the detail feature classifier and the basic feature classifier.

[0027] The present invention provides an artificial intelligence-based visible light infrared modality pedestrian re-identification method. Compared with the existing technology, it has the following advantages: 1. Through an end-to-end foundation-detail feature learning framework, this invention can simultaneously extract modality-specific detail features and modality-shared foundational features, effectively fusing information from visible and infrared modalities. This fusion strategy significantly improves the model's recognition accuracy in complex environments, particularly in low-light and backlit scenarios where traditional single-modality systems perform poorly.

[0028] 2. The Detail Feature Extraction (DFE) module utilizes a reversible neural network block group to achieve lossless detail feature extraction, preserving modality-specific details such as texture and color. Simultaneously, the Basic Embedding Generation (BEG) module projects features into both detail feature subspaces and basic feature subspaces using a parameterizable projection matrix, generating independent basic embeddings. This ensures that the basic features contain only modality-shared knowledge, such as contours and motion features. This dual-path feature extraction approach significantly enhances the richness and discriminative power of feature representation.

[0029] 3. By optimizing the network structure and loss function, computational complexity is reduced and efficiency is improved. The detail feature extraction module adopts a cross-modal joint processing strategy, embedding and splicing detail features from the visible and infrared modalities into a cross-attention-based Transformer module. This promotes cross-modal feature interaction and information fusion while reducing computational complexity. Furthermore, the design of the proprietary-shared knowledge distillation (SKD) loss function optimizes cross-modal feature correlation, further improving the model's inference speed.

[0030] 4. Through cross-modal feature interaction and information fusion, the present invention enhances the robustness and generalization ability of the model in different scenarios. The fusion of detailed features and basic features enables the model to capture both modality-specific and modality-shared information, thereby better coping with cross-scenario and cross-modal recognition tasks. In addition, by constraining the distribution of classification results of detailed features and comprehensive features to remain consistent, the present invention ensures the consistency of semantic representation and further improves the generalization ability of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 Flowchart of the present invention.

[0032] In the figure: 1. ResNet50 backbone network; 2. Basic embedding generation module; 3. Detail feature extraction module; 4. Reversible neural network module; 5. Layer normalization operation; 6. Channel dimension cascade operation; 7. Cross attention operation; 8. Intermediate features; 9. Extracted detail features; 10. Backbone network total output features; 11. Parameterizable projection matrix; 12. Basic embedding generation module generates detail features; 13. Basic embedding generation module generates basic features; 14. Batch dimension attention map; 15. Matrix bitwise multiplication operation; 16. Matrix bitwise addition operation; 17. Channel dimension attention map; 18. Batch attention enhances detail features; 19. Basic embedding generation module generates fusion features; 20. Detail feature space; 21. Basic feature space; 22. Detail feature classifier; 23. Basic feature classifier; 24. Network loss function; 25. Global average pooling operation. DETAILED DESCRIPTION

[0033] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0034] See also Figure 1 The present invention provides a technical solution: an artificial intelligence visible light infrared modality pedestrian re-identification method, comprising: Build an end-to-end basic-detail feature learning framework, losslessly extract modality-specific detail features through the detail feature extraction module (DFE), and generate modality-shared basic features using the basic embedding generation module (BEG); Based on the exclusive-shared knowledge distillation (SKD) loss function, the correlation of cross-modal features is optimized, and detailed features and basic features are jointly trained to finally form discriminative comprehensive features for pedestrian re-identification.

[0035] Specifically, the detail feature extraction module uses a reversible neural network block group to achieve lossless detail feature extraction. It also embeds and splices the detail features of the visible light and infrared modalities through a cross-modal joint processing strategy and then inputs them into the cross-attention-based Transformer module to promote cross-modal feature interaction and information fusion. The details are as follows: The detail feature extraction module (DFE) aims to obtain detail features containing modality-specific information from intermediate embedded features by adopting an invertible neural network (INN) block group. This design can effectively preserve detail characteristics and alleviate information loss in feature extraction through the reciprocal generation mechanism of input and output embeddings. In the final stage of the DFE module, a cross-modal joint processing strategy is adopted to reduce computational complexity by splicing the detail feature embeddings extracted from the two modalities. Specifically, the extracted detail feature embeddings are input into the cross-attention-based Transformer module to promote cross-modal feature reasoning and information interaction. This mechanism can not only integrate multimodal detail feature knowledge, but also more effectively focus on distinguishability information, thereby enhancing the robustness and effectiveness of semantic representation. The extracted detail features It can significantly enhance the modality-specific detail knowledge, so it is specifically designed for detail features. Built a dedicated classifier , and designed a shared classifier To process the basic embedding features and the obtained comprehensive features In addition, a commonly used identity loss function based on cross entropy is used to enhance detail features. Finally, the detail features and comprehensive characteristics There are differences in the distribution of classification results, and this inconsistency may hinder the use of detailed knowledge to enhance Therefore, the constraint The predicted probability distribution is The distribution of is kept consistent, thus ensuring the consistency of the semantic representation of the two.

[0036] Specifically, the basic embedding generation module projects the backbone network output features into the detail feature subspace and the basic feature subspace through a parameterizable projection matrix to generate independent basic embedding features. It then fuses the basic features of the two modalities through a cross-modal feature fusion method to generate auxiliary features, ensuring that the basic features only contain modality-shared knowledge, as follows: The base embedding generation module (BEG) aims to utilize the acquired detailed features from Generate basic embedding features in . It should be noted that there is a significant semantic difference between the detail information that is specific to the modality, such as color and texture, and the basic information that is shared by the modalities, such as motion and contour. To this end, a method is developed to ensure that the detail (basic) features only contain distinguishable knowledge that is specific to the modality (shared by the modality), thereby maximizing the collection of these two types of information. In addition, the proposed DFE and BEG modules can synchronously learn these two types of knowledge without interfering with each other. Based on this premise, the detail embedding and the basic embedding are independent of each other, that is, By making Fully approaching , and impose independence constraints between details and basic embeddings, the proposed BEG module can exclude , and then generate the basic embedding shared by the modalities.

[0037] The approximation process can be divided into three components: feature space approximation, semantic representation alignment, and and In terms of feature space approximation, we first calculate the mini-batch samples and The distance between all embedded features and obtain the difference map, thus forcing The distance distribution to Convergence. In addition, by adjusting The predicted probability distribution of The distribution of , realizes the alignment of the semantic representation of the two. Considering the detailed features generated by the BEG module Should Having the same correlation characteristics, it achieves and The correlation is consistent.

[0038] Based on the above processing, Eliminate detail features To generate basic features Given that cross-modal basic information (such as contour and motion features) should have high similarity, a cross-modal feature fusion method is constructed to generate auxiliary features by fusing the basic features of the two modalities. , thus ensuring Only contains the shared knowledge between modalities. Implementing cross-modal semantic alignment to enhance the collection of modality-shared knowledge.

[0039] About the exclusive-shared knowledge distillation (SKD) loss function: The visible light (VIS) and infrared (IR) modalities share significant similarities in basic information, such as contours and motion, while the similarities in specific information, such as color, texture, and thermodynamic details, are suppressed. By respectively enhancing and weakening the correlation between the two modalities, basic and detailed features can be generated. Based on this, we propose a new dedicated-shared knowledge distillation (SKD) loss function that is numerically smoother and easier to optimize. Its expression is as follows:

[0040] in, represents the basic features generated by the proposed BEG module, represents the detail features extracted from the proposed DFE module. is the Pearson correlation coefficient, and is a constant used to ensure that the denominator is non-zero. By optimizing the SKD loss function, the correlation between the basic features and detail features of the visible light (VIS) and infrared (IR) modalities (i.e., the formula ) and This method enables the proposed detail feature extraction module (DFE) to extract embedding features rich in detail knowledge, while the proposed base embedding generation (BEG) module is able to generate base embedding features that contain more modality shared knowledge.

[0041] In addition, the basic-detail feature learning framework, training and inference steps are as follows: (1) Use the single-stream ResNet-50 network 1 as the backbone network, and input the image into the neural network after preprocessing to obtain the intermediate layer feature output And the total network output .

[0042] (2) The intermediate features output by some layers of the backbone network 8 Input the proposed detail feature extraction module 3, the feature extraction network constructed by the reversible neural network module 4 combined with the layer normalization operation 5, and after the channel dimension feature cascade 6, the cross-modal cross attention operation 7 is performed to generate detail features ; (3) At the same time, the features output by the backbone network 10 is processed by the global pooling operation 25 and input into the base embedding generation module 2, which is transformed by the parameterizable projection matrix 11 10 is projected into two mutually orthogonal subspaces, namely detail feature subspace 20 and basic feature subspace 21. The features generated by the projection are the detail features generated by the basic embedding generation module 12. Basic features generated by the basic embedding generation module 13.

[0043] (4) Through the cross-modal feature fusion method, the basic embedding generation module generated by the proposed basic embedding generation module is used to generate basic features 13, and the batch dimension attention map 14 and the channel dimension attention map 17 are calculated. Combined with the matrix bitwise multiplication operation 15 and the matrix bitwise addition operation 16, the detailed features enhanced by the batch dimension attention are generated. 18. Finally, construct the modal shared features 19. Supplement basic features in the best way.

[0044] (5) The detailed features extracted above 9. Detailed features generated by the basic embedding generation module 12. Basic features generated by the basic embedding generation module 13 and the fusion features generated by the basic embedding generation module 19 are input into the corresponding detail feature classifier 22 and basic feature classifier 23. Output the predicted probability.

[0045] (6) Use the above output probabilities to calculate the loss function required by the network 24, which includes the proposed novel modality-specific-shared knowledge distillation (SKD) loss function to ensure that the generated detail (basic) features contain modality-specific (modality-shared) knowledge as much as possible, thereby effectively utilizing modality-specific and shared information.

[0046] (7) In the inference phase, only the comprehensive features output by the backbone network are used Performance evaluation is performed because the proposed detail feature extraction module and basic embedding generation module effectively enhance the discriminative ability of comprehensive features by incorporating additional details and basic information.

[0047] In summary, this method constructs an end-to-end base-detail feature learning framework. The detail feature extraction module (DFE) losslessly captures modality-specific detail features, such as texture and color differences, while the base embedding generation module (BEG) generates modality-shared base features, such as contour and motion features. The DFE module uses a reversible neural network block group to achieve lossless detail feature extraction. A cross-modal joint processing strategy, combined with a cross-attention Transformer module, promotes cross-modal feature interaction and information fusion. The BEG module projects features into detail feature subspaces and base feature subspaces using a parameterizable projection matrix, generating independent base embedding features that ensure the base features contain only modality-shared knowledge. To optimize cross-modal feature relevance, the method designs a dedicated-shared knowledge distillation (SKD) loss function that generates base and detail features by enhancing and weakening the correlation between the two modalities. In implementation, a single-stream ResNet-50 network is used as the backbone network. The DFE and BEG modules extract detail and base features, which are then fused and fed into a classifier, where a loss function is calculated for optimization. During the inference phase, only the comprehensive features output by the backbone network are used for performance evaluation. This method significantly improves cross-modal feature fusion and model discrimination capabilities, while reducing computational complexity and enhancing the model's robustness and generalization capabilities across different scenarios. It has important application value for all-weather monitoring needs in smart cities, public security, and other fields.

Claims

1. An artificial intelligence visible light infrared modality pedestrian re-identification method, characterized by: include: Build an end-to-end basic-detail feature learning framework, losslessly extract modality-specific detail features through the detail feature extraction module, and generate modality-shared basic features using the basic embedding generation module; Based on the exclusive-shared knowledge distillation loss function, the cross-modal feature correlation is optimized, and the detailed features and basic features are jointly trained to finally form a discriminative comprehensive feature for pedestrian re-identification.

2. The artificial intelligence visible light infrared modality pedestrian re-identification method according to claim 1 is characterized in that: The detail feature extraction module adopts a reversible neural network block group to realize lossless detail feature extraction, and embeds and splices the detail features of visible light and infrared modalities through a cross-modal joint processing strategy and inputs them into a cross-attention based Transformer module to promote cross-modal feature interaction and information fusion.

3. The artificial intelligence visible light infrared modality pedestrian re-identification method according to claim 1, characterized in that: The basic embedding generation module projects the backbone network output features into the detail feature subspace and the basic feature subspace through a parameterizable projection matrix to generate independent basic embedding features; The basic features of the two modalities are fused through a cross-modal feature fusion method to generate auxiliary features, ensuring that the basic features only contain modality shared knowledge.

4. The artificial intelligence visible light infrared modality pedestrian re-identification method according to claim 1, characterized in that: The exclusive-shared knowledge distillation loss function optimizes the correlation between basic features and detailed features of visible light and infrared modalities, thereby enhancing the discrimination between modality shared knowledge and modality exclusive knowledge. Its expression is: ， in, represents the basic features generated by the proposed basic embedding generation module; represents the detail features extracted from the proposed detail feature extraction module; is the Pearson correlation coefficient operation; is a constant used to ensure that the denominator is non-zero; By optimizing the exclusive-shared knowledge distillation loss function, the correlation between the basic features and detail features of the visible light and infrared modalities is calculated as: in and are simultaneously enhanced and inhibited.

5. The artificial intelligence visible light infrared modality pedestrian re-identification method according to claim 1 is characterized in that: The base-detail feature learning framework imposes semantic space independence constraints on detail features and base features during the training phase, and ensures that detail features only carry modality-specific information and base features only contain modality-shared knowledge by minimizing mutual information loss.

6. The artificial intelligence visible light infrared modality pedestrian re-identification method according to claim 3 is characterized in that: The inference stage only uses the comprehensive features output by the backbone network for performance evaluation, and the comprehensive features enhance the discrimination capability by fusing the detail features extracted by the detail feature extraction module with the basic features generated by the basic embedding generation module.

7. The artificial intelligence visible light infrared modality pedestrian re-identification method according to claim 2, characterized in that: The detail feature extraction module adopts channel dimension feature concatenation and cross-modal cross attention operation after feature extraction to reduce computational complexity and focus on distinguishability information.

8. The artificial intelligence visible light infrared modality pedestrian re-identification method according to claim 3 is characterized in that: The basic embedding generation module generates basic features through feature space approximation, semantic representation alignment and cross-modal correlation constraints, wherein feature space approximation is achieved by forcing the distance distribution of detailed features and basic features to converge.

9. The artificial intelligence visible light infrared modality pedestrian re-identification method according to claim 4, characterized in that: In the SKD loss function, the basic feature correlation of visible light and infrared modalities is suppressed, and the detail feature correlation is enhanced, so as to strengthen the expression capabilities of modality shared knowledge and modality-specific knowledge respectively.

10. The artificial intelligence visible light infrared modality pedestrian re-identification method according to claim 6, characterized in that: The backbone network adopts a single-stream ResNet-50 architecture. The intermediate features output by some layers are input into the DFE module, and the features after global pooling are input into the BEG module. Finally, the prediction probability is output through the detail feature classifier and the basic feature classifier.

Images