Domain generalization personnel re-identification method and system based on multi-modal fusion and structure perception enhancement

Abstract

The invention discloses a domain generalization personnel re-identification method and system based on multi-modal fusion and structure perception enhancement. According to the method, a grey-scale map mode is innovatively introduced to extract biological characteristic information irrelevant to dressing, so that the identification limitation of a visible light image under the conditions of uniform shielding and severe illumination is effectively made up; meanwhile, a structure perception data enhancement strategy is provided, a key identification area is protected through a semantic segmentation technology, and damage to effective features is avoided while data diversity is improved; besides, by introducing targeted alignment loss, uniformity loss and intra-domain uniformity loss, the distribution characteristics of the feature space are directly optimized and regularized, and the generalization performance in an unknown substation scene is significantly improved. According to the method, superior detection performance can be realized under the condition of limited annotation data, the dependence on large-scale annotation data is effectively reduced, and the deployment cost is reduced.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of computer vision and deep learning, and in particular to a domain generalization person re-identification method and system based on multi-modal fusion and structure perception enhancement. BACKGROUND

[0002] In the power scene, the person re-identification technology faces unique and severe challenges. The core difficulty lies in the extreme similarity of appearance features and the extreme complexity of the application environment. First, all workers wear standard uniforms, which makes the most distinctive appearance identification features such as clothing color, style, and texture almost completely ineffective for traditional re-identification systems. The system must rely on more subtle and difficult-to-capture biological features such as body shape, gait, and face. Second, the work environment is complex and variable, including strong light contrast in the substation, equipment occlusion, rainwater reflection, and imaging differences from different perspectives such as fixed cameras, drones, and other factors that greatly interfere with feature extraction and matching accuracy. The most fundamental challenge is the domain difference. A power system usually contains multiple substations, each with significant differences in layout, lighting, and camera models. A model trained on data collected at one station (source domain) performs very well, but when deployed directly to another unseen station (target domain), the performance will suffer a catastrophic decline due to the inevitable mixing of "false associations" such as background and lighting specific to the station learned by the model. SUMMARY

[0003] The technical problem to be solved by the present application is to overcome the problem of excessive dependence on appearance features and lack of generalization ability in the prior art. The present application provides a domain generalization person re-identification method and system based on multi-modal fusion and structure perception enhancement, which discards the excessive dependence on single and fragile appearance features and instead learns more essential and robust domain-invariant identity features through multi-modal information fusion and structured regularization constraints, thereby achieving true generalization ability.

[0004] To this end, the present application adopts the following technical solutions.

[0005] In a first aspect, the present application provides a domain generalization person re-identification method based on multi-modal fusion and structure perception enhancement, which includes the following steps:

[0006] 1) Multi-modal data input and preprocessing, the multi-modal data including visible light images and grayscale images; 2) Only the visible light images are processed, a pre-trained semantic segmentation network is used to generate a structure mask, which identifies the key identification area that needs to be protected. Random enhancement operations are applied to the background area outside the mask to obtain a visible light enhanced image; 3) Two independent pre-trained convolutional neural networks are used to process visible light images and grayscale images respectively, and their unique features are extracted from the two modalities of data respectively; the convolutional neural network used to extract visible light image features shares weights, processes the visible light enhancement image and extracts features, so that the convolutional neural network learns to be invariant to enhancement changes; 4) An attention mechanism is introduced to fuse the features of the visible light image and the grayscale image, as well as the features of the visible light enhanced image and the grayscale image, respectively. The adaptive weights of each are calculated to dynamically weigh the importance of the two modalities in a specific scene, and finally two fused features are obtained. 5) Dynamically update the domain feature knowledge base using the aforementioned fusion features, and maintain a dynamic feature vector for each person in each domain; 6) Based on the updated domain feature knowledge base, the two fused features are jointly optimized by calculating identity loss, alignment loss, intra-domain uniformity loss, and modality contrast loss; 7) Extract query features using only convolutional neural networks that extract visible light image features, and sort them by calculating their cosine similarity with all features in the domain feature knowledge base, outputting the most matching person.

[0007] To address the issues of traditional methods' over-reliance on clothing features and weak generalization ability, this invention proposes an innovative approach that integrates multimodal information with domain generalization techniques. First, by introducing grayscale image modalities and fusing them with visible light images, biometric features unrelated to clothing are effectively captured, thus compensating for the limitations of visible light images in uniform clothing scenarios. Second, a structure-aware data augmentation strategy is proposed, utilizing semantic segmentation to identify and protect key discriminative regions on workwear and / or safety helmets (such as the areas containing name tags and serial numbers), enhancing only non-critical background areas to avoid compromising rare discriminative features (such as name tags and serial numbers) while increasing data diversity. Finally, by designing a domain-aware prototype memory and a loss function that balances alignment and uniformity, domain-invariant feature representations are explicitly learned, focusing on the inherent characteristics of the personnel rather than environmental background noise, thereby significantly improving generalization capabilities in unknown substations.

[0008] Furthermore, in step 1), the multimodal data is... , Visible light image, It is a grayscale image. For personnel identification tags, The fields are labeled, where H, W, and C represent the image's height, width, and number of channels, respectively.

[0009] Further, the pre-processing of image normalization and standardization in step 1) provides complementary information sources; image normalization scales pixel values from [0, 255] to [-1, 1], and standardization uses the mean and standard deviation of ImageNet.

[0010] Further, the specific content of step 2) includes: 2.1) Process the visible light image using the pre-trained semantic segmentation network SegNet Generate a structure mask to protect key discriminative areas on the tooling or / and safety helmet (such as the area where the nameplate, number, etc. are located), to avoid destroying these key areas by enhancement operations: , where the structure mask is a binary mask of the same size as the visible light image , 1 represents the key area that needs to be protected, and 0 represents the background area outside the mask; 2.2) Perform random enhancement operations (such as erasing, flipping, color jittering, etc.) on the background area outside the mask to protect discriminative features: , where represents element-wise multiplication, is a randomly selected regular enhancement operation, is the visible light enhanced image.

[0011] Further, the specific content of step 3) includes: 3.1) Extract the features of the visible light image using a pre-trained convolutional neural network: ; where is the convolutional neural network for extracting features of the visible light image; 3.2) Extract the features of the visible light enhanced image using the shared convolutional neural network: ; 3.3) Extract the features of the grayscale image using another pre-trained convolutional neural network to supplement visible light information: ; where is the convolutional neural network for extracting features of the grayscale image.

[0012] Further, the specific content of step 4) includes: 4.1) Characteristics of Visible Light Images Features of grayscale images Adaptive fusion, introducing an attention mechanism to calculate the first adaptive weights. The weight is determined by the input features, dynamically balancing the importance of the two modalities in a specific scenario:

[0013] in, For feature splicing, It is a fully connected layer. This represents the sigmoid activation function; Adaptive weighted fusion features:

[0014] In the formula, This refers to the fusion features of visible light images and grayscale images; If the visible light image quality is good Close to 1; if lighting conditions are poor, Smaller size increases the weight of grayscale image features; 4.2) Features of visible light enhancement maps Features of grayscale images Similarly, adaptive fusion is performed, and an attention mechanism is introduced to calculate the second adaptive weights. :

[0015]

[0016] In the formula, This is a fusion feature of the visible light enhancement image and the grayscale image features.

[0017] Furthermore, step 5) specifically includes: 5.1) Maintain a domain feature knowledge base for each domain, storing the feature vector of each sample in that domain.

[0018] in, For domain tags, This represents the number of samples in this domain. This indicates that the identity tags of the people in this domain are y The dynamic feature vector; 5.2) Initially, it is a zero vector or randomly initialized. During training, a momentum update strategy is used, updating only when a sample belongs to the corresponding domain and identity.

[0019] in, It is the momentum coefficient. This is a fusion feature of visible light image and grayscale image features.

[0020] Furthermore, step 6) specifically includes: 6.1) Loss of identity Cross-entropy loss and triplet loss composition,

[0021] in, For the first i The fusion features of visible light images and grayscale images of individual samples. For the first i Personnel identification tags for each sample For a linear classification layer, softmax It is a non-linear activation function. B The size of a training batch;

[0022] in, For the first i Anchor sample features of each sample For the first i Positive sample features of a sample For the first i Negative sample features of each sample d It is a Euclidean distance metric. For boundary parameters, ultimately, ; 6.2) Alignment Loss This makes the enhanced fused features and the original fused features close in feature space:

[0023] in, Represents the square of the L2 norm; For the first i The fusion features of the visible light enhancement image and grayscale image of each sample; 6.3) Loss of uniformity within the domain This makes the feature distribution within the domain more uniform: +

[0024] in, For domain d Zhongyu An index set of the most recent samples from different individuals. For domain d The number of samples; Representation domain d The identity label of the Chinese personnel is j The dynamic feature vector; 6.4) Modal contrast loss Zooming in on the visible light and grayscale features of the same sample:

[0025] in, For temperature coefficient, sim Cosine similarity; Indicates the first i Visible light fusion characteristics of individual samples; Indicates the first i Grayscale image fusion features of individual samples; Indicates the first j The grayscale image fusion features of each sample, where j is not equal to i; Total loss for: .

[0026] Furthermore, step 7) specifically includes: 7.1) During the reasoning phase, for the query graph Heku Images The query features are extracted using only convolutional neural networks that extract features from visible light images:

[0027]

[0028] In the formula, A convolutional neural network for extracting features from visible light images; 7.2) Calculate the cosine similarity between the query feature and all features in the database:

[0029] Based on cosine similarity Sort the images in descending order and output the top ten most matching images to complete the re-identification task.

[0030] Secondly, the present invention provides a domain-generalized person re-identification system based on multimodal fusion and structure-aware enhancement, comprising: Data input and preprocessing module: used for multimodal data input and preprocessing, wherein the multimodal data includes visible light images and grayscale images; Structure-aware data augmentation module: Processes only visible light images, uses a pre-trained semantic segmentation network to generate a structure mask, identifies the key identification regions that need to be protected, and applies random augmentation operations to the background regions outside the mask to obtain a visible light augmented image; Deep feature extraction module: Two independent pre-trained convolutional neural networks are used to process visible light images and grayscale images respectively, extracting unique features from the two modalities of data; the convolutional neural network used to extract visible light image features shares weights, processes the visible light enhancement image and extracts features; Feature fusion module: An attention mechanism is introduced to fuse visible light image and grayscale image features and visible light enhanced image and grayscale image features respectively, calculate their respective adaptive weights, dynamically weigh the importance of the two modalities in a specific scene, and finally obtain two fused features; Domain Feature Knowledge Base Update Module: Dynamically updates the domain feature knowledge base using the fused features, maintaining a dynamic feature vector for each person in each domain; Joint optimization module for fused features: Based on the updated domain feature knowledge base, it jointly optimizes the two fused features by calculating identity loss, alignment loss, intra-domain uniformity loss and modality contrast loss; Inference module: During inference, only convolutional neural networks that extract visible light image features are used to extract query features, and the query features are sorted by calculating their cosine similarity with all features in the domain feature knowledge base, and the most matching person is output.

[0031] The beneficial effects of this invention are as follows: Addressing the core challenges of uniform attire, minimal appearance differences, and poor cross-scenario generalization in power industry scenarios, this invention innovatively introduces grayscale image modalities to extract biometric information unrelated to clothing, effectively compensating for the limitations of visible light images in recognition under uniform occlusion and adverse lighting conditions. Simultaneously, this invention proposes a structure-aware data enhancement strategy, protecting key identification regions of workwear and / or safety helmets through semantic segmentation technology, improving data diversity while avoiding the destruction of effective features. Furthermore, this invention designs a domain generalization framework based on a balance between alignment and uniformity. By introducing targeted alignment loss, uniformity loss, and intra-domain uniformity loss, it directly optimizes and regularizes the distribution characteristics of the feature space, learning domain-invariant identity features and significantly improving generalization performance in unknown substation scenarios.

[0032] Compared with existing methods, this invention achieves superior detection performance with limited labeled data, effectively reducing reliance on large-scale labeled data and lowering deployment costs. Attached Figure Description

[0033] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0034] Figure 1 This is a flowchart of a domain generalization person re-identification method based on multimodal fusion and structure perception enhancement according to the present invention; Figure 2 This is a schematic diagram of the architecture of a domain generalization person re-identification method based on multimodal fusion and structure perception enhancement according to the present invention. Figure 3 This is a structural diagram of a domain generalization person re-identification system based on multimodal fusion and structure perception enhancement according to the present invention. Detailed Implementation

[0035] The technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0036] Example 1 As shown in Figure 1 and Figure 2 As shown, this invention is a domain-generalized person re-identification method based on multimodal fusion and structure-aware enhancement, and its steps are as follows: Step 1) Multimodal data input and preprocessing: Visible light images and grayscale images are used as input. Preprocessing operations such as normalization and standardization are performed on the images to provide complementary information sources. Image normalization scales pixel values ​​from [0, 255] to [-1, 1], using the mean and standard deviation from ImageNet for standardization.

[0037] The input for each multimodal data (i.e., training sample) is , (Where H, W, and C represent height, width, and number of channels, respectively) is a visible light image. It is a grayscale image. For personnel identification tags, For domain tags.

[0038] Step 2) Structure-aware data augmentation: Only the visible light image is processed. A pre-trained semantic segmentation network is used to generate a structure mask to identify key identification areas that need to be protected, such as work clothes and safety helmets. Random enhancement transformations (such as erasure and color dithering) are applied to the background areas outside the mask to obtain the visible light enhanced image.

[0039] First, visible light images are processed using the pre-trained semantic segmentation network SegNet. Generate a structural mask to protect key identification areas on workwear and safety helmets (such as the areas where name tags and serial numbers are located) and prevent enhancements from damaging these key features (such as name tags and serial numbers).

[0040] Among them, the structure mask It is a with A binary mask of the same size, where 1 represents the area that needs to be protected and 0 represents the area that can be enhanced.

[0041] Secondly, non-critical areas are enhanced to protect discriminative features:

[0042] in, This indicates element-wise multiplication. For randomly selected routine enhancement operations (such as erase, flip, color dithering, etc.). This is a visible light enhancement image.

[0043] Step 3) Deep Feature Extraction: Two independent pre-trained convolutional neural networks are used to process the visible light image (i.e., the original visible light image and the enhanced visible light image) and the grayscale image, respectively, to extract unique feature representations from each modality. A convolutional neural network with shared weights processes the original visible light image and the enhanced visible light image, allowing the network to learn invariance to enhancement changes.

[0044] First, a pre-trained convolutional neural network is used to extract visible light images. Features : ; In the formula, A convolutional neural network for extracting features from visible light images; Then, a shared convolutional neural network extracts the visible light enhancement map. Features : ; Finally, another pre-trained convolutional neural network is used to extract the grayscale image. Features Supplementing visible light information: ; In the formula, A convolutional neural network for extracting grayscale image features.

[0045] Step 4) Adaptive fusion of multimodal features: An attention mechanism is introduced to fuse visible light image (i.e., the original visible light image and the enhanced visible light image) with grayscale image features, calculate adaptive weights, dynamically weigh the importance of the two modalities in a specific scene, and finally fuse the features.

[0046] First, visible light image features and grayscale image features Adaptive fusion, introducing an attention mechanism to calculate the first adaptive weights. The weight is determined by the input features, dynamically balancing the importance of the two modalities in a specific scenario:

[0047] in, For feature splicing, It is a fully connected layer. This represents the sigmoid activation function. Then, adaptive weighted fusion features:

[0048] In the formula, This refers to the fusion features of visible light images and grayscale images; If the visible light image quality is good The value is close to 1, mainly depending on visible light characteristics; if the lighting conditions are poor, The smaller the size, the greater the weight of the grayscale image features.

[0049] Finally, the features of the visible light enhancement map Features of grayscale images Similarly, adaptive fusion is performed, and an attention mechanism is introduced to calculate the second adaptive weights. :

[0050]

[0051] In the formula, This is a fusion feature of the visible light enhancement image and the grayscale image features.

[0052] Step 5) Dynamically update the domain feature knowledge base using the fused features, maintaining a dynamic feature vector for each person in each scene (domain). During training, a momentum update strategy is used to update only when a sample belongs to the corresponding domain and identity.

[0053] First, maintain a domain feature knowledge base for each domain. Save the feature vector of each sample in this domain.

[0054] in, For domain tags, This represents the number of samples in this domain. This indicates that the identity tags of the people in this domain are y The dynamic feature vector; Then, Initially, the vector is either zero or randomly initialized. During training, a momentum update strategy is used, updating only when a sample belongs to the corresponding domain and identity.

[0055] in, It is the momentum coefficient.

[0056] Step 6) Based on the updated domain feature knowledge base, the two fused features are jointly optimized by calculating identity loss, alignment loss, intra-domain uniformity loss and modality contrast loss, so as to obtain the final feature representation with strong representation ability and domain invariance.

[0057] First, loss of identity Cross-entropy loss and triplet loss composition,

[0058] in, For the first i The fusion features of visible light images and grayscale images of individual samples. For the first i Personnel identification tags for each sample For a linear classification layer, softmax It is a non-linear activation function. B The size of a batch of training data.

[0059]

[0060] in, For the first i Anchor sample features of each sample For the first i Positive sample features of a sample For the first i Negative sample features of each sample d It is a Euclidean distance metric. For boundary parameters, ultimately,

[0061] Subsequently, alignment loss This makes the enhanced fused features and the original fused features close in feature space.

[0062] in, Denotes the square of the L2 norm. For the first i The fusion features of the visible light enhancement image and grayscale image of each sample; Then, the loss of uniformity within the domain. This makes the feature distribution within the domain more uniform. +

[0063] in, For domain d Zhongyu An index set of the most recent samples from different individuals. For domain d The number of samples, Representation domain d The identity label of the Chinese personnel is j The dynamic feature vector; Finally, modal contrast loss Zoom in on the visible light image and grayscale image features of the same sample.

[0064] in, For temperature coefficient, sim Cosine similarity; Indicates the first i Visible light fusion characteristics of individual samples; Indicates the first i Grayscale image fusion features of individual samples; Indicates the first j The grayscale image fusion features of each sample, where j is not equal to i; Total loss for: .

[0065] Step 7) Extract query features using only a convolutional neural network that extracts visible light image features, and sort them by calculating their cosine similarity with all features in the domain feature knowledge base, outputting the most matching person.

[0066] First, for query graphs Heku Images Features are extracted using only its visible light branch.

[0067]

[0068] Then, the cosine similarity between the query feature and all features in the domain feature knowledge base is calculated.

[0069] By similarity Sort the images in descending order and output the top ten most matching images to complete the re-identification task.

[0070] This invention proposes a domain-generalized person re-identification method based on multimodal fusion and structure-aware enhancement, and proposes a multi-branch architecture MMDG model based on this method. Compared with existing methods, MMDG does not overly rely on appearance features, can better cope with scene changes, and has strong generalization ability and high robustness in complex scenarios such as substation and transmission line inspection.

[0071] This invention utilizes publicly available datasets commonly used in people re-identification tasks: Market1501, MSMT17, CUHK03, and the self-made power scene dataset GRID01. The model is trained on the three public datasets, and then evaluated on GRID01 to test its generalization ability.

[0072] The experimental setup is as follows: 1. The convolutional neural network used is ResNet-50. 2. Batch size... B =64, 16 personnel samples, 4 images collected for each sample. 3. Hyperparameter settings: Temperature coefficient τ is 0.05, momentum coefficient... The initial learning rate is 0.0001, with a value of 0.1. During training, the Adam optimizer is used, with an L2 regularization weight decay factor of 1e. 6. The total number of training rounds is 60. The learning rate is increased linearly in the first 10 rounds for warm-up, and then decreased by a factor of 10 in the 30th and 50th rounds.

[0073] This invention uses mean accuracy (mAP) and Rank-1, Rank-5, and Rank-10 accuracy to evaluate the performance of the MMDG model. Compared with other methods, the MMDG model proposed in this invention significantly improves the evaluation metrics, indicating that the MMDG model of this invention can more accurately identify people and has good generalization ability, as shown in Table 1.

[0074] Table 1 Performance Evaluation of the Model

[0075] Example 2 This embodiment provides a domain-generalized person re-identification system based on multimodal fusion and structure-aware enhancement, such as... Figure 3As shown, it consists of a data input and preprocessing module, a structure-aware data enhancement module, a deep feature extraction module, a feature fusion module, a domain feature knowledge base update module, a fusion feature joint optimization module, and an inference module.

[0076] Data input and preprocessing module: used for multimodal data input and preprocessing, the multimodal data including visible light images and grayscale images; used to implement step 1) in embodiment 1, which will not be repeated here.

[0077] Structure-aware data augmentation module: It processes only visible light images, uses a pre-trained semantic segmentation network to generate a structure mask, identifies the key identification regions that need to be protected, and applies random augmentation operations to the background regions outside the mask to obtain a visible light augmentation map; it is used to implement step 2 in embodiment 1, and will not be described again here.

[0078] Deep feature extraction module: Two independent pre-trained convolutional neural networks are used to process visible light images and grayscale images respectively, and extract unique features from the two modal data respectively; the convolutional neural network used to extract visible light image features shares weights, processes the visible light enhancement image and extracts features; it is used to implement step 3) in embodiment 1, which will not be repeated here.

[0079] Feature fusion module: An attention mechanism is introduced to fuse the features of visible light image and grayscale image, as well as the features of visible light enhanced image and grayscale image, respectively. The adaptive weights of each are calculated to dynamically weigh the importance of the two modalities in a specific scene, and finally two fused features are obtained; this is used to implement step 4 in embodiment 1, which will not be repeated here.

[0080] Domain Feature Knowledge Base Update Module: Dynamically updates the domain feature knowledge base using the fused features, maintaining a dynamic feature vector for each person in each domain; used to implement step 5 in Example 1, which will not be repeated here.

[0081] The joint optimization module for fused features: Based on the updated domain feature knowledge base, it jointly optimizes the two fused features by calculating identity loss, alignment loss, intra-domain uniformity loss and modality contrast loss; it is used to implement step 6 in Example 1, and will not be repeated here.

[0082] Inference module: During inference, only the convolutional neural network that extracts visible light image features is used to extract query features, and the query features are sorted by calculating the cosine similarity between the query features and all features in the domain feature knowledge base, and the most matching person is output; this is used to implement step 7 in embodiment 1, and will not be repeated here.

[0083] It should be noted that each module in the aforementioned domain generalization person re-identification system based on multimodal fusion and structure-aware enhancement can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the corresponding operations of each module. For specific limitations regarding the domain generalization person re-identification system based on multimodal fusion and structure-aware enhancement, please refer to the limitations of the domain generalization person re-identification method based on multimodal fusion and structure-aware enhancement mentioned above; both have the same function and role, and will not be repeated here.

[0084] It should be noted that the order of the above embodiments of the present invention is merely for descriptive purposes and does not represent the superiority or inferiority of the embodiments. Furthermore, the above description focuses on specific embodiments of this specification. Additionally, the processes depicted in the accompanying drawings do not necessarily require a specific or sequential order to achieve the desired results. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0085] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

[0086] This specification and accompanying drawings are merely illustrative examples of the present invention and are intended to cover any and all modifications, variations, combinations, or equivalents within the scope of the present invention. Clearly, those skilled in the art can make various alterations and modifications to the present invention without departing from its scope. Therefore, if such modifications and variations fall within the scope of the present invention and its equivalents, the present invention intends to include these modifications and variations.

Claims

1. A domain-generalized person re-identification method based on multimodal fusion and structure-aware enhancement, characterized in that, Including the following steps: 1) Multimodal data input and preprocessing, wherein the multimodal data includes visible light images and grayscale images; 2) Only the visible light image is processed. A pre-trained semantic segmentation network is used to generate a structural mask to identify the key identification regions that need to be protected. Random enhancement operations are applied to the background regions outside the mask to obtain the visible light enhanced image. 3) Two independent pre-trained convolutional neural networks are used to process visible light images and grayscale images respectively, and their unique features are extracted from the two modalities of data respectively; the convolutional neural network used to extract visible light image features shares weights, processes the visible light enhancement image and extracts features; 4) An attention mechanism is introduced to fuse the features of the visible light image and the grayscale image, as well as the features of the visible light enhanced image and the grayscale image, respectively. The adaptive weights of each are calculated to dynamically weigh the importance of the two modalities in a specific scene, and finally two fused features are obtained. 5) Dynamically update the domain feature knowledge base using the aforementioned fusion features, and maintain a dynamic feature vector for each person in each domain; 6) Based on the updated domain feature knowledge base, the two fused features are jointly optimized by calculating identity loss, alignment loss, intra-domain uniformity loss, and modality contrast loss; 7) Extract query features using only convolutional neural networks that extract visible light image features, and sort them by calculating their cosine similarity with all features in the domain feature knowledge base, outputting the most matching person.

2. The domain generalization person re-identification method based on multimodal fusion and structure-aware enhancement according to claim 1, characterized in that, In step 1), the multimodal data is , Visible light image, It is a grayscale image. For personnel identification tags, The fields are labeled, where H, W, and C represent the image's height, width, and number of channels, respectively.

3. The domain generalization person re-identification method based on multimodal fusion and structure-aware enhancement according to claim 1, characterized in that, In step 1), the image is preprocessed by normalization and standardization; image normalization scales the pixel values ​​from [0, 255] to [-1, 1] and standardizes them using the mean and standard deviation of ImageNet.

4. The domain generalization person re-identification method based on multimodal fusion and structure-aware enhancement according to claim 1, characterized in that, Step 2) includes the following: 2.1) Process visible light images using the pre-trained semantic segmentation network SegNet. Generate a structural mask to protect critical areas on tooling and / or safety helmets: ， Among them, the structure mask It is a visible light image For binary masks of the same size, 1 represents the critical area that needs to be protected, and 0 represents the background area outside the mask; 2.2) Perform random enhancement operations on the background area outside the mask to protect discriminative features: ， in, This indicates element-wise multiplication. For randomly selected routine enhancement operations, This is a visible light enhancement image.

5. The domain generalization person re-identification method based on multimodal fusion and structure-aware enhancement according to claim 1, characterized in that, Step 3) includes the following: 3.1) Extract visible light images using a pre-trained convolutional neural network. Features : ； In the formula, A convolutional neural network for extracting features from visible light images; 3.2) Extracting Visible Light Enhancement Maps Using Shared Convolutional Neural Networks Features : ； 3.3) Extract grayscale images using another pre-trained convolutional neural network. Features Supplementing visible light information: ； In the formula, A convolutional neural network for extracting grayscale image features.

6. The domain generalization person re-identification method based on multimodal fusion and structure-aware enhancement according to claim 1, characterized in that, Step 4) includes the following: 4.1) Characteristics of Visible Light Images Features of grayscale images Adaptive fusion, introducing an attention mechanism to calculate the first adaptive weights. The weight is determined by the input features, dynamically balancing the importance of the two modalities in a specific scenario: ， in, For feature splicing, It is a fully connected layer. This represents the sigmoid activation function; Adaptive weighted fusion features: ， In the formula, This refers to the fusion features of visible light images and grayscale images; If the visible light image quality is good Close to 1; if lighting conditions are poor, Smaller size increases the weight of grayscale image features; 4.2) Features of visible light enhancement maps Features of grayscale images Similarly, adaptive fusion is performed, and an attention mechanism is introduced to calculate the second adaptive weights. : ， ， In the formula, This is a fusion feature of the visible light enhancement image and the grayscale image features.

7. The domain generalization person re-identification method based on multimodal fusion and structure-aware enhancement according to claim 1, characterized in that, Step 5) includes the following: 5.1) Maintain a domain feature knowledge base for each domain, storing the feature vector of each sample in that domain. ， in, For domain tags, This represents the number of samples in this domain. This indicates that the identity tags of the people in this domain are y The dynamic feature vector; 5.2) Initially, it is a zero vector or randomly initialized. During training, a momentum update strategy is used, updating only when a sample belongs to the corresponding domain and identity. ， in, It is the momentum coefficient. This is a fusion feature of visible light image and grayscale image features.

8. The domain generalization person re-identification method based on multimodal fusion and structure-aware enhancement according to claim 7, characterized in that, Step 6) includes the following: 6.1) Loss of identity Cross-entropy loss and triplet loss composition, ， in, For the first i The fusion features of visible light images and grayscale images of individual samples. For the first i Personnel identification tags for each sample For a linear classification layer, softmax It is a non-linear activation function. B The size of a training batch; ， in, For the first i Anchor sample features of each sample For the first i Positive sample features of a sample For the first i Negative sample features of each sample d It is a Euclidean distance metric. For boundary parameters, ultimately, ； 6.2) Alignment Loss This makes the enhanced fused features and the original fused features close in feature space: ， in, Represents the square of the L2 norm; For the first i The fusion features of the visible light enhancement image and grayscale image of each sample; 6.3) Loss of uniformity within the domain This makes the feature distribution within the domain more uniform: + ， in, For domain d Zhongyu An index set of the most recent samples from different individuals. For domain d The number of samples; Representation domain d The identity label of the Chinese personnel is j The dynamic feature vector; 6.4) Modal contrast loss Zooming in on the visible light and grayscale features of the same sample: ， in, For temperature coefficient, sim Cosine similarity; Indicates the first i Visible light fusion characteristics of individual samples; Indicates the first i Grayscale image fusion features of individual samples; Indicates the first j The grayscale image fusion features of each sample, where j is not equal to i; Total loss for: .

9. The domain generalization person re-identification method based on multimodal fusion and structure-aware enhancement according to claim 1, characterized in that, Step 7) includes the following: 7.1) During the reasoning phase, for the query graph Heku Images The query features are extracted using only convolutional neural networks that extract features from visible light images: ， ， In the formula, A convolutional neural network for extracting features from visible light images; 7.2) Calculate the cosine similarity between the query feature and all features in the database: ， Based on cosine similarity Sort the images in descending order and output the top ten most matching images to complete the re-identification task.

10. A domain-generalized person re-identification system based on multimodal fusion and structure-aware enhancement, characterized in that, include: Data input and preprocessing module: used for multimodal data input and preprocessing, wherein the multimodal data includes visible light images and grayscale images; Structure-aware data augmentation module: Processes only visible light images, uses a pre-trained semantic segmentation network to generate a structure mask, identifies the key identification regions that need to be protected, and applies random augmentation operations to the background regions outside the mask to obtain a visible light augmented image; Deep feature extraction module: Two independent pre-trained convolutional neural networks are used to process visible light images and grayscale images respectively, extracting unique features from the two modalities of data; the convolutional neural network used to extract visible light image features shares weights, processes the visible light enhancement image and extracts features; Feature fusion module: An attention mechanism is introduced to fuse visible light image and grayscale image features and visible light enhanced image and grayscale image features respectively, calculate their respective adaptive weights, dynamically weigh the importance of the two modalities in a specific scene, and finally obtain two fused features; Domain Feature Knowledge Base Update Module: Dynamically updates the domain feature knowledge base using the fused features, maintaining a dynamic feature vector for each person in each domain; Joint optimization module for fused features: Based on the updated domain feature knowledge base, it jointly optimizes the two fused features by calculating identity loss, alignment loss, intra-domain uniformity loss and modality contrast loss; Inference module: During inference, only convolutional neural networks that extract visible light image features are used to extract query features, and the query features are sorted by calculating their cosine similarity with all features in the domain feature knowledge base, and the most matching person is output.

Images