Image matching method and system based on multi-feature fusion, electronic device and medium

Abstract

The application discloses an image matching method and system based on multi-feature fusion, electronic equipment and medium. The method comprises the following steps: constructing a target network model containing a specific feature branch and a shared feature branch, and connecting a first classifier at the end of the specific feature branch and a second classifier at the end of the shared feature branch; constructing a target loss function based on the features output by the first classifier, the second classifier, a first feature extraction module, an information purification module, a first global average pooling, a second feature extraction module, a plurality of convolutional layers and a plurality of second global average poolings; training the target network model based on the target loss function by using a plurality of visible light images and a plurality of infrared images to obtain a trained target network model; and performing image matching on a target visible light image and a target infrared image by using the shared feature branch in the trained target network model to obtain an image matching result. The application can improve the accuracy of cross-modal image matching.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image matching, in particular to an image matching method and system based on multi-feature fusion, an electronic device and a medium. BACKGROUND

[0002] Visible light pedestrian image and infrared pedestrian image matching as an important research direction in the field of cross-modal recognition aims to match the same pedestrian in images obtained by different spectral imaging devices at different times of day and night. Since infrared images have stronger robustness at night and visible light images contain more rich texture and color information, how to achieve effective feature matching and identity recognition between the two modalities becomes a challenge. The existing technology mainly solves the modal difference problem through image-level conversion or feature-level alignment, but these methods still have many bottlenecks.

[0003] The image-level conversion method reduces the pixel domain difference through modal migration or image fusion, which alleviates the apparent difference between modalities to some extent, but easily introduces noise information such as artifacts and structural distortion, affecting the recognition accuracy. The feature-level alignment method usually constructs a double-flow neural network to extract shared embedding features between modalities, and realizes feature alignment through projection learning or modal confrontation. However, most methods emphasize modal invariance at the expense of modal exclusive discriminative clues, especially in infrared images where the lack of texture leads to weak identity information.

[0004] In addition, due to the high cost of cross-modal pedestrian re-identification (ReID) annotation, the existing data sets are generally small in size, resulting in insufficient training supervision signals, easy overfitting of the model, and poor robustness. Therefore, the existing technology has relatively low accuracy for cross-modal image matching. SUMMARY

[0005] The present application aims to provide an image matching method and system based on multi-feature fusion, which can improve the accuracy of cross-modal image matching.

[0006] In a first aspect, an image matching method based on multi-feature fusion is provided, which comprises:

[0007] Obtaining a target visible light image and a target infrared image to be matched, and obtaining a plurality of visible light images and a plurality of infrared images for model training;

[0008] Based on the ResNet-50 network, a target network model containing a specific feature branch and a shared feature branch is constructed, and a first classifier is connected at the end of the specific feature branch, and a second classifier is connected at the end of the shared feature branch; wherein the specific feature branch includes a first feature extraction module, an information purification module and a first global average pooling, and the shared feature branch includes a second feature extraction module, a plurality of convolutional layers and a plurality of second global average poolings;

[0009] Based on the features output by the first classifier, the second classifier, the first feature extraction module, the information purification module, the first global average pooling, the second feature extraction module, the plurality of convolutional layers and the plurality of second global average poolings respectively, a target loss function is constructed;

[0010] Based on the target loss function, the target network model is trained using the plurality of visible light images and the plurality of infrared images to obtain a trained target network model;

[0011] Through the shared feature branch in the trained target network model, image matching is performed on the target visible light image and the target infrared image to obtain an image matching result.

[0012] Compared with the prior art, the first aspect of the present application has the following beneficial effects:

[0013] The method comprises the following steps: acquiring a target visible light image and a target infrared image to be matched, and acquiring a plurality of visible light images and a plurality of infrared images for model training; based on a ResNet-50 network, a target network model containing a specific feature branch and a shared feature branch is constructed, and a first classifier is connected at the end of the specific feature branch, and a second classifier is connected at the end of the shared feature branch; wherein the specific feature branch comprises a first feature extraction module, an information purification module and a first global average pooling, and the shared feature branch comprises a second feature extraction module, a plurality of convolutional layers and a plurality of second global average poolings; based on the features output by the first classifier, the second classifier, the first feature extraction module, the information purification module, the first global average pooling, the second feature extraction module, the plurality of convolutional layers and the plurality of second global average poolings respectively, a target loss function is constructed; based on the target loss function, the target network model is trained by using the plurality of visible light images and the plurality of infrared images, and a trained target network model is obtained; and the target visible light image and the target infrared image are subjected to image matching by using the shared feature branch in the trained target network model, and an image matching result is obtained. In this way, by using the features output by the first classifier, the second classifier, the first feature extraction module, the information purification module, the first global average pooling, the second feature extraction module, the plurality of convolutional layers and the plurality of second global average poolings in the specific feature branch and the shared feature branch respectively, the target loss function is constructed, a plurality of features are combined to construct the target loss function, modal separation and cross-modal consistency alignment can be realized, the target network model can be driven to learn a shared representation with modal invariance and identity distinguishability while retaining modal specific discriminative information, and therefore the accuracy of cross-modal image matching is improved.

[0014] In some embodiments, the constructing, based on the features output by the first classifier, the second classifier, the first feature extraction module, the information purification module, the first global average pooling, the second feature extraction module, the plurality of convolutional layers and the plurality of second global average poolings respectively, a target loss function comprises:

[0015] constructing, based on the features output by the first feature extraction module, the information purification module and the first global average pooling respectively, a loss function corresponding to the specific feature branch;

[0016] constructing, based on the features output by the second feature extraction module, the plurality of convolutional layers and the plurality of second global average poolings respectively, a loss function corresponding to the shared feature branch;

[0017] constructing, based on the features finally output by the specific feature branch and the features finally output by the shared feature branch, a spherical contrast distillation loss function;

[0018] constructing a fine-grained distillation loss function based on the result output by the first classifier and the result output by the second classifier;

[0019] constructing a target loss function based on the loss function corresponding to the specific feature branch, the loss function corresponding to the shared feature branch, the spherical contrast distillation loss function, and the fine-grained distillation loss function.

[0020] In some embodiments, the constructing the loss function corresponding to the specific feature branch based on the features output by the first feature extraction module, the information purification module, and the first global average pooling includes:

[0021] inputting the feature output by the first feature extraction module into a modal discriminator to perform modal classification, and constructing a modal discrimination loss function;

[0022] inputting the feature output by the first feature extraction module into the information purification module to perform information purification, to obtain a feature output by the information purification module;

[0023] performing the first global average pooling operation on the feature output by the information purification module, to obtain a feature finally output by the specific feature branch;

[0024] constructing a first person re-identification loss function according to the feature finally output by the specific feature branch;

[0025] taking the modal discrimination loss function and the first person re-identification loss function as the loss function corresponding to the specific feature branch.

[0026] In some embodiments, the information purification module includes an instance normalization, a convolutional neural network, a plurality of convolutional operations, and a pooling operation, and the inputting the feature output by the first feature extraction module into the information purification module to perform information purification to obtain a feature output by the information purification module includes:

[0027] performing instance normalization on the feature output by the first feature extraction module to obtain a normalized feature;

[0028] generating, by the convolutional neural network, a first mask corresponding to the feature output by the first feature extraction module and a second mask corresponding to the normalized feature based on the normalized feature and the feature output by the first feature extraction module;

[0029] performing convolutional operation on the feature output by the first feature extraction module and the first mask to obtain a first feature;

[0030] performing convolutional operation on the normalized feature and the second mask to obtain a second feature;

[0031] performing convolution operation on the first feature and the second mask to obtain a third feature;

[0032] performing convolution operation on the second feature and the first mask to obtain a fourth feature;

[0033] splicing the third feature and the fourth feature to obtain a spliced feature;

[0034] performing pooling operation on the spliced feature to obtain a feature output by the information purification module.

[0035] In some embodiments, the loss function corresponding to the shared feature branch is constructed based on the features output by the second feature extraction module, the plurality of convolution layers and the plurality of second global average pooling, comprising:

[0036] inputting the feature output by the second feature extraction module into a modal classifier to perform modal classification, and constructing a modal classification loss function;

[0037] processing the feature output by the second feature extraction module through the plurality of convolution layers and one second global average pooling to obtain a learnable visual feature;

[0038] combining the histogram of oriented gradient features and the texture features extracted from the visible light image and the infrared image into a target domain prior vector;

[0039] constructing a prior loss function based on the learnable visual feature and the target domain prior vector;

[0040] processing the feature output by the second feature extraction module through one second global average pooling to obtain a deep learning feature;

[0041] constructing a pseudo perception module based on a UNet network, and performing feature extraction and feature fusion on the visible light image and the infrared image through the pseudo perception module to obtain image fusion encoding features;

[0042] constructing a pseudo perception loss function based on the deep learning feature and the image fusion encoding feature;

[0043] processing the learnable visual feature through one second global average pooling to obtain an average pooling result, and splicing the average pooling result and the deep learning feature to obtain a feature finally output by the shared feature branch;

[0044] constructing a second pedestrian re-identification loss function according to the feature finally output by the shared feature branch;

[0045] The modal classification loss function, the prior loss function, the pseudo-perception loss function, and the second pedestrian re-identification loss function are taken as loss functions corresponding to the shared feature branch.

[0046] In some embodiments, based on the features output by the specific feature branch and the features output by the shared feature branch, a spherical contrast distillation loss function is constructed, including:

[0047] ;

[0048] wherein, indicates the spherical contrast distillation loss function, indicates a spherical projection head, indicates the features output by the specific feature branch corresponding to the visible light image, indicates the features output by the shared feature branch corresponding to the visible light image, indicates the features output by the specific feature branch corresponding to the infrared image, indicates the features output by the shared feature branch corresponding to the infrared image, indicates an L2 norm.

[0049] In some embodiments, based on the results output by the first classifier and the results output by the second classifier, a fine-grained distillation loss function is constructed, including:

[0050] The results output by the first classifier are divided into a plurality of first spatial units, and the results output by the second classifier are divided into a plurality of second spatial units;

[0051] For each first spatial unit, a local logit corresponding to the specific feature branch is calculated to obtain a plurality of first local logits;

[0052] For each second spatial unit, a local logit corresponding to the shared feature branch is calculated to obtain a plurality of second local logits;

[0053] The KL divergence between the first local logit and the second local logit corresponding to the same spatial unit under the same modal is calculated;

[0054] Based on the KL divergence, a fine-grained distillation loss function is constructed.

[0055] In a second aspect, the embodiments of the present application further provide an image matching system based on multi-feature fusion, the system comprising:

[0056] An image acquisition unit is configured to acquire a target visible light image and a target infrared image to be matched, and acquire a plurality of visible light images and a plurality of infrared images for model training.

[0057] a model construction unit, configured to construct a target network model comprising a specific feature branch and a shared feature branch based on a ResNet-50 network, and connect a first classifier at the end of the specific feature branch and connect a second classifier at the end of the shared feature branch; wherein the specific feature branch comprises a first feature extraction module, an information purification module and a first global average pooling, and the shared feature branch comprises a second feature extraction module, a plurality of convolutional layers and a plurality of second global average poolings;

[0058] a function construction unit, configured to construct a target loss function based on features output by the first classifier, the second classifier, the first feature extraction module, the information purification module, the first global average pooling, the second feature extraction module, the plurality of convolutional layers and the plurality of second global average poolings respectively;

[0059] a model training unit, configured to train the target network model based on the target loss function and the plurality of visible light images and the plurality of infrared images, to obtain a trained target network model;

[0060] an image matching unit, configured to perform image matching on the target visible light image and the target infrared image through the shared feature branch in the trained target network model, to obtain an image matching result.

[0061] In a third aspect, an embodiment of the present application further provides an electronic device, including at least one control processor and a memory connected with the at least one control processor in communication; the memory stores instructions executable by the at least one control processor, and the instructions are executed by the at least one control processor to enable the at least one control processor to perform the image matching method based on multi-feature fusion as described above.

[0062] In a fourth aspect, an embodiment of the present application further provides a computer readable storage medium, which stores computer executable instructions for causing a computer to perform the image matching method based on multi-feature fusion as described above.

[0063] It can be understood that the beneficial effects of the second aspect to the fourth aspect compared with the related art are the same as the beneficial effects of the first aspect compared with the related art, and reference can be made to the related description in the first aspect, which will not be repeated here. BRIEF DESCRIPTION OF DRAWINGS

[0064] The above and / or additional aspects and advantages of the present application will become apparent and be readily appreciated from the following description, including the accompanying drawings, in which:

[0065] Figure 1 is a flowchart of an embodiment of the image matching method based on multi-feature fusion provided by the present application;

[0066] Figure 2 is a schematic diagram of an overall network model architecture in the best embodiment of the image matching method based on multi-feature fusion provided by the present application;

[0067] Figure 3 is a schematic diagram of an information purification process in the best embodiment of the image matching method based on multi-feature fusion provided by the present application;

[0068] Figure 4 is a schematic diagram of image fusion in a pseudo perception module in the best embodiment of the image matching method based on multi-feature fusion provided by the present application;

[0069] Figure 5 is a structural schematic diagram of an embodiment of the image matching system based on multi-feature fusion provided by the present application;

[0070] Figure 6 is a structural schematic diagram of an embodiment of the electronic device provided by the present application. DETAILED DESCRIPTION

[0071] Embodiments of the present application are described in detail below with reference to the accompanying drawings. The embodiments described below are examples for explaining the present application and should not be construed as limiting the present application.

[0072] In the description of the present application, if there is a description of first, second, etc., it is only for the purpose of distinguishing technical features and should not be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features or the order of indicated technical features.

[0073] In the description of the present application, it should be understood that the orientation description, such as the orientation or position relationship indicated by up, down, etc., is based on the orientation or position relationship shown in the drawings, only for the purpose of facilitating the description of the present application and simplifying the description, and is not intended to indicate or imply that the device or element referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore should not be construed as limiting the present application.

[0074] In the description of the present application, it should be noted that, unless otherwise explicitly limited, the words such as setting, installing, connecting, etc. should be broadly understood, and those skilled in the art can reasonably determine the specific meaning of the above words in the present application in combination with the specific content of the technical solution.

[0075] The prior art mainly solves the modal difference problem through image-level conversion or feature-level alignment, but these methods still have many bottlenecks. The image-level conversion method reduces the pixel domain difference through modal migration or image fusion, which alleviates the inter-modal apparent difference to a certain extent, but easily introduces noise information such as artifacts and structural distortion, affecting the recognition accuracy. The feature-level alignment method usually constructs a double-flow neural network to extract the shared embedding features between modalities, and realizes feature alignment through projection learning or modal confrontation. However, most methods emphasize modal invariance at the expense of modal-specific discriminative clues, especially in infrared images where the identity information is weakened due to the lack of texture.

[0076] In addition, due to the high cost of cross-modal pedestrian ReID (ReID) annotation, the existing data sets are generally small in size, resulting in insufficient training supervision signals, and the model is prone to overfitting and poor robustness. Therefore, the prior art has relatively low accuracy in matching cross-modal images.

[0077] To solve the problem of low accuracy in matching cross-modal images in the prior art, the present application provides an image matching method, system, electronic device and medium based on multi-feature fusion.

[0078] Reference Figure 1 The image matching method based on multi-feature fusion provided by the embodiments of the present application. The image matching method based on multi-feature fusion is applied to an electronic device, which can be a server or a mobile terminal, etc. As shown in Figure 1 The image matching method based on multi-feature fusion can include the following steps:

[0079] Step S101, obtaining a target visible light image and a target infrared image to be matched, and obtaining a plurality of visible light images and a plurality of infrared images for model training;

[0080] Step S102, based on the ResNet-50 network, constructing a target network model containing a specific feature branch and a shared feature branch, and connecting a first classifier at the end of the specific feature branch and a second classifier at the end of the shared feature branch; wherein the specific feature branch includes a first feature extraction module, an information purification module and a first global average pooling, and the shared feature branch includes a second feature extraction module, a plurality of convolutional layers and a plurality of second global average pooling;

[0081] Step S103, based on the features output by the first classifier, the second classifier, the first feature extraction module, the information purification module, the first global average pooling, the second feature extraction module, the plurality of convolutional layers and the plurality of second global average pooling, constructing a target loss function;

[0082] Step S104, training the target network model based on the target loss function, using the plurality of visible light images and the plurality of infrared images, to obtain a trained target network model.

[0083] Step S105, performing image matching on the target visible light image and the target infrared image through the shared feature branch in the trained target network model, to obtain an image matching result.

[0084] In the embodiment, the target visible light image and the target infrared image to be matched are obtained, and the plurality of visible light images and the plurality of infrared images used for model training are obtained; based on the ResNet-50 network, a target network model containing a specific feature branch and a shared feature branch is constructed, and a first classifier is connected at the end of the specific feature branch and a second classifier is connected at the end of the shared feature branch; wherein the specific feature branch includes a first feature extraction module, an information purification module and a first global average pooling, and the shared feature branch includes a second feature extraction module, a plurality of convolutional layers and a plurality of second global average poolings; based on the features output by the first classifier, the second classifier, the first feature extraction module, the information purification module, the first global average pooling, the second feature extraction module, the plurality of convolutional layers and the plurality of second global average poolings respectively, a target loss function is constructed; the target network model is trained based on the target loss function, using the plurality of visible light images and the plurality of infrared images, to obtain a trained target network model; and the target visible light image and the target infrared image are matched through the shared feature branch in the trained target network model, to obtain an image matching result. In this way, the target loss function is constructed based on the features output by the first classifier, the second classifier, the first feature extraction module, the information purification module, the first global average pooling, the second feature extraction module, the plurality of convolutional layers and the plurality of second global average poolings in the specific feature branch and the shared feature branch, and the target loss function is constructed in combination with a plurality of features, which can realize modal separation and cross-modal consistency alignment, can drive the target network model to learn a shared representation with modal invariance and identity distinguishability while preserving modal specific discriminative information, and thus improves the accuracy of cross-modal image matching.

[0085] The above obtaining the target visible light image and the target infrared image to be matched, and obtaining the plurality of visible light images and the plurality of infrared images used for model training can be image acquisition through different spectral imaging devices.

[0086] In some embodiments, constructing the target loss function based on the features output by the first classifier, the second classifier, the first feature extraction module, the information purification module, the first global average pooling, the second feature extraction module, the plurality of convolutional layers and the plurality of second global average poolings respectively includes:

[0087] construct a loss function corresponding to the specific feature branch based on the features output by the first feature extraction module, the information purification module, and the first global average pooling respectively;

[0088] construct a loss function corresponding to the shared feature branch based on the features output by the second feature extraction module, the plurality of convolutional layers, and the plurality of second global average poolings respectively;

[0089] construct a spherical contrast distillation loss function based on the features finally output by the specific feature branch and the features finally output by the shared feature branch;

[0090] construct a fine-grained distillation loss function based on the results output by the first classifier and the results output by the second classifier;

[0091] construct a target loss function by combining the loss function corresponding to the specific feature branch, the loss function corresponding to the shared feature branch, the spherical contrast distillation loss function, and the fine-grained distillation loss function.

[0092] In this embodiment, the loss function corresponding to the specific feature branch is constructed based on the features output by the first feature extraction module, the information purification module, and the first global average pooling respectively; the loss function corresponding to the shared feature branch is constructed based on the features output by the second feature extraction module, the plurality of convolutional layers, and the plurality of second global average poolings respectively; the spherical contrast distillation loss function is constructed based on the features finally output by the specific feature branch and the features finally output by the shared feature branch; the fine-grained distillation loss function is constructed based on the results output by the first classifier and the results output by the second classifier; and the target loss function is constructed by combining the loss function corresponding to the specific feature branch, the loss function corresponding to the shared feature branch, the spherical contrast distillation loss function, and the fine-grained distillation loss function. In this way, by combining the loss function corresponding to the specific feature branch, the loss function corresponding to the shared feature branch, the spherical contrast distillation loss function, and the fine-grained distillation loss function, the target loss function for end-to-end optimization is constructed, which can drive the target network model to learn a shared representation with modal invariance and identity distinguishability while preserving modal-specific discriminative information. Moreover, the spherical contrast distillation loss function and the fine-grained distillation loss function can share the knowledge distillation signals of the specific feature branch to the shared feature branch, guiding the shared feature branch to learn implicit discriminative knowledge, thereby enhancing feature compactness, achieving fine-grained semantic alignment, and enhancing the discriminative ability of shared features in the context of subtle differences between classes.

[0093] The information purification module described above is used to extract features with stronger discriminative ability and smaller modal differences, laying a good data foundation for image matching in the later stage.

[0094] In some embodiments, based on the features output by the first feature extraction module, the information purification module and the first global average pooling, a loss function corresponding to the specific feature branch is constructed, including:

[0095] The features output by the first feature extraction module are input into the modal discriminator for modal classification, and a modal discrimination loss function is constructed.

[0096] The features output by the first feature extraction module are input into the information purification module for information purification to obtain features output by the information purification module.

[0097] The features output by the information purification module are subjected to the first global average pooling operation to obtain features finally output by the specific feature branch.

[0098] According to the features finally output by the specific feature branch, a first person re-identification loss function is constructed.

[0099] The modal discrimination loss function and the first person re-identification loss function are taken as the loss function corresponding to the specific feature branch.

[0100] In the present embodiment, by inputting the features output by the first feature extraction module into the modal discriminator for modal classification and constructing the modal discrimination loss function, the decoupling separation of the modal specific features and the shared modal features is realized, avoiding the mutual interference of the modal characteristics and the shared semantics in the traditional method. Then, the features output by the first feature extraction module are input into the information purification module for information purification to obtain features output by the information purification module, which can extract more discriminative features and features with smaller modal differences, laying a good data foundation for image matching in the later stage. Finally, the modal discrimination loss function and the first person re-identification loss function are taken as the loss function corresponding to the specific feature branch, which can lay a good data foundation for constructing a comprehensive target loss function in the later stage.

[0101] In some embodiments, the information purification module includes instance normalization, a convolutional neural network, a plurality of convolution operations and a pooling operation, and the features output by the first feature extraction module are input into the information purification module for information purification to obtain features output by the information purification module, including:

[0102] The features output by the first feature extraction module are subjected to instance normalization to obtain normalized features.

[0103] Based on the normalized features and the features output by the first feature extraction module, a first mask corresponding to the features output by the first feature extraction module and a second mask corresponding to the normalized features are generated by the convolutional neural network.

[0104] The features output by the first feature extraction module and the first mask are subjected to convolution operation to obtain first features.

[0105] performing convolution operation on the normalized feature and the second mask to obtain a second feature;

[0106] performing convolution operation on the first feature and the second mask to obtain a third feature;

[0107] performing convolution operation on the second feature and the first mask to obtain a fourth feature;

[0108] performing splicing on the third feature and the fourth feature to obtain a spliced feature;

[0109] performing pooling operation on the spliced feature to obtain a feature output by the information purification module.

[0110] In the embodiment, the feature output by the first feature extraction module is input into the information purification module to perform information purification, so as to obtain the feature output by the information purification module, which can extract features with stronger discriminability and smaller modal difference, and lay a good data foundation for image matching in the later stage.

[0111] In some embodiments, based on the features output by the second feature extraction module, the plurality of convolution layers and the plurality of second global average pooling, a loss function corresponding to the shared feature branch is constructed, including:

[0112] inputting the feature output by the second feature extraction module into the modal classifier to perform modal classification, and constructing a modal classification loss function;

[0113] processing the feature output by the second feature extraction module through the plurality of convolution layers and one second global average pooling to obtain a learnable visual feature;

[0114] combining the histogram of oriented gradient features and the texture features extracted from the visible light image and the infrared image into a target domain prior vector;

[0115] based on the learnable visual feature and the target domain prior vector, constructing a prior loss function;

[0116] processing the feature output by the second feature extraction module through one second global average pooling to obtain a deep learning feature;

[0117] constructing a pseudo perception module based on the UNet network, and performing feature extraction and feature fusion on the visible light image and the infrared image through the pseudo perception module to obtain image fusion encoding features;

[0118] based on the deep learning feature and the image fusion encoding feature, constructing a pseudo perception loss function;

[0119] The learnable visual feature is subjected to a second global average pooling processing to obtain an average pooling result, and the average pooling result and the deep learning feature are spliced to obtain a feature finally output by the shared feature branch;

[0120] A second pedestrian re-identification loss function is constructed according to the feature finally output by the shared feature branch.

[0121] The modal classification loss function, the prior loss function, the pseudo-perception loss function and the second pedestrian re-identification loss function are taken as the loss functions corresponding to the shared feature branch.

[0122] In the embodiment, by inputting the feature output by the second feature extraction module into the modal classifier for modal classification and constructing the modal classification loss function, the decoupling and separation of the modal-specific feature and the shared modal feature can be realized, and the mutual interference between the modal characteristics and the shared semantics in the traditional method is avoided. Then, based on the learnable visual feature and the target domain prior vector, the prior loss function is constructed, the domain prior feature embedding module is introduced to enhance the feature representation capability of the infrared modal, the manually designed visual descriptor is fused with the deep feature to inject the prior knowledge with physical meaning into the target network model, and the insufficient representation capability caused by the weak texture of the infrared image is made up. Based on the deep learning feature and the image fusion coding feature, the pseudo-perception loss function is constructed, which can minimize the L2 distance between the shared feature and the fusion coding feature, construct an effective supervision signal in the absence of real fusion data, and guide the shared feature to learn the cross-modal structural consistency. Finally, the modal classification loss function, the prior loss function, the pseudo-perception loss function and the second pedestrian re-identification loss function are taken as the loss functions corresponding to the shared feature branch, which can lay a good data foundation for later construction of a target loss function that considers comprehensively.

[0123] In some embodiments, based on the feature finally output by the specific feature branch and the feature finally output by the shared feature branch, a spherical contrast distillation loss function is constructed, including:

[0124] ;

[0125] wherein, the spherical contrast distillation loss function is represented by Ls, the spherical projection head is represented by fsp, the feature finally output by the specific feature branch corresponding to the visible light image is represented by fvis, the feature finally output by the shared feature branch corresponding to the visible light image is represented by fvis-sh, the feature finally output by the specific feature branch corresponding to the infrared image is represented by fir, the feature finally output by the shared feature branch corresponding to the infrared image is represented by fir-sh, the L2 norm is represented by ||·||2.

[0126] In the embodiment, the spherical contrast distillation loss function guides the shared feature branch to capture the discriminative knowledge from the specific feature branch, which can lay a good data foundation for subsequent cross-modal image matching, thereby improving the accuracy of cross-modal image matching.

[0127] The spherical projection head can be composed of two layers of perception mechanisms. Unlike ordinary multi-layer perception, the special design introduces spherical logarithmic mapping and exponential mapping mechanism, so that the features can be linearly transformed after being projected to the spherical tangent space and then mapped back to the sphere. This design ensures the consistency of the direction of the features on the unit sphere, which is conducive to modeling the angle between high-dimensional features, thereby providing a stable and geometrically consistent representation basis for subsequent spherical contrast distillation.

[0128] The spherical contrast distillation can be a mapping of the features to the spherical space. The spherical contrast distillation is a distillation method for aligning cross-modal features, and the core idea is to pull the features of the same identity and different modalities closer in the spherical space, and pull the features of different identities farther away, thereby enhancing the discriminability of the model when matching. At the same time, the discriminative knowledge of the specific feature branch is guided into the shared feature branch.

[0129] In some embodiments, based on the results output by the first classifier and the results output by the second classifier, a fine-grained distillation loss function is constructed, including:

[0130] The results output by the first classifier are divided into a plurality of first spatial units, and the results output by the second classifier are divided into a plurality of second spatial units;

[0131] For each first spatial unit, a local logit corresponding to the specific feature branch is calculated, obtaining a plurality of first local logits;

[0132] For each second spatial unit, a local logit corresponding to the shared feature branch is calculated, obtaining a plurality of second local logits;

[0133] The KL divergence between the first local logit and the second local logit corresponding to the same spatial unit under the same modality is calculated;

[0134] Based on the KL divergence, a fine-grained distillation loss function is constructed.

[0135] In the embodiment, the fine-grained distillation loss function is constructed to focus on the classification prediction value of the classifier and align at the fine-grained level, thereby extracting key semantic knowledge in tasks with small inter-class differences, which can lay a good data foundation for subsequent cross-modal image matching, thereby improving the accuracy of cross-modal image matching.

[0136] For the convenience of those skilled in the art to understand, a set of best embodiments is provided below:

[0137] Visible light pedestrian image and infrared pedestrian image matching as an important research direction in the field of cross-modal recognition aims to match the same pedestrian in the images obtained by different spectral imaging devices at different times of day and night. Since infrared images have stronger robustness at night, visible light images contain more rich texture and color information, therefore, how to realize effective feature matching and identity recognition between the two modalities becomes a challenge. The existing technology mainly solves the modal difference problem through image-level conversion or feature-level alignment, but there are still many bottlenecks.

[0138] The image-level conversion method reduces the pixel domain difference through modal migration or image fusion, although it alleviates the apparent difference between modalities to some extent, but it is easy to introduce noise information such as artifacts and structural distortion, which affects the recognition accuracy. The feature-level alignment method usually constructs a double-flow neural network to extract shared embedding features between modalities, and realizes feature alignment through projection learning or modal confrontation. However, most methods emphasize modal invariance at the expense of modal-specific discriminative clues, especially in infrared images, the identity information is weakened due to the lack of texture. In addition, due to the high cost of cross-modal pedestrian re-identification (ReID) labeling, the existing data sets are generally small in size, resulting in insufficient training supervision signals, and the model is prone to overfitting and poor robustness. Therefore, it is urgent to propose an efficient recognition method with modal decoupling ability, fusion of multiple supervision forms and suitable for complex environments.

[0139] Image matching as a key technology for all-weather intelligent monitoring needs to realize pedestrian identity matching between visible light and infrared modalities with different spectral characteristics, but it faces challenges including: the cross-modal apparent difference leads to the visual feature difference of the same pedestrian in different modalities far exceeding the individual difference, the traditional method is easy to lose the modal-specific discriminative clues through the strategy of projecting features to the same space; the existing framework excessively pursues modal invariance, ignores the modal-specific “identity perception information”, resulting in blurred feature representation and weakened identity discrimination ability; in addition, the infrared imaging has inherent lack of texture and structural details, combined with limited cross-modal labeled data, making the shared feature learning lack clear supervision and easily degenerate into modal-independent noise.

[0140] The embodiment proposes a visible light-infrared image matching method based on a multi-feature fusion network with domain prior, aiming to solve the core problems of significant modal difference, identity discriminative feature loss and insufficient supervision signal in cross-modal recognition. The method combines the multi-feature fusion network with domain prior modeling, pseudo-perception supervision and multi-level distillation mechanism, which can break through the limitations of traditional methods in identity discrimination ability, modal alignment accuracy and insufficient training data.

[0141] The embodiment method specifically includes the following contents:

[0142] To solve the problems of too large modal difference, lack of identity discrimination information and insufficient supervision in existing visible light image and infrared image matching methods, an embodiment proposes a visible light-infrared image matching method based on domain prior multi-feature fusion network. Referring to Figure 2 , the method adopts a double-branch network structure, extracts modal-specific features and modal-shared features respectively, and combines a knowledge-driven domain prior embedding module and a pseudo-perception module to realize modal separation and cross-modal consistency alignment. In view of the problems of insufficient texture information of infrared images, lack of supervised data and significant difference between modal features, a domain prior constraint mechanism is proposed to fuse hand-crafted descriptors and deep features, and a pseudo-perception module is introduced to introduce intermediate supervision information, thereby improving the perception ability of the target network model to identity features under low light conditions. In addition, in the implicit knowledge distillation part, the contrast distillation and fine-grained semantic distillation strategies in the spherical space are combined, so that the target network model learns more compact and discriminative shared embedding features. The method proposed in the embodiment has strong generalization ability and practical applicability. Specifically, it includes:

[0143] 1. First, process the cross-modal pedestrian dataset containing visible light images and infrared images, and divide the visible light pedestrian images and infrared pedestrian images according to the camera number. The embodiment can perform image matching on visible light pedestrian images and infrared pedestrian images, or perform image matching on visible light images and infrared images in other fields, for example, visible light images and infrared images of vehicles, which are not limited in the embodiment.

[0144] 2. Input the visible light image and the infrared image into the double-branch network structure (i.e. the target network model) based on the ResNet-50 network. The visible light image and the infrared image are first extracted by the first three blocks (i.e. Block1, Block2 and Block3 in Figure 2 ) in the ResNet-50 network to extract preliminary features. Then, through two independent branches, the specific feature branch extracts modal-specific features (i.e. extracts through the first feature extraction module (including Block4 and Block5 in ResNet-50)), and the shared feature branch extracts modal-shared features (i.e. extracts through the second feature extraction module (including Block4 and Block5 in ResNet-50)).

[0145] 3. To separate the representations of specific modal and shared modal, two complementary loss functions are introduced to The process involves applying a modality discriminator to a specific feature branch to classify the modality of each sample. The corresponding modality discriminant loss function is defined as follows: A modality classifier is used on the shared feature branch, and the corresponding modality classification loss function is defined as follows: The formulas for the two loss functions are shown below:

[0146] (1);

[0147] (2);

[0148] in, This indicates that for each input sample image, there is a modality label. 0 represents the visible light mode, and 1 represents the infrared mode. It is a modal discriminator. It is a modality classifier. This represents the output of the softmax classifier. Represents the first in a specific feature branch Modal-specific features of an image, Indicating the first branch in the shared feature branch Modal sharing features of images This indicates the number of images for a single modality in a single sample. Typically, the number of images for both modalities is equal in a single sample. That is, the total number of images sampled in one operation. This represents the number of images of each class in a single modality. In modality classification loss... This represents the output of the gradient inversion layer. The gradient inversion layer is introduced here to incentivize the shared feature branches to learn modality-invariant features.

[0149] 4. Modal-specific features In the information purification process, specific features are divided into two branches. One branch first undergoes instance normalization followed by masking and convolution operations, while the other branch directly performs convolution and masking operations. Finally, both branches undergo a second convolution and are then concatenated to obtain the feature. (That is, the features output by the information purification module). Then, the features are obtained through a global average pooling layer (i.e., the first global average pooling). (That is, the features ultimately output by a specific feature branch), which serve as the input for the specific feature branch in the subsequent knowledge distillation module. A schematic diagram of the information purification process is shown below. Figure 3 As shown.

[0150] First, for modality-specific features Normalized features are obtained by applying instance normalization. This leads to another branch;

[0151] Then, two channel masks are generated by using a convolutional neural network and The features in the two branches and are respectively processed by the mask operation to obtain the first feature with stronger discriminability (i.e., the first feature) and the second feature with smaller modal difference (i.e., the second feature), which are formulated as follows:

[0152] (3);

[0153] Finally, the normalized branches and the original branches are integrated alternately, and the mask is applied to and the mask is applied to . After that, the results of the mask processing are spliced and pooled to obtain the modal-specific features after information purification. This process can be formulated as follows:

[0154] (4);

[0155] In the above two formulas, is the pooling operation, is the convolution operation.

[0156] 5. Modal shared features (i.e., the features output by the second feature extraction module) are also divided into two branches as shown in Figure 2 , one branch is directly subjected to global average pooling (i.e., the second global average pooling) to obtain deep learning features . The other branch is first processed by three consecutive 3x3 convolution layers (i.e., Conv3x3), and then subjected to global average pooling (i.e., the second global average pooling) to generate a learnable visual feature , which is formulated as follows:

[0157] (5);

[0158] wherein, denotes global average pooling, denotes a convolution layer.

[0159] 6. To embed the modality-aware prior into the target network model, physically meaningful handcrafted features were directly extracted from the original image, including Histogram of Oriented Gradients (HOG) features and texture features based on the Gray-Level Co-occurrence Matrix (GLCM). It should be noted that the HOG and GLCM texture features directly utilize existing techniques for image feature extraction; this embodiment does not impose specific limitations or descriptions. These features are combined into a target domain prior vector. Prior loss is measured by L2 distance from the visual descriptor. This forms domain prior supervision, which in turn embeds modality-aware priors into the model. Prior loss function. The formula is shown below:

[0160] (6);

[0161] in, This represents the number of features in the prior vector of the target domain. Indicates the first A learnable visual feature.

[0162] 7. Another deep learning feature in modality sharing features Image-based coded features Spurious sensing loss is obtained by using the L2 norm. Supervision is achieved by minimizing the distance between the two, using the pseudo-perceptual loss function. The formula is shown below:

[0163] (7);

[0164] in, Indicates the number of features. Indicates the first A deep learning feature, Indicates the first Image fusion coding features, image fusion coding features This is a fusion representation combining visible light and infrared modal complementary fusion information generated by a state-of-the-art (SOTA) image fusion model based on the UNet architecture. It should be noted that the state-of-the-art (SOTA) image fusion model based on the UNet architecture (i.e., the pseudo-perception module) is prior art known to those skilled in the art, and will not be specifically described in this embodiment. The architecture of the image fusion model is as follows: Figure 4 As shown.

[0165] 8. Learnable visual features in modality sharing After global average pooling (i.e., the second global average pooling), and combined with deep learning features Features are obtained by splicing. (i.e. the features output by the specific feature branch), and pedestrian re-identification (ReID) loss functions are applied to the specific feature branch and the shared feature branch respectively to obtain corresponding loss values (i.e. the loss value obtained by the first pedestrian re-identification loss function) and (i.e. the loss value obtained by the second pedestrian re-identification loss function), wherein the first pedestrian re-identification loss function and the second pedestrian re-identification loss function are both pedestrian re-identification (ReID) loss functions known to those skilled in the art, and the present embodiment will not be described in detail. The features separated from the two branches are domain prior feature embedding modules of the entire model, and the loss of this part is defined as , and the formula is as follows:

[0166] (8).

[0167] 9. The spherical mapping module and the multi-scale logit classification module are connected at the end of the specific feature branch and the shared feature branch respectively to perform implicit knowledge distillation. The implicit knowledge distillation is subdivided into spherical contrast distillation and fine-grained distillation. In the distillation process, the spherical space cross-modal alignment loss function (i.e. the spherical contrast distillation loss function) and the fine-grained distillation loss function are introduced for optimization, and the formula is as follows:

[0168] (9);

[0169] wherein, represents the visible light modality, represents the infrared modality, represents the unique features of the visible light image (the unique features are the features output by the specific feature branch), represents the shared features of the visible light image (the shared features are the features output by the shared feature branch), represents the unique features of the infrared image, represents the shared features of the infrared image, represents the spherical projection head, which is essentially a variant of the multi-layer perception machine. In the fine-grained distillation process, two classifiers (for example, neural networks, logistic regression, etc.) are introduced to enhance the ability to distinguish between blurred samples in the person re-identification process. The first classifier is used for the specific feature branch, and the second classifier is used for the shared feature branch, and the logical outputs of the two classifiers are represented as

[0170] (10);

[0171] Fine-grained distillation (FGD) strategy decomposes the original output logits of features after passing through the classifier (i.e., the original output values of the classifier after linear transformation of the input features) into multiple local logits representations, enabling shared feature branches to capture more detailed semantic information. These local logits are then distilled independently. Let and denote the logits maps of specific modalities and shared modalities, respectively. At each scale m, the logits map is divided into multiple spatial units and average pooling is performed within each unit, resulting in local logits outputs. For the nth unit at scale m, the local logits of specific feature branches and shared feature branches are calculated as follows:

[0172] (11) ;

[0173] (12) ;

[0174] where, denotes the spatial region of the nth unit at scale m, is the coordinate within the unit. For each pair of local logits, the distillation loss is defined as the KL divergence between their softened normalized values:

[0175] (13) ;

[0176] where, denotes the softmax function, denotes the local logits of specific feature branches under the visible light modality, denotes the local logits of shared feature branches under the visible light modality, denotes the local logits of specific feature branches under the infrared modality, denotes the local logits of shared feature branches under the infrared modality, denotes the KL divergence, and the final FGD loss formula is:

[0177] (14) ;

[0178] where, represents the set of all scales, then denotes the set of all cells corresponding to scale m.

[0179] 10. Finally, end-to-end training is achieved through the combination of multiple losses. The total loss consists of four parts: the loss of the domain prior feature embedding module , the loss of the pseudo perception module and two-part loss of the tacit knowledge distillation module, i.e. spherical contrast distillation loss and fine-grained distillation loss By calculating the total loss and back propagation, the parameters are updated. The final total loss (i.e. target loss function) is defined as follows:

[0180] (15).

[0181] wherein, , and are weight coefficients, which can be changed according to actual conditions.

[0182] 11. After the target network model is trained by the loss function and the plurality of visible light images and the plurality of infrared images used for model training, the image matching of the target visible light image and the target infrared image to be matched is performed by the shared feature branch in the trained target network model, and the image matching result is obtained, so as to realize high-precision cross-modal image matching. The shared feature branch can perform retrieval or distance calculation according to the extracted modal shared features , thereby realizing image matching.

[0183] Compared with the prior art, the method of the embodiment has the following advantages:

[0184] The method of the embodiment adopts an innovative double-branch architecture, realizes decoupling separation of modal-specific features and shared modal features through adversarial training of a modal discriminator and a modal classifier, and avoids mutual interference between modal characteristics and shared semantics in traditional methods. To enhance the feature representation capability of the infrared modal, the method introduces a domain prior feature embedding module, fuses a manually designed visual descriptor with a deep feature, injects prior knowledge with physical meaning into the network, and makes up for the insufficient representation capability caused by weak texture of the infrared image. In view of the problem of lack of supervision signal of the shared feature, the method designs a pseudo perception module, generates fusion encoding features of a visible light image and an infrared image pair by using an SOTA image fusion model, minimizes the L2 distance between the shared feature and the fusion encoding features, constructs an effective supervision signal in the absence of real fusion data, and guides the shared feature to learn the cross-modal structural consistency. To further improve the discriminability and cross-modal consistency of the shared feature, the method proposes a spherical contrast distillation and fine-grained distillation strategy, enhances the feature compactness, realizes fine-grained semantic alignment, and ensures the discriminability of the shared feature in the subtle difference between classes. The method integrates a modal separation loss, a domain prior loss, a pseudo perception loss, and a distillation loss to form a total loss function optimized in an end-to-end manner, drives the target network model to learn a shared representation with modal invariance and identity distinguishability while retaining modal-specific discriminative information. Through the cooperation of data-driven and model-driven, the method balances the uniqueness and cross-modal consistency of the feature, effectively improves the pedestrian re-identification precision and robustness in complex scenes such as low light, occlusion, and view change, and provides an efficient technical solution for all-weather intelligent security, multi-modal trajectory reconstruction and other applications.

[0185] Reference Figure 5 The embodiment of the present application provides a kind of image matching system based on multi-feature fusion, which comprises image acquisition unit 501, model construction unit 502, function construction unit 503, model training unit 504 and image matching unit 505, wherein:

[0186] Image acquisition unit 501 is used to obtain target visible light image and target infrared image to be matched, and obtain multiple visible light images and multiple infrared images for model training;

[0187] Model construction unit 502 is used to construct target network model containing specific feature branch and shared feature branch based on ResNet-50 network, and access first classifier at the end of specific feature branch and access second classifier at the end of shared feature branch;Wherein, specific feature branch includes first feature extraction module, information purification module and first global average pooling, and shared feature branch includes second feature extraction module, multiple convolution layers and multiple second global average pooling;

[0188] The function construction unit 503 is configured to construct a target loss function based on the features output by the first classifier, the second classifier, the first feature extraction module, the information purification module, the first global average pooling, the second feature extraction module, the plurality of convolutional layers, and the plurality of second global average poolings.

[0189] The model training unit 504 is configured to train the target network model based on the target loss function, using the plurality of visible light images and the plurality of infrared images, to obtain a trained target network model.

[0190] The image matching unit 505 is configured to perform image matching on the target visible light image and the target infrared image by using the shared feature branch in the trained target network model, to obtain an image matching result.

[0191] It should be noted that, since the image matching system based on multi-feature fusion in the embodiment and the image matching method based on multi-feature fusion described above are based on the same inventive concept, the corresponding content in the method embodiment is also applicable to the system embodiment, and thus will not be described in detail herein.

[0192] With reference to Figure 6 The embodiment of the present application further provides an electronic device, and the electronic device comprises:

[0193] at least one memory;

[0194] at least one processor;

[0195] at least one program;

[0196] The program is stored in the memory, and the processor executes the at least one program to implement the image matching method based on multi-feature fusion described above.

[0197] The electronic device can be any intelligent terminal, such as a mobile phone, a tablet computer, a personal digital assistant (PDA), a vehicle-mounted computer, etc.

[0198] The electronic device of the embodiment of the present application will be described in detail below.

[0199] The processor 1600 can be implemented in a general central processing unit (CPU), a microprocessor, an application specific integrated circuit (ASIC), or one or more integrated circuits, and is configured to execute a related program to implement the technical solutions provided by the embodiments of the present application.

[0200] The memory 1700 can be implemented in the form of a Read Only Memory (ROM), a static storage device, a dynamic storage device, or a Random Access Memory (RAM), etc. The memory 1700 can store an operating system and other application programs. When the technical solutions provided in the embodiments of the present disclosure are implemented by software or firmware, the related program codes are stored in the memory 1700 and are invoked and executed by the processor 1600 to implement the multi-feature fusion based image matching method of the embodiments of the present disclosure.

[0201] The input / output interface 1800 is configured to realize information input and output.

[0202] The communication interface 1900 is configured to realize the communication interaction between the device and other devices. The communication can be realized in a wired manner (for example, a USB, a network cable, etc.) or in a wireless manner (for example, a mobile network, WIFI, Bluetooth, etc.).

[0203] The bus 2000 is configured to transmit information between various components (for example, the processor 1600, the memory 1700, the input / output interface 1800, and the communication interface 1900) of the device.

[0204] The processor 1600, the memory 1700, the input / output interface 1800, and the communication interface 1900 are connected to each other through the bus 2000 to realize the communication connection between them in the device.

[0205] The present disclosure further provides a storage medium, which is a computer readable storage medium, and stores computer executable instructions for causing a computer to execute the multi-feature fusion based image matching method.

[0206] The memory is a non-transitory computer readable storage medium, which can be used to store non-transitory software programs and non-transitory computer executable programs. In addition, the memory can include a high-speed random access memory and can also include a non-transitory memory, such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state memory device. In some embodiments, the memory can optionally include a memory remotely arranged relative to the processor, and these remote memories can be connected to the processor through a network. Examples of the above network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0207] The embodiments described in the present disclosure are to more clearly illustrate the technical solutions of the present disclosure, and do not constitute a limitation on the technical solutions provided by the present disclosure. Those skilled in the art can know that, as technology evolves and new application scenarios appear, the technical solutions provided by the present disclosure are also applicable to similar technical problems.

[0208] Those skilled in the art can understand that the technical solutions shown in the figures do not constitute a limitation on the present disclosure, and can include more or fewer steps than shown in the figures, or combine certain steps, or different steps.

[0209] The device embodiments described above are merely illustrative, and units described as separate components can or can not be physically separated, that is, can be located in one place, or can be distributed on multiple network units. Part or all of the modules can be selected according to actual needs to achieve the purpose of the present embodiment.

[0210] Those skilled in the art can understand that all or some steps in the above disclosed method, the functions of the modules / units in the system and the device can be implemented as software, firmware, hardware and their appropriate combinations.

[0211] The terms "first", "second", "third", "fourth" and the like (if any) in the specification of the present application and the above-described drawings are used to distinguish similar objects, and do not necessarily have to describe a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device including a series of steps or units does not have to be limited to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0212] It should be understood that, in the application, "at least one" refers to one or more, and "multiple" refers to two or more. "And / or" is used to describe the association relationship of the associated objects, which means that there can be three relationships, for example, "A and / or B" can represent three cases of only A, only B and A and B existing at the same time, wherein A and B can be singular or plural. The character " / " generally represents an "or" relationship between the associated objects before and after it. "At least one of the following" or similar expressions means any combination of these items, including any combination of single or multiple items. For example, at least one of a, b or c can represent a, b, c, "a and b", "a and c", "b and c", or "a and b and c", wherein a, b and c can be single or multiple.

[0213] In several embodiments provided in the application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of units is only a logical function division, and actual implementation can have another division manner, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the displayed or discussed mutual units can be indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0214] The units described as separate components can or can not be physically separated, and the components displayed as units can or can not be physical units, that is, they can be located in one place, or they can be distributed on multiple network units. According to actual needs, part or all of the units can be selected to achieve the purpose of the embodiment scheme.

[0215] In addition, the functional units in each embodiment of the application can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0216] The integrated unit, if implemented in the form of a software function unit and sold or used as an independent product, can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application essentially or say the part that contributes to the prior art or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes multiple instructions for causing an electronic device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (Read-Only Memory, ROM), a random access memory (Random Access Memory, RAM), a magnetic disk or an optical disk, and various program storage media. The above embodiments of the present application are described in detail in combination with the drawings, but the present application is not limited to the above embodiments, and various changes can be made within the knowledge range of ordinary skilled persons in the art without departing from the purpose of the present application.

[0217] The above embodiments of the present application are described in detail in combination with the drawings, but the present application is not limited to the above embodiments, and various changes can be made within the knowledge range of ordinary skilled persons in the art without departing from the purpose of the present application.

Claims

1. An image matching method based on multi-feature fusion, characterized in that, The method includes: Acquire the visible light image and infrared image of the target to be matched, and acquire multiple visible light images and multiple infrared images for model training; Based on the ResNet-50 network, a target network model is constructed that includes specific feature branches and shared feature branches. A first classifier is connected at the end of the specific feature branches, and a second classifier is connected at the end of the shared feature branches. The specific feature branches include a first feature extraction module, an information purification module, and a first global average pooling. The shared feature branches include a second feature extraction module, multiple convolutional layers, and multiple second global average pooling. Based on the features output by the first classifier, the second classifier, the first feature extraction module, the information purification module, the first global average pooling, the second feature extraction module, the multiple convolutional layers, and the multiple second global average pooling layers, a target loss function is constructed, including: Based on the features output by the first feature extraction module, the information purification module, and the first global average pooling, a loss function corresponding to the specific feature branch is constructed. Based on the features output by the second feature extraction module, the multiple convolutional layers, and the multiple second global average pooling layers, a loss function corresponding to the shared feature branch is constructed. Based on the features finally output by the specific feature branch and the features finally output by the shared feature branch, a spherical contrastive distillation loss function is constructed, including: ； in, This represents the loss function of spherical contrast distillation. Indicates spherical projection head, This represents the final output feature of a specific feature branch corresponding to a visible light image. This represents the final output feature of the shared feature branch corresponding to the visible light image. This represents the final output feature of a specific feature branch corresponding to an infrared image. This represents the feature output by the shared feature branch corresponding to the infrared image. Represents the L2 norm; Based on the results of the first classifier and the second classifier, a fine-grained distillation loss function is constructed. A target loss function is constructed by combining the loss function corresponding to the specific feature branch, the loss function corresponding to the shared feature branch, the spherical contrast distillation loss function, and the fine-grained distillation loss function. Based on the target loss function, the target network model is trained using the multiple visible light images and the multiple infrared images to obtain the trained target network model; Image matching is performed on the target visible light image and the target infrared image by using the shared feature branches in the trained target network model to obtain the image matching result.

2. The image matching method based on multi-feature fusion according to claim 1, characterized in that, The step of constructing the loss function corresponding to the specific feature branch based on the features output by the first feature extraction module, the information purification module, and the first global average pooling includes: The features output by the first feature extraction module are input into the modality discriminator for modality classification, and a modality discrimination loss function is constructed. The features output by the first feature extraction module are input into the information purification module for information purification, and the features output by the information purification module are obtained. The features output by the information purification module are processed by the first global average pooling operation to obtain the final output features of the specific feature branch. Based on the features finally output by the specific feature branch, a first pedestrian re-identification loss function is constructed. The modality discrimination loss function and the first pedestrian re-identification loss function are used as the loss functions corresponding to the specific feature branch.

3. The image matching method based on multi-feature fusion according to claim 2, characterized in that, The information purification module includes instance normalization, convolutional neural network, multiple convolution operations, and pooling operations. The step of inputting the features output by the first feature extraction module into the information purification module for information purification, resulting in features output by the information purification module, includes: The features output by the first feature extraction module are normalized by instance to obtain normalized features; Based on the normalized features and the features output by the first feature extraction module, the convolutional neural network generates a first mask corresponding to the features output by the first feature extraction module and a second mask corresponding to the normalized features. The first feature is obtained by convolving the features output by the first feature extraction module with the first mask. The normalized feature and the second mask are convolved to obtain the second feature; The first feature and the second mask are convolved to obtain the third feature; The second feature and the first mask are convolved to obtain the fourth feature; The third feature and the fourth feature are concatenated to obtain the concatenated feature; The spliced ​​features are pooled to obtain the features output by the information purification module.

4. The image matching method based on multi-feature fusion according to claim 1, characterized in that, The step of constructing the loss function corresponding to the shared feature branch based on the features output by the second feature extraction module, the multiple convolutional layers, and the multiple second global average pooling layers includes: The features output by the second feature extraction module are input into the modality classifier for modality classification, and a modality classification loss function is constructed. The features output by the second feature extraction module are processed through the multiple convolutional layers and a second global average pooling to obtain learnable visual features. The directional gradient histogram features and texture features extracted from visible light images and infrared images are combined into a target domain prior vector; Based on the learnable visual features and the target domain prior vector, a prior loss function is constructed; The features output by the second feature extraction module are processed by a second global average pooling process to obtain deep learning features; A pseudo-perception module is constructed based on the UNet network, and feature extraction and feature fusion are performed on visible light images and infrared images through the pseudo-perception module to obtain image fusion coding features; Based on the deep learning features and the image fusion coding features, a pseudo-perceptual loss function is constructed; The learnable visual features are processed by a second global average pooling process to obtain the average pooling result. The average pooling result is then concatenated with the deep learning features to obtain the final output features of the shared feature branch. Based on the features finally output by the shared feature branch, a second pedestrian re-identification loss function is constructed; The modality classification loss function, the prior loss function, the pseudo-perception loss function, and the second pedestrian re-identification loss function are used as the loss functions corresponding to the shared feature branch.

5. The image matching method based on multi-feature fusion according to claim 1, characterized in that, The step of constructing a fine-grained distillation loss function based on the results output by the first classifier and the second classifier includes: The output of the first classifier is divided into multiple first spatial units, and the output of the second classifier is divided into multiple second spatial units; For each first spatial unit, calculate the local logit corresponding to a specific feature branch to obtain multiple first local logits; For each second spatial unit, calculate the local logit corresponding to the shared feature branch; thus obtaining multiple second local logits. Calculate the KL divergence between the first local logit and the second local logit corresponding to the same spatial cell in the same mode; Based on the KL divergence, a fine-grained distillation loss function is constructed.

6. An image matching system based on multi-feature fusion, characterized in that, The system includes: The image acquisition unit is used to acquire the visible light image and infrared image of the target to be matched, and to acquire multiple visible light images and multiple infrared images for model training; The model building unit is used to build a target network model based on the ResNet-50 network, which includes a specific feature branch and a shared feature branch. A first classifier is connected to the end of the specific feature branch, and a second classifier is connected to the end of the shared feature branch. The specific feature branch includes a first feature extraction module, an information purification module, and a first global average pooling. The shared feature branch includes a second feature extraction module, multiple convolutional layers, and multiple second global average pooling. The function construction unit is used to construct a target loss function based on the features output by the first classifier, the second classifier, the first feature extraction module, the information purification module, the first global average pooling, the second feature extraction module, the plurality of convolutional layers, and the plurality of second global average pooling layers, including: Based on the features output by the first feature extraction module, the information purification module, and the first global average pooling, a loss function corresponding to the specific feature branch is constructed. Based on the features output by the second feature extraction module, the multiple convolutional layers, and the multiple second global average pooling layers, a loss function corresponding to the shared feature branch is constructed. Based on the features finally output by the specific feature branch and the features finally output by the shared feature branch, a spherical contrastive distillation loss function is constructed, including: ； in, This represents the loss function of spherical contrast distillation. Indicates spherical projection head, This represents the final output feature of a specific feature branch corresponding to a visible light image. This represents the final output feature of the shared feature branch corresponding to the visible light image. This represents the final output feature of a specific feature branch corresponding to an infrared image. This represents the feature output by the shared feature branch corresponding to the infrared image. Represents the L2 norm; Based on the results of the first classifier and the second classifier, a fine-grained distillation loss function is constructed. A target loss function is constructed by combining the loss function corresponding to the specific feature branch, the loss function corresponding to the shared feature branch, the spherical contrast distillation loss function, and the fine-grained distillation loss function. The model training unit is used to train the target network model based on the target loss function using the multiple visible light images and the multiple infrared images, so as to obtain the trained target network model. The image matching unit is used to perform image matching on the target visible light image and the target infrared image through the shared feature branches in the trained target network model to obtain the image matching result.

7. An electronic device, characterized in that, It includes at least one control processor and a memory for communicatively connecting to the at least one control processor; the memory stores instructions executable by the at least one control processor, which, when executed by the at least one control processor, enable the at least one control processor to perform the image matching method based on multi-feature fusion as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions for causing a computer to perform the image matching method based on multi-feature fusion as described in any one of claims 1 to 5.

Images