Cross-modal feature complementation and feature projection network system

Abstract

The invention relates to the field of computer vision, and discloses a cross-modal feature complementation and feature projection network system, which comprises a modal complementation module, a cross-modal feature projection module, a multi-channel feature mining module and a training and optimizing module, and is characterized in that the modal complementation module is used for reducing the modal difference between an infrared light IR image and a visible light VI S image; the cross-modal feature projection module is used for projecting feature data of different modals into a shared feature space; and the multi-channel feature mining module is used for further mining and aggregating multi-level and multi-channel cross-modal features in the shared feature space. Through the modal complementation module, the modal difference between the VI S image and the IR image can be effectively reduced, the features of coarse granularity and fine granularity are extracted by adopting a cavity convolution mode and a small convolution kernel convolution mode, modal complementation is carried out through feature weights, the complementation of modal features is realized on the premise that key information is not lost, and the accuracy of the modal complementation is improved. Therefore, the similarity of cross-modal features is greatly improved.

Description

Technical Field

[0001] The present invention relates to the field of computer vision, and in particular to a cross-modal feature complementation and feature projection network system. Background Art

[0002] Person Re-identification (ReID) refers to the task of retrieving the same pedestrian from different cameras. This task is of great significance in smart city construction and public safety, and has attracted widespread attention in recent years. Most existing methods have studied visible light person re-identification and have achieved good research results in the virtual world.

[0003] However, in the real world, due to poor lighting conditions or at night, only infrared cameras can obtain poor quality infrared images. Due to the modal differences between visible light images and infrared images, conventional visible light pedestrian re-identification models may not work well. Most methods lack rich cross-modal feature correspondences and find it difficult to effectively project the features of VIS and IR images, resulting in significant modal differences and unsatisfactory recognition results. It is still challenging to build effective cross-modal feature complementation and feature projection methods. Summary of the invention

[0004] In view of the shortcomings of the prior art, the present invention provides a cross-modal feature complementation and feature projection network system, which solves the problems that the existing VIReID method only relies on feature extraction of a single modality, lacks rich cross-modal feature correspondences, and is difficult to effectively project the features of VIS and IR images, resulting in significant modal differences and unsatisfactory recognition effects.

[0005] To achieve the above objectives, the present invention is implemented through the following technical solutions: a cross-modal feature complementation and feature projection network system, including a modality complementation module, a cross-modal feature projection module, a multi-channel feature mining module and a training and optimization module, wherein the modality complementation module is used to reduce the modality difference between infrared light IR images and visible light VIS images, and the modality complementation module realizes the complementation between the two modalities by extracting and fusing the coarse-grained and fine-grained features of the two modalities, so that the data of the two modalities are more similar;

[0006] The cross-modal feature projection module is used to project feature data of different modalities into a shared feature space. By calculating the correspondence between local features, the module can effectively align VIS and IR features.

[0007] The multi-channel feature mining module is used to further mine and aggregate multi-level, multi-channel cross-modal features in a shared feature space. The multi-channel feature mining module enhances the richness and diversity of the overall feature representation by improving channel-level features.

[0008] Preferably, the modality complementation module includes a feature extraction layer and a modality complementation layer;

[0009] The feature extraction layer comprises:

[0010] Two convolutional neural networks are used to extract VIS image features and IR image features respectively;

[0011] Use a dilated convolutional neural network to extract coarse-grained and wide-view features to avoid the gridding effect problem. Use a dilated convolutional neural network with a dilation rate of (1,2,3,1,2,3) to extract coarse-grained and wide-view features.

[0012] Use small MLP to process high-dimensional feature data, learn more modal features, and extract generalized features through average pooling;

[0013] Use 1x1 convolution kernel to extract fine-grained features, learn modal feature data through small MLP, and then use maximum pooling to focus on extracting features with large modal differences.

[0014] Preferably, the modality complementation layer obtains the complementary feature data by performing dot multiplication of the coarse-grained feature weights and the fine-grained feature weights of the two modalities with the VIS and IR feature data respectively, and then adding them together.

[0015] Preferably, the cross-modal feature projection module includes a feature splitting layer, a covariance matrix calculation layer, and a feature projection layer;

[0016] The feature splitting layer splits the VIS and IR image feature data into multiple data blocks by row and column respectively;

[0017] The covariance matrix calculation layer is used to calculate the covariance between each data block and obtain the horizontal and vertical correspondence relationship of the local features;

[0018] The feature projection layer uses the horizontal and vertical cross-modal feature correspondences as projection coefficients and projects the features into a shared feature space through a linear function.

[0019] Preferably, the multi-channel feature mining module includes a channel promotion layer, a feature aggregation layer, and a double-layer aggregation layer;

[0020] The channel boosting layer uses a 1x1 convolutional layer to increase the number of channels of feature data and fold the width and height into one spatial dimension;

[0021] The feature aggregation layer calculates the channel space feature relationship matrix and multiplies it with the feature data to aggregate channel level features;

[0022] The double-layer aggregation adds the aggregated features to the original feature data to obtain the final output data.

[0023] Preferably, the training and optimization module uses a pre-trained ResNet-50 as a backbone network through a model, randomly and equally selects identity data as input training data for each training batch, uses an SGD optimizer, and for different data sets, uses a single-layer or double-layer modality complementary network combined with a cross-modal feature projection network.

[0024] Preferably, the training and optimization module uses a single-layer modality complementation network combined with a single-layer cross-modal feature projection network for the RegDB dataset;

[0025] For the SYSU-MM01 dataset, a two-layer modality complementarity network combined with a two-layer cross-modal feature projection network is used.

[0026] The present invention provides a cross-modal feature complementation and feature projection network system. It has the following beneficial effects:

[0027] 1. The present invention can effectively reduce the modal difference between VIS and IR images through the modal complementation module, extract coarse-grained and fine-grained features by using dilated convolution and small convolution kernel convolution, and perform modal complementation through feature weights to ensure that the complementarity of modal features is achieved without losing key information. This greatly improves the similarity of cross-modal features and lays a solid foundation for subsequent feature projection and recognition.

[0028] 2. The present invention projects feature data of different modalities into a shared feature space by calculating the correspondence of local features through a cross-modal feature projection module, and obtains the correspondence of cross-modal features through the splitting and covariance calculation of horizontal and vertical features, so that the projected features are consistent in the shared space. This method ensures the effective alignment and projection of cross-modal features and significantly reduces the impact of modal differences on recognition performance.

[0029] 3. The present invention improves channel-level features through a multi-channel feature mining module, further mines and aggregates multi-level cross-modal features, calculates the channel space feature relationship matrix, and aggregates double-layer channel-level features to enhance the richness and diversity of feature representation. This multi-level, multi-channel feature mining method enables the model to capture more details and difference information, thereby improving the accuracy of pedestrian re-identification.

[0030] 4. The present invention generates a multi-branch convolution structure, which can generate more embeddings when the training data is insufficient, thereby alleviating the problem of insufficient training data. At the same time, through data enhancement and random preprocessing techniques (such as random erasing and random horizontal flipping), the robustness and adaptability of the system in different environments are improved, ensuring the stability and reliability of the model in practical applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 Schematic diagram of the cross-modal feature complementation and feature projection model architecture of the present invention;

[0032] Figure 2 A schematic diagram of the cross-modal complementary module of the present invention using dual branches to extract features of different granularities as weights, and complementing the features of the two modalities according to the weights;

[0033] Figure 3 A schematic diagram of a cross-modal feature projection module of the present invention using local features to calculate cross-modal feature correspondences;

[0034] Figure 4 A schematic diagram of improving channel features by the multi-channel and multi-level feature mining module of the present invention;

[0035] Figure 5 A schematic diagram of a cross-modal feature complementation and feature projection network system architecture of the present invention;

[0036] Figure 6 It is a schematic diagram of the modal complementarity module architecture of the present invention;

[0037] Figure 7 Schematic diagram of the cross-modal feature projection module architecture of the present invention;

[0038] Figure 8 It is a schematic diagram of the architecture of the multi-channel feature mining module and the training and optimization module of the present invention. DETAILED DESCRIPTION

[0039] The technical solution of the present invention will be described clearly and completely below in conjunction with the accompanying drawings of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0040] Please see attached Figure 1 - Attachment Figure 8The embodiment of the present invention provides a cross-modal feature complementation and feature projection network system, including a modality complementation module, a cross-modal feature projection module, a multi-channel feature mining module and a training and optimization module, characterized in that the modality complementation module is used to reduce the modality difference between infrared light IR images and visible light VIS images, and the modality complementation module realizes the complementation between the two modalities by extracting and fusing the coarse-grained and fine-grained features of the two modalities, so that the data of the two modalities are more similar;

[0041] The cross-modal feature projection module is used to project the feature data of different modalities into a shared feature space. By calculating the correspondence between local features, this module can effectively align VIS and IR features.

[0042] The multi-channel feature mining module is used to further mine and aggregate multi-level, multi-channel cross-modal features in the shared feature space. The multi-channel feature mining module enhances the richness and diversity of the overall feature representation by improving the channel-level features.

[0043] The modality complementation module includes a feature extraction layer and a modality complementation layer;

[0044] The feature extraction layer includes:

[0045] Two convolutional neural networks are used to extract VIS image features and IR image features respectively;

[0046] Use a dilated convolutional neural network to extract coarse-grained and wide-view features to avoid the gridding effect problem. Use a dilated convolutional neural network with a dilation rate of (1,2,3,1,2,3) to extract coarse-grained and wide-view features.

[0047] Use small MLP to process high-dimensional feature data, learn more modal features, and extract generalized features through average pooling;

[0048] Use 1x1 convolution kernel to extract fine-grained features, learn modal feature data through small MLP, and then use maximum pooling to focus on extracting features with large modal differences.

[0049] The modal complementation layer obtains the complementary feature data by performing dot multiplication of the coarse-grained feature weights and fine-grained feature weights of the two modalities with the VIS and IR feature data respectively, and then adding them together.

[0050] The cross-modal feature projection module includes a feature splitting layer, a covariance matrix calculation layer, and a feature projection layer;

[0051] The feature splitting layer splits the VIS and IR image feature data into multiple data blocks by rows and columns respectively;

[0052] The covariance matrix calculation layer is used to calculate the covariance between each data block and obtain the horizontal and vertical correspondence of local features;

[0053] The feature projection layer uses the horizontal and vertical cross-modal feature correspondences as projection coefficients and projects the features into the shared feature space through a linear function.

[0054] The multi-channel feature mining module includes a channel enhancement layer, a feature aggregation layer, and a double-layer aggregation layer;

[0055] The channel boosting layer uses a 1x1 convolutional layer to increase the number of channels of the feature data and collapse the width and height into one spatial dimension;

[0056] The feature aggregation layer calculates the channel space feature relationship matrix and multiplies it with the feature data to aggregate channel-level features;

[0057] Double-layer aggregation adds the aggregated features to the original feature data to obtain the final output data.

[0058] The training and optimization module uses the pre-trained ResNet-50 as the backbone network through the model. Each training batch randomly and equally selects identity data as input training data, uses the SGD optimizer, and for different data sets, uses a single-layer or double-layer modality complementary network combined with a cross-modal feature projection network.

[0059] For the RegDB dataset, the training and optimization module uses a single-layer modality complementation network combined with a single-layer cross-modal feature projection network;

[0060] For the SYSU-MM01 dataset, a two-layer modality complementarity network combined with a two-layer cross-modal feature projection network is used.

[0061] The cross-modal feature complementation and feature projection model is built using the pytorch deep learning framework. The model uses the pre-trained ResNet-50 as the backbone network. Each training batch randomly and equally selects identity data as the input training data, and the input VIS image and IR image are randomly erased and randomly horizontally flipped. The SGD optimizer is used, and the learning rate is initially set to 0.1. The learning rate is reduced by 10 times every 20 rounds during training. There are two hyperparameters in the modality complementation module, λ1=0.001, λ2=0.001, which represent the modality complementation rate. For the RegDB dataset, a single-layer modality complementation network is used in combination with a single-layer cross-modal feature projection network. For the SYSU-MM01 dataset, a double-layer modality complementation network is used in combination with a double-layer cross-modal feature projection network.

[0062] Model Architecture

[0063] like Figure 1As shown in the figure, the cross-modal feature complementation and feature projection model architecture uses the ResNet-50 network as the backbone. More embeddings are generated through a multi-branch convolutional generation structure to alleviate the problem of insufficient training data. The VIS-IR features are input into the proposed modal complementation module to obtain VIS-IR feature data with smaller modal differences. This data is then input into the cross-modal feature projection module to project the data of the two modalities into the same modal feature shared space. A multi-channel feature mining module is used in the modal feature shared space to obtain more diverse features.

[0064] Figure 1 The proposed cross-modal feature complementation and feature projection model architecture uses a two-layer COM module for modal complementation, calculates the cross-modal feature correspondence (FC), uses a projection head (FP) to project the cross-modal features into the feature space, and uses the FEM module to mine multi-channel and multi-level features.

[0065] Figure 2 The cross-modal complementary module uses dual branches to extract features of different granularities as weights, and complements the features of the two modalities according to the weights.

[0066] The model first uses two convolutional neural networks to extract VIS image features and IR image features, and then passes the features into two modality complementation modules (COM) to perform modality complementation on cross-modal features. Figure 2 As shown in the figure, the modality complementarity module first uses two branches to extract features and obtain features of different granularities. The VIS image feature is defined as X V , define the IR image feature as X R . Branch 1: Use a dilated convolutional neural network to extract coarse-grained, wide-view features. In order to avoid the gridding effect problem, a convolutional neural network with a dilation coefficient of (1, 2, 3, 1, 2, 3) is used. Each layer of convolutional neural network is spliced ​​with a ReLU activation layer F ReLU To improve the ability to express nonlinear features. Then the hole convolutional neural network Conv D The definition of is shown in Formula 1.

[0067] Conv D =F ReLU (φ i ) Formula 1

[0068] Pass the obtained features into a small In the process, we process high-dimensional feature data, learn more modal features, and then use average pooling F avg_pool , extract generalized features.

[0069]

[0070] The eigenvalues ​​of the two modes and Add and average, and pass it into the Sigmoid activation function F sigmoid Get large-grained feature weights.

[0071]

[0072] Branch 2: Use a convolutional neural network Conv with a convolution kernel size of 1x1 to extract fine-grained features, splice the same small MLP to learn the modal feature data, and then use the maximum pooling F max_pool , focusing on extracting features with large modal differences.

[0073]

[0074] The eigenvalues ​​of the two modes and Add and average, and pass it into the Sigmoid activation function to obtain the small-grained feature weights.

[0075]

[0076] Finally, the coarse-grained feature weight W w and the fine-grained feature weight W m Multiply it by VIS (IR) feature data, and then add it to IR (VIS) feature data to get the feature data after modal complementation.

[0077] X' V =X V +λ1*W w *X R +λ2*W m *X R Formula 8

[0078] X' R =X R +λ1*W w *X V +λ2*W m *X V Formula 9

[0079] Where λ1 and λ2 are hyperparameters, representing the modal complementarity rate. v and X′ R That is, the VIS image features and IR image features output after modal complementarity.

[0080] Figure 3 The cross-modal feature projection module uses local features to calculate the cross-modal feature correspondence, and then projects the features of the two modalities into the feature space through the projection head.

[0081] The cross-modal feature projection module uses the original feature data and the feature data after modal complementation to calculate the cross-modal feature correspondence. Covariance is generally used to indicate the similarity between two random variables. The cross-modal feature fusion module uses two methods to obtain the cross-modal feature correspondence based on the covariance matrix and project the cross-modal features. Due to the large difference in modal features, multiple local features are used as the corresponding features of the two modalities, and the corresponding features are obtained based on the horizontal features and the vertical features. Figure 3 As shown, method 1: first divide the VIS image feature data into 4 data blocks according to the row The IR image feature data is also divided into 4 data blocks according to the row The VIS feature matrix block is shown in Matrix 1, and the IR feature matrix block is shown in Matrix 2

[0082]

[0083] Calculate the covariance of corresponding feature data of different modes

[0084]

[0085] The covariance matrix of the four lateral cross-modal features is calculated using formula 10: To obtain the lateral correspondence of local cross-modal features, in order to extend the local cross-modal feature correspondence to the overall cross-modal feature correspondence, the covariance matrices between the four lateral cross-modal feature data are added and the mean is calculated to obtain the cross-modal feature correspondence of the breadth feature.

[0086]

[0087] Method 2: Then divide the VIS image feature data into 4 data blocks by column The IR image feature data is also divided into 4 data blocks by column The VIS feature matrix block is shown in Matrix 3, and the IR feature matrix block is shown in Matrix 4

[0088]

[0089] Calculate the covariance of feature data with the same relative position but different modes.

[0090]

[0091] The covariance matrix of the four longitudinal cross-modal features is calculated using formula 12: The longitudinal local cross-modal feature correspondence is obtained, the covariance matrices between the four longitudinal cross-modal feature data are added, and the mean is calculated. The cross-modal feature correspondence of the overall deep feature is obtained from multiple local cross-modal feature correspondences.

[0092]

[0093] The lateral cross-modal feature correspondence matrix Cov is calculated by the above two methods. r And the longitudinal cross-modal feature correspondence matrix Cov c By using two simple projection components, the correspondence between the two cross-modal features is used as the projection coefficient, the two cross-modal features are used as parameters, and a linear function F is used. row Map the lateral cross-modal features and use another linear function F col Map the longitudinal cross-modal features and project the features of the two modalities. With the help of the correspondence between the two cross-modal features, the cross-modal features are more effectively projected into the feature sharing space, making the feature sharing space more compact, compensating for the feature loss caused by direct projection, and effectively reducing the modal difference.

[0094]

[0095] Where X′ represents the bimodal features finally projected into the feature shared space.

[0096] Figure 4 The multi-channel and multi-level feature mining module enhances channel features and aggregates double-layer features to mine diverse features.

[0097] The cross-modal features are projected into the modal feature shared space. In order to aggregate multi-level and multi-stage features and mine different channel feature representations, the patent proposes a multi-level and multi-channel feature mining module. Figure 4 As shown in the figure, the module consists of three small components. Component 1 is mainly responsible for calculating the relationship matrix used to aggregate multi-channel features, and components 2 and 3 aggregate two-layer channel-level features.

[0098] Image feature data f B×C×H×W There are 4 dimensions. To focus on mining multi-channel feature data, component 1 first uses 2 1x1 convolutional layers. Increase the number of channels of feature data and collapse the width and height of the feature into one spatial dimension f B×C′×(H×W) (C′>C). Then, the two feature data are multiplied by a matrix to obtain a channel space feature relationship matrix M based on the multi-channel feature and having (H×W) dimensional features in its space.

[0099]

[0100] Component 2 receives the original feature data f B×C×H×W , and passes through a 1x1 convolutional layer Extract multi-channel features, multiply the channel space feature relationship matrix with the feature data, aggregate channel-level features, and on the other hand, use a 1x1 convolution layer to convert the original feature data into Increase the number of channels again as the input data f of component 1 B×C″×(H×W) (C″>C′) Obtain the channel space feature relationship matrix M′ of the feature data of the higher channel.

[0101]

[0102] In formula 16, f′ is the single-layer polymerization characteristic.

[0103] Component 3 receives the feature f after component 2 increases the number of channels again B×C″×(H×W) (C″>C′), and multiply it by M′ with the spatial feature relationship matrix of the higher channel calculated by component 1, and aggregate the high channel level features again.

[0104]

[0105] In formula 17, f″ is the characteristic of double-layer polymerization.

[0106] Finally, two convolutional layers are used Change the original feature data size f in , and the features after two layers of aggregation are added to obtain the final output data f out .

[0107] f out =f′+f″+f in Formula 18

[0108] The multi-level and multi-channel feature mining module aims to enhance the overall feature representation by exploiting the relationship between features at different levels and channels. This approach not only ensures the effective aggregation of features, but also improves the performance of the VIReID task.

[0109] To demonstrate the superiority of FCFPN, the proposed model is compared with several advanced methods. The experimental results on SYSU-MM01 and RegDB datasets are shown in Table 1.

[0110] As can be seen from Table 1, the test results on the two datasets show that the FCFPN model proposed in the patent outperforms most of the current advanced models in multiple performance indicators. Specifically, in the All-Search mode on the SYSU-MM01 dataset, FCFPN achieved a Rank-1 accuracy of 75.31% and a mAP of 72.38%, and in the Indoor search mode, FCFPN achieved a Rank-1 accuracy of 84.69% and a mAP of 86.87%. In the VIS to IR search mode on the RegDB dataset, FCFPN achieved a Rank-1 accuracy of 92.96% and a mAP of 87.27%, and in the IR to VIS search mode, the FCFPN model also achieved a Rank-1 accuracy of 91.36% and a mAP of 85.08%. The above experimental results verify the effectiveness of the method proposed in the article, and also verify that the method proposed in the article is more effective in projecting cross-modal features and reducing the modality gap between VIS images and IR images.

[0111] Table 1 Comparison of FPEN model with some advanced methods on SYSU-MM01 and RegDB datasets

[0112]

[0113] Two benchmark datasets were used in this experiment, namely RegDB and SYSU-MM01. The RegDB dataset contains data of 412 identities. Each identity data consists of two modalities, infrared and visible light. Each modal data contains 10 images. The identity data is evenly and randomly divided into training set and test set, with two test modes: visible light to infrared light and infrared light to visible light. The SYSU-MM01 dataset contains data of 491 identities captured by 4 VIS cameras and 2 IR cameras, including 296 people in the training set, 20,284 RGB images, and 9,929 infrared images; the test set contains 96 people, 3,803 infrared images used as queries, and 301 randomly selected RGB images used as galleries. This dataset has two test modes: full search mode, using images captured by all VIS cameras as galleries; indoor search mode, using images captured by two indoor VIS cameras as galleries.

[0114] Among the surveillance cameras installed in public places (such as airports, stations, shopping malls, etc.), some cameras use infrared light to capture images in low-light conditions. Through VI Re ID technology, daytime and nighttime surveillance data can be matched to identify and track specific people.

[0115] In the construction of smart cities, VI Re ID technology can be used in traffic management, accident handling, public safety, etc. For example, by installing infrared and visible light cameras at traffic intersections, pedestrians can be identified through VI Re ID technology to improve traffic management efficiency.

[0116] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A cross-modal feature complementation and feature projection network system, comprising a modality complementation module, a cross-modal feature projection module, a multi-channel feature mining module and a training and optimization module, characterized in that: The modality complementation module is used to reduce the modality difference between the infrared light IR image and the visible light VIS image. The modality complementation module realizes the complementation between the two modalities by extracting and fusing the coarse-grained and fine-grained features of the two modalities, so that the data of the two modalities are more similar; The cross-modal feature projection module is used to project feature data of different modalities into a shared feature space. By calculating the correspondence between local features, the module can effectively align VIS and IR features. The multi-channel feature mining module is used to further mine and aggregate multi-level, multi-channel cross-modal features in a shared feature space. The multi-channel feature mining module enhances the richness and diversity of the overall feature representation by improving channel-level features.

2. According to claim 1, a cross-modal feature complementation and feature projection network system is characterized in that: The modality complementation module includes a feature extraction layer and a modality complementation layer; The feature extraction layer comprises: Two convolutional neural networks are used to extract VIS image features and IR image features respectively; Use a dilated convolutional neural network to extract coarse-grained and wide-view features to avoid the gridding effect problem. Use a dilated convolutional neural network with a dilation rate of (1,2,3,1,2,3) to extract coarse-grained and wide-view features. Use small MLP to process high-dimensional feature data, learn more modal features, and extract generalized features through average pooling; Use 1x1 convolution kernel to extract fine-grained features, learn modal feature data through small MLP, and then use maximum pooling to focus on extracting features with large modal differences.

3. A cross-modal feature complementation and feature projection network system according to claim 2, characterized in that: The modality complementation layer obtains complementary feature data by performing dot multiplication of the coarse-grained feature weights and fine-grained feature weights of the two modalities with the VIS and IR feature data respectively, and then adding them together.

4. According to claim 1, a cross-modal feature complementation and feature projection network system is characterized in that: The cross-modal feature projection module includes a feature splitting layer, a covariance matrix calculation layer, and a feature projection layer; The feature splitting layer splits the VIS and IR image feature data into multiple data blocks by row and column respectively; The covariance matrix calculation layer is used to calculate the covariance between each data block and obtain the horizontal and vertical correspondence relationship of the local features; The feature projection layer uses the horizontal and vertical cross-modal feature correspondences as projection coefficients and projects the features into a shared feature space through a linear function.

5. The cross-modal feature complementation and feature projection network system according to claim 1, characterized in that: The multi-channel feature mining module includes a channel promotion layer, a feature aggregation layer, and a double-layer aggregation layer; The channel boosting layer uses a 1x1 convolutional layer to increase the number of channels of feature data and fold the width and height into one spatial dimension; The feature aggregation layer calculates the channel space feature relationship matrix and multiplies it with the feature data to aggregate channel level features; The double-layer aggregation adds the aggregated features to the original feature data to obtain the final output data.

6. A cross-modal feature complementation and feature projection network system according to claim 1, characterized in that: The training and optimization module uses the pre-trained ResNet-50 as the backbone network through the model, randomly and equally selects identity data as input training data in each training batch, uses the SGD optimizer, and for different data sets, uses a single-layer or double-layer modality complementary network combined with a cross-modal feature projection network.

7. A cross-modal feature complementation and feature projection network system according to claim 6, characterized in that: The training and optimization module uses a single-layer modality complementation network combined with a single-layer cross-modal feature projection network for the RegDB dataset; For the SYSU-MM01 dataset, a two-layer modality complementarity network combined with a two-layer cross-modal feature projection network is used.

Images