A cross-modal target re-identification method based on dynamic MLP

Abstract

The application discloses a cross-modal target re-identification method based on a dynamic MLP, and comprises the following steps: step 1, obtaining an image to be identified; step 2, inputting the image to be identified into a trained identification network to obtain a feature vector to be identified, and matching the feature vector to be identified with feature vectors in a feature vector library to obtain a target identification result. The application solves the problem of unstable gradient descent in the training process of parameter mapping of a parallel neural network by standardizing input and output. The cross-modal target re-identification method based on the dynamic MLP can enhance the representation ability of modal feature vectors, reduce the difference between modes, further improve the model precision, and improve the accuracy of matching of different modal targets.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of image processing, and particularly relates to a cross-modal target re-identification method based on a dynamic MLP. BACKGROUND

[0002] Object re-identification (Object ReID) is a technology of judging a query target in an image or a video clip by using computer vision technology given a specific target image. The object re-identification technology is generally considered as a branch of the image retrieval field. Given a target image, the target image is searched in a cross-camera and cross-scene manner, and the technology can be widely used in public places such as communities, airports and railway stations that need intelligent video monitoring, tracking and identifying dangerous goods and cracking down on criminal behavior.

[0003] The object re-identification network framework is similar to the general image processing framework of deep learning. First, the relevant images are input into a convolutional neural network (CNN) to extract the feature vectors of the images and perform classification prediction. Then, the metric learning is used to classify and judge by using the image labels, calculate the loss value, and finally perform back propagation and update the network parameters.

[0004] Cross-modal target re-identification mainly studies that given a specific visible light image or infrared image, the same identity target image is searched in the image library in the visible light mode and the infrared mode. In a real application scenario, the camera should be able to operate all day and night, and the single-modal target re-identification only targets the daytime scene and has limited monitoring effect on the night scene. Therefore, the cross-modal target re-identification is proposed to realize all-day and all-night joint defense monitoring. In the cross-modal target re-identification task, the modal difference between the visible light image and the infrared image seriously affects the performance of the cross-modal target re-identification model. Therefore, how to reduce the modal difference and reduce the identification difficulty between the modes has become the mainstream direction of the cross-modal target re-identification research. The current research work on cross-modal target re-identification can be divided into two categories: one category is to use a generative adversarial network (GAN) to generate an intermediate modal image as an intermediate bridge to narrow the distance between the visible light mode and the infrared mode, and the other category is to directly convert the visible light image into a grayscale image by a physical means to make it more close to the infrared image, so as to reduce the modal difference. However, the method of using an "intermediate" does not fundamentally solve the problem of large modal difference.

[0005] Therefore, how to improve the accuracy of target matching of different modes has become a problem to be solved. SUMMARY

[0006] To solve the above problems existing in the prior art, the application provides a cross-modal target re-identification method based on a dynamic MLP.

[0007] The technical problem to be solved by the application is solved by the following technical scheme:

[0008] A cross-modal target re-identification method based on a dynamic MLP, the cross-modal target re-identification method comprising:

[0009] Step 1, obtaining an image to be identified;

[0010] Step 2, inputting the image to be identified into a trained identification network to obtain a feature vector to be identified, matching the feature vector to be identified with feature vectors in a feature vector library to obtain a target identification result, the identification network comprising a dynamic MLP module; wherein,

[0011] The training method of the identification network comprises: obtaining a first embedded feature vector and a second embedded feature vector based on the dynamic MLP module, updating parameters of the identification network by using the first embedded feature vector and the second embedded feature vector to obtain the trained identification network, wherein the first embedded feature vector is a visible light image feature vector embedding an infrared image feature vector, and the second embedded feature vector is an infrared image feature vector embedding a visible light image feature vector.

[0012] Optionally, the identification network further comprises a first feature vector extraction module and a second feature vector extraction module.

[0013] The first embedded feature vector and the second embedded feature vector obtained based on the dynamic MLP module comprise:

[0014] Obtaining a visible light image and an infrared image;

[0015] Inputting the visible light image and the infrared image into the first feature vector extraction module and the second feature vector extraction module respectively to obtain a visible light image feature vector and an infrared image feature vector;

[0016] Embedding the infrared image feature vector into the visible light image feature vector through the dynamic MLP module to obtain the first embedded feature vector;

[0017] Embedding the visible light image feature vector into the infrared image feature vector through the dynamic MLP module to obtain the second embedded feature vector.

[0018] Optionally, the first feature vector extraction module and the second feature vector extraction module are both modified ResNet-50 networks, and the first feature vector extraction module and the second feature vector extraction module are two networks that do not share parameters.

[0019] The modified ResNet-50 network is a network in which the last average pooling layer and the structure after the last average pooling layer of the ResNet-50 network are removed.

[0020] Optionally, the infrared image feature vector is embedded into the visible light image feature vector through a dynamic MLP module to obtain the first embedded feature vector, including:

[0021] S3.1, inputting the infrared image feature vector into an MLP in the dynamic MLP module to convert the one-dimensional infrared image feature vector into a first two-dimensional matrix;

[0022] S3.2, inputting the visible light image feature vector into the dynamic MLP module to obtain a visible light image feature vector after one iteration based on the visible light image feature vector and the first two-dimensional matrix, and the visible light image feature vector after one iteration is represented as:

[0023]

[0024] wherein, represents the visible light image feature vector after one iteration, X r represents the visible light image feature vector, ReLU(·) represents a linear activation function, LN(·) represents layer normalization, Q i represents the first two-dimensional matrix;

[0025] S3.3, taking the visible light image feature vector after one iteration obtained in S3.2 as the input of the dynamic MLP module in S3.2 in the next iteration, and repeating steps S3.1 to S3.3, and after N1 iterations, selecting a visible light image feature vector after one iteration that meets a first preset condition in the N1 iterations as the first embedded feature vector.

[0026] Optionally, selecting the visible light image feature vector after one iteration that meets the first preset condition in the N1 iterations as the first embedded feature vector includes:

[0027] Selecting a visible light image feature vector after one iteration that has the highest Rank-1 and mAP or the highest Rank-1 and not the highest mAP in the N1 iterations as the first embedded feature vector.

[0028] Optionally, the visible light image feature vector is embedded into the infrared image feature vector through a dynamic MLP module to obtain the second embedded feature vector, including:

[0029] S4.1, input the visible light image feature vector into an MLP in the dynamic MLP module to convert the one-dimensional visible light image feature vector into a second two-dimensional matrix;

[0030] S4.2, input the infrared image feature vector into the dynamic MLP module to obtain an infrared image feature vector after one iteration based on the infrared image feature vector and the second two-dimensional matrix, and the infrared image feature vector after one iteration is represented as:

[0031]

[0032] wherein, indicates the infrared image feature vector after one iteration, X i indicates the infrared image feature vector, ReLU(·) indicates a linear activation function, LN(·) indicates layer normalization, Q r indicates the second two-dimensional matrix;

[0033] S4.3, taking the infrared image feature vector after one iteration obtained in S4.2 as the input of the dynamic MLP module in S4.2 in the next iteration, and repeating S4.1 to S4.3, after N2 iterations, selecting the infrared image feature vector after one iteration that meets the second preset condition in the N2 iterations as the second embedded feature vector.

[0034] Optionally, selecting the infrared image feature vector after one iteration that meets the second preset condition in the N2 iterations as the second embedded feature vector includes:

[0035] selecting the infrared image feature vector after one iteration that has the highest Rank-1 and mAP or the highest Rank-1 and not the highest mAP in the N2 iterations as the second embedded feature vector.

[0036] Optionally, updating the parameters of the recognition network using the first embedded feature vector and the second embedded feature vector to obtain the trained recognition network, including:

[0037] performing center aggregation loss calculation on the first embedded feature vector and the second embedded feature vector using a first loss function to obtain a first loss;

[0038] passing the first embedded feature vector and the second embedded feature vector through a classifier to obtain a first classification result and a second classification result, respectively;

[0039] The first classification result and the second classification result are calculated respectively by using a second loss function, to obtain a second loss and a third loss;

[0040] The first loss, the second loss and the third loss are back propagated to update the network parameters of the identification network until the identification network converges, to obtain a trained identification network.

[0041] Optionally, the first loss function is represented as:

[0042]

[0043]

[0044] wherein C v represents a feature mean center of the first embedding feature vector, C i represents a feature mean center of the second embedding feature vector, represents an MSE mean square error loss function, represents an i-th first embedding feature vector of a j-th target in the visible light image, represents an i-th second embedding feature vector of a j-th target in the infrared image, N rgb represents a total number of the visible light images, N ir represents a total number of the infrared images.

[0045] Optionally, the second loss function is represented as:

[0046]

[0047] wherein N represents a total number of the first embedding feature vectors or the second embedding feature vectors, represents an i-th first classification result corresponding to the i-th first embedding feature vector or an i-th second classification result corresponding to the i-th second embedding feature vector, Y i represents an identity label of an i-th visible light image or infrared image, L ce represents a cross-entropy loss function.

[0048] Compared with the prior art, the present application has the following beneficial effects:

[0049] The target re-identification method of the present application obtains the visible light image feature vector embedding the infrared image feature vector and the infrared image feature vector embedding the visible light image feature vector based on the dynamic MLP module, to update the network parameters of the identification network, which can enhance the representation ability of the modal feature vector, reduce the inter-modal difference, further improve the model precision, and improve the accuracy of matching targets of different modalities.

[0050] The present application will be further described in detail below with reference to the accompanying drawings. BRIEF DESCRIPTION OF DRAWINGS

[0051] Figure 1 is a flowchart of a cross-modal target re-identification method based on dynamic MLP provided by an embodiment of the present application;

[0052] Figure 2 is a general framework diagram of a cross-modal target re-identification method based on dynamic MLP provided by an embodiment of the present application;

[0053] Figure 3 is a dynamic iteration diagram of a cross-modal target re-identification method based on dynamic MLP provided by an embodiment of the present application. DETAILED DESCRIPTION

[0054] The present application will be further described in detail below in conjunction with specific embodiments, but the embodiments of the present application are not limited thereto.

[0055] Embodiment One

[0056] Please refer to Figure 1 and Figure 2 , Figure 1 is a flowchart of a cross-modal target re-identification method based on dynamic MLP provided by an embodiment of the present application, Figure 2 is a general framework diagram of a cross-modal target re-identification method based on dynamic MLP provided by an embodiment of the present application, and the cross-modal target re-identification method based on dynamic MLP comprises:

[0057] Step 1, obtaining an image to be identified.

[0058] Here, the image to be identified can be a visible light image to be identified or an infrared image to be identified.

[0059] Step 2, inputting the image to be identified into a trained identification network to obtain a feature vector to be identified, matching the feature vector to be identified with a feature vector in a feature vector library to obtain a target identification result, wherein the identification network comprises a dynamic MLP module.

[0060] Here, the feature vector library can be pre-established to include feature vectors obtained by a trained recognition network from a large number of visible light images and infrared images, and the feature vectors are used for matching with the image to be recognized, so as to take the result with the highest similarity as the final target recognition result. In addition, the feature vector library can also be established in real time by obtaining feature vectors of visible light images and infrared images for matching through the trained recognition network, and then selecting the result with the highest similarity as the final target recognition result. The similarity can be calculated by calculating the Euclidean distance between the feature vector to be recognized and the feature vector in the feature vector library, so the result with the highest similarity is the one with the closest Euclidean distance, and thus the final target recognition result can be obtained.

[0061] The training method of the recognition network comprises: obtaining a first embedded feature vector and a second embedded feature vector based on the dynamic MLP module, updating the parameters of the recognition network by using the first embedded feature vector and the second embedded feature vector to obtain a trained recognition network, wherein the first embedded feature vector is a visible light image feature vector embedding an infrared image feature vector, and the second embedded feature vector is an infrared image feature vector embedding a visible light image feature vector.

[0062] Optionally, the dynamic MLP module is, for example, Figure 2 The dashed part in the middle. Therefore, the present application obtains a visible light image feature vector embedding an infrared image feature vector and an infrared image feature vector embedding a visible light image feature vector based on the dynamic MLP module, and updates the network parameters of the recognition network, which can enhance the representation ability of the modal feature vector, reduce the difference between the modal, further improve the model precision, and improve the accuracy of matching targets of different modal.

[0063] In one specific embodiment, the recognition network further comprises a first feature vector extraction module and a second feature vector extraction module.

[0064] Further, the first feature vector extraction module and the second feature vector extraction module are both modified ResNet-50 networks, and the first feature vector extraction module and the second feature vector extraction module are two networks with unshared parameters.

[0065] The modified ResNet-50 network is a network removing the last average pooling layer of the ResNet-50 network and the structure after the last average pooling layer.

[0066] The application adopts ResNet-50 as a backbone network. ResNet-50 is composed of five convolutional blocks, and the convolution kernel size of the convolutional layer of the first convolutional block is 7x7. The remaining four convolutional blocks include three convolutional layers, and the convolution kernel size of the first convolutional layer of each convolutional block is 1x1, the convolution kernel size of the second convolutional layer is 3x3, and the convolution kernel size of the third convolutional layer is 1x1. In addition, ResNet-50 designs a residual structure, which can solve the problem of accuracy reduction caused by the increase of network model depth.

[0067] The two modified ResNet-50 networks of the application adopt non-shared and independent network parameters. Due to the difference in imaging principles of visible light images and infrared images, the independent double-flow structure can enable the two networks to learn the respective characteristics of the visible light mode and the infrared mode, more accurately extract the identity correlation features of the two modes, and retain the respective modal information.

[0068] Please refer to Figure 3 and Figure 3 In the embodiment, the first embedding feature vector and the second embedding feature vector are obtained based on the dynamic MLP module, which comprises:

[0069] S1, obtaining a visible light image and an infrared image.

[0070] S2, inputting the visible light image and the infrared image into a first feature vector extraction module and a second feature vector extraction module respectively to obtain a visible light image feature vector X r and an infrared image feature vector X i .

[0071] S3, embedding the infrared image feature vector into the visible light image feature vector through a dynamic MLP module to obtain the first embedding feature vector.

[0072] In one specific embodiment, S3 can comprise:

[0073] S3.1, inputting the infrared image feature vector into an MLP (MLP can be a linear layer in particular) in the dynamic MLP module, so that the one-dimensional infrared image feature vector is converted into a first two-dimensional matrix, and the specific formula is:

[0074] Q i = Reshape(f(X i ))

[0075] Wherein, Reshape(·) represents conversion from a one-dimensional vector to a two-dimensional matrix, f(·) represents a full connection layer, and Q i represents a two-dimensional matrix converted from the infrared image feature vector.

[0076] S3.2, input the visible light image feature vector into the dynamic MLP module to obtain a visible light image feature vector after one iteration based on the visible light image feature vector and the first two-dimensional matrix, and the visible light image feature vector after one iteration is represented as:

[0077]

[0078] wherein, represents the visible light image feature vector after one iteration, X r represents the visible light image feature vector, ReLU(·) represents a linear activation function, LN(·) represents layer normalization, Q i represents the first two-dimensional matrix.

[0079] S3.3, taking the visible light image feature vector after one iteration obtained in S3.2 as the input of the dynamic MLP module in S3.2 in the next iteration, and repeating steps S3.1 to S3.3, after N1 iterations, selecting the visible light image feature vector after one iteration that meets the first preset condition in N1 iterations as the first embedding feature vector, represented as:

[0080]

[0081] wherein, represents the visible light image feature vector after the nth iteration, X represents the visible light image feature vector after the (n-1)th iteration.

[0082] Further, selecting the visible light image feature vector after one iteration that meets the first preset condition in N1 iterations as the first embedding feature vector comprises:

[0083] Selecting the visible light image feature vector after one iteration that has the highest Rank-1 and mAP or the highest Rank-1 and not the highest mAP in N1 iterations as the first embedding feature vector.

[0084] Here, generally, the changes of Rank-1 and mAP are consistent, that is, when Rank-1 is the highest, mAP is also the highest, at this time, the visible light image feature vector after one iteration that has the highest Rank-1 and mAP is selected as the first embedding feature vector, however, when Rank-1 and mAP are not the highest at the same time, the visible light image feature vector after one iteration that has the highest Rank-1 is selected as the first embedding feature vector, mAP is the average accuracy, mAP = the number of correctly classified pictures / total number of pictures, Rank-n is the recognition rate, which represents the ratio of the number of tests that can be correctly judged to the total number of test samples after matching according to the similarity matching rule.

[0085] After feature extraction, the infrared image feature vector and the visible light image feature vector are input into the dynamic MLP module. First, the infrared image feature vector is converted from a one-dimensional vector to a two-dimensional matrix by the MLP, which is embedded into the visible light image feature vector, so that the infrared image feature vector can interact with the visible light image feature vector in a higher and wider dimension. The two-dimensional matrix converted from the infrared image feature vector is constantly updated with changes in network parameters, and is not a linear structure. Therefore, the dynamic MLP module can be regarded as a kind of adaptive nonlinear dynamic projection, which enables the visible light image feature vector to effectively learn the information distribution weight of the infrared image, reduces the difficulty of modal recognition, and produces a new visible light image feature with better identity discrimination. At the same time, since more dynamic projection can lead to better interaction, the experimental performance is improved. Therefore, the overall architecture is designed as a recursive architecture, which is iterated N times, where N is 2 for example.

[0086] S4. embedding the visible light image feature vector into the infrared image feature vector through the dynamic MLP module to obtain a second embedded feature vector.

[0087] S4.1, inputting the visible light image feature vector into the MLP in the dynamic MLP module to convert the one-dimensional visible light image feature vector into a second two-dimensional matrix, and the specific formula is:

[0088] Q r = Reshape(f(X r ))

[0089] wherein, Q r represents the two-dimensional matrix converted from the visible light image feature vector.

[0090] S4.2, inputting the infrared image feature vector into the dynamic MLP module to obtain the infrared image feature vector after one iteration based on the infrared image feature vector and the second two-dimensional matrix, and the infrared image feature vector after one iteration is represented as:

[0091]

[0092] wherein, represents the infrared image feature vector after one iteration, X i represents the infrared image feature vector, Q r represents the second two-dimensional matrix.

[0093] S4.3. Use the infrared image feature vector obtained in S4.2 after one iteration as the input of the dynamic MLP module in S4.2 in the next iteration, and repeat S4.1 to S4.3. After performing N2 iterations, select the infrared image feature vector after one iteration that meets the second preset condition in the N2 iterations as the second embedded feature vector, expressed as:

[0094]

[0095] in, represents the infrared image feature vector of the nth iteration, represents the infrared feature vector of the n-1th iteration.

[0096] Furthermore, selecting an infrared image feature vector after one iteration that meets the second preset condition in the N2 iterations as the second embedded feature vector includes:

[0097] The infrared image feature vector after the iteration with the highest Rank-1 and mAP in N2 iterations or the one with the highest mAP but not the highest Rank-1 is selected as the second embedding feature vector.

[0098] After feature extraction, the present invention inputs the visible light image feature vector and the infrared image feature vector into a dynamic MLP module. First, the visible light image feature vector is converted from a one-dimensional vector to a two-dimensional matrix through MLP, and is embedded into the infrared image feature vector. The visible light image feature vector can interact with the infrared image feature vector in a higher and wider dimension. The two-dimensional matrix converted from the visible light image feature vector is continuously updated as the network parameters change, and it is not a linear structure. Therefore, the dynamic MLP module can be regarded as an adaptive nonlinear dynamic projection, which enables the infrared image feature vector to effectively learn the information distribution weight of the visible light image, reduce the difficulty of modal recognition, and generate new infrared image features with more identity discriminative properties. At the same time, more dynamic projections can lead to better interactions and improve experimental performance. Therefore, the overall architecture is designed as a recursive architecture, with a total of N iterations.

[0099] See Rank-1 The present invention designs a recursive structure that allows the feature vector to pass through the dynamic MLP multiple times. Except for the channel dimensions, the N recursive blocks are identical. To reduce computational overhead, the intermediate channel dimensions are set smaller than the input channel dimensions. During the final pass through the dynamic MLP, the intermediate channel dimensions are restored to the original input channel dimensions through the MLP.

[0100] In this embodiment, the first embedded feature vector and the second embedded feature vector are used to update the parameters of the recognition network to obtain a trained recognition network, including:

[0101] S5, center aggregation loss calculation is performed on the first embedding feature vector and the second embedding feature vector by using a first loss function, to obtain a first loss.

[0102] Here, the first loss function is represented as:

[0103]

[0104]

[0105] wherein C v represents the feature mean center of the first embedding feature vector, C i represents the feature mean center of the second embedding feature vector, represents the MSE mean square error loss function, represents the i-th first embedding feature vector of the j-th target in the visible light image, represents the i-th second embedding feature vector of the j-th target in the infrared image, N rgb represents the total number of visible light images, N ir represents the total number of infrared images.

[0106] S6, the first embedding feature vector and the second embedding feature vector are input into a classifier to obtain a first classification result and a second classification result.

[0107] S7, the first classification result and the second classification result are calculated by using a second loss function, to obtain a second loss and a third loss.

[0108] Here, the second loss function is represented as:

[0109]

[0110] wherein N represents the total number of the first embedding feature vectors or the second embedding feature vectors, i.e. N=N rgb , or N=N ir , represents the first classification result corresponding to the i-th first embedding feature vector or the second classification result corresponding to the i-th second embedding feature vector, Y i represents the identity label of the i-th visible light image or infrared image, L ce represents the cross-entropy loss function.

[0111] The application utilizes a central aggregation loss to narrow the feature center distance of different modalities of the same identity target, narrows the centers of the visible light modality and the infrared modality, instead of narrowing each visible light image feature vector and infrared image feature vector, reduces the calculation overhead, and meanwhile improves the similarity of the cross-modality within the class. Meanwhile, the classification loss is utilized to extract visible light and infrared features under the identity discrimination.

[0112] S8, the first loss, the second loss and the third loss are propagated back to update the network parameters of the identification network until the identification network converges, and a trained identification network is obtained.

[0113] That is, the total loss of the identification network is the sum of the first loss, the second loss and the third loss, and therefore the network parameters of the identification network can be updated by the back propagation of the first loss, the second loss and the third loss. After multiple iterations, the identification network converges, and thus a trained identification network can be obtained.

[0114] The cross-modality target re-identification method based on the dynamic MLP provided by the application proposes a dynamic MLP module, which converts the weight distribution of one modality information into a two-dimensional matrix, the two-dimensional matrix dynamically changes with the network parameter update, and is adaptively projected onto another modality feature vector. Two kinds of modal features can interact with each other in a higher and more extensive dimension, enhance the representation ability of the modal feature vector, reduce the difference between the modalities, and further improve the model precision and the accuracy of matching targets of different modalities.

[0115] The method described in the application is verified based on experiments:

[0116] Database: The application is evaluated and verified on two challenging and public datasets, namely SYSU-MM01 and RegDB. The SYSU-MM01 dataset is composed of visible light images and infrared images taken by 6 cameras (4 visible light cameras and 2 infrared cameras). Among them, the visible light images are 287628, the infrared images are 15972, and 491 pedestrians are photographed. Camera 1 and 2 are visible light cameras, which are taken in a well-lit indoor environment, camera 3 and 6 are infrared cameras, which are taken at night with infrared light. Camera 4 and 5 are visible light cameras, which are taken in an outdoor scene. The RegDB dataset is more processed than the SYSU-MM01 dataset, which is composed of 412 pedestrians, including 254 women and 158 men. Among them, each pedestrian has 20 images, including 10 visible light images and 10 infrared images. In the test stage, RegDB has two modes: visible light to infrared and infrared to visible light. The visible light to infrared mode means that the visible light image is the query image and the infrared image is the gallery image; the infrared to visible light mode means that the infrared image is the query image and the visible light image is the gallery image.

[0117] Evaluation criteria: The evaluation criteria used in this embodiment are Cumulated Match Curve (CMC) and Mean Average Precision (mAP). CMC is generally replaced by Rank-1, Rank-10, Rank-20, etc., reflecting the retrieval accuracy; Rank-n recognition rate is the ratio of the number of correct labels judged in the nth time to the total number of test samples after matching according to the similarity matching rule.

[0118] Table 1 Cross-modal target re-identification accuracy (SYSU-MM01 dataset)

[0119] Rank-10 Rank-20 mAP Baseline model Dynamic MLP 63.59 92.74 97.52 59.34 ​ 71.30 95.54 98.78 67.73

[0120] Table 2 Cross-modal target re-identification accuracy (RegDB dataset)

[0121]

[0122] As can be seen from Tables 1 and 2, the method proposed in the present application has good results on the SYSU-MM01 dataset and the RegDB dataset, especially in the two key indicators of Rank-1 and mAP, which are significantly higher than the baseline model. The experimental results on the SYSU-MM01 dataset can reach 63.59% and 59.34%, respectively, which are 7.71% and 8.39% higher than the baseline model structure, respectively. In the experimental results on the RegDB dataset, in the visible light to infrared mode, Rank-1 and mAP can reach 94.32% and 89.02%, respectively, and in the infrared to visible light mode, Rank-1 and mAP can reach 93.27% and 88.77%, respectively. The results corresponding to the Rank-1 and mAP indicators are 11.35% and 10.33% higher than the baseline model results in the visible light to infrared mode, and 11.34% and 11.12% higher than the baseline model results in the infrared to visible light mode. The above experimental results show that the design of the dynamic MLP module, the use of adaptive dynamic weighted projection, and multiple iterations can make the visible light image feature vector and the infrared image feature vector fully act, reduce the difference between the modes, and greatly improve the accuracy of retrieval.

[0123] It should be noted that the terms "first", "second", etc. are used only for descriptive purposes and are not to be construed as indicating or implying relative importance or an indicated number of technical features. Thus, features defined with "first", "second" can explicitly or implicitly include one or more features. In the description of the present application, the meaning of "a plurality" is two or more, unless otherwise expressly and specifically limited.

[0124] In the description of the present application, the description of the terms "one embodiment", "some embodiments", "an example", "a specific example", or "some examples" means that the specific features or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In the present description, the illustrative description of the above terms is not necessarily directed to the same embodiment or example. Moreover, the specific features or characteristics described can be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine different embodiments or examples described in the present description.

[0125] Although the present application is described herein in conjunction with various embodiments, other variations of the disclosed embodiments can be understood and effected by those skilled in the art in practicing the claimed application, from a study of the drawings, the disclosure and the appended claims. In the description, the word "comprising" does not exclude other components or steps, and "a" or "one" does not exclude a plurality. Some measures are described in mutually different embodiments, but this does not mean that these measures cannot be combined to produce good results.

[0126] The above is a further detailed description of the present application in conjunction with specific preferred embodiments, and the specific implementation of the present application cannot be limited to these descriptions. For those skilled in the art to which the present application belongs, without departing from the concept of the present application, a number of simple deductions or substitutions can be made, which should be considered as falling within the scope of protection of the present application.

Claims

1. A cross-modal object re-identification method based on dynamic MLP, characterized by: The cross-modal target re-identification method includes: Step 1: Obtain the image to be identified; Step 2: Input the image to be identified into the trained recognition network to obtain the feature vector to be identified, and match the feature vector to be identified with the feature vector in the feature vector library to obtain the target recognition result. The recognition network includes a dynamic MLP module, a first feature vector extraction module and a second feature vector extraction module; wherein, The training method of the recognition network includes: obtaining a first embedded feature vector and a second embedded feature vector based on a dynamic MLP module, and updating parameters of the recognition network using the first embedded feature vector and the second embedded feature vector to obtain the trained recognition network; The first embedding feature vector and the second embedding feature vector are obtained based on the dynamic MLP module, including: Acquire visible light images and infrared images; Inputting the visible light image and the infrared image into a first feature vector extraction module and a second feature vector extraction module respectively to obtain a visible light image feature vector and an infrared image feature vector; Embedding the infrared image feature vector into the visible light image feature vector through a dynamic MLP module to obtain the first embedded feature vector includes: S3.

1. Inputting the infrared image feature vector into the MLP in the dynamic MLP module to convert the one-dimensional infrared image feature vector into a first two-dimensional matrix; S3.

2. Input the visible light image feature vector into the dynamic MLP module to obtain a visible light image feature vector after one iteration based on the visible light image feature vector and the first two-dimensional matrix. The visible light image feature vector after one iteration is expressed as: in, represents the visible light image feature vector after one iteration, represents the visible light image feature vector, represents the linear activation function, Representation layer normalization, represents the first two-dimensional matrix; S3.

3. Using the visible light image feature vector obtained in S3.2 after one iteration as the input to the dynamic MLP module in S3.2 for the next iteration, and repeating steps S3.1 to S3.

3. After performing N1 iterations, select the visible light image feature vector after one iteration that meets the first preset condition among the N1 iterations as the first embedded feature vector; The visible light image feature vector is embedded into the infrared image feature vector through a dynamic MLP module to obtain the second embedded feature vector.

2. The cross-modal object re-identification method based on dynamic MLP according to claim 1, characterized in that The first feature vector extraction module and the second feature vector extraction module are both modified ResNet-50 networks, and the first feature vector extraction module and the second feature vector extraction module are two networks that do not share parameters; The modified ResNet-50 network is a network in which the last average pooling layer of the ResNet-50 network and the structure after the last average pooling layer are removed.

3. The cross-modal object re-identification method based on dynamic MLP according to claim 1, characterized in that Selecting a visible light image feature vector after one iteration that meets the first preset condition in the N1 iterations as the first embedded feature vector, including: The visible light image feature vector after an iteration in which both Rank-1 and mAP are the highest or mAP is not the highest but Rank-1 is the highest in N1 iterations is selected as the first embedded feature vector.

4. The cross-modal object re-identification method based on dynamic MLP according to claim 1, characterized in that Embedding the visible light image feature vector into the infrared image feature vector through a dynamic MLP module to obtain a second embedded feature vector includes: S4.

1. Inputting the visible light image feature vector into the MLP in the dynamic MLP module to convert the one-dimensional visible light image feature vector into a second two-dimensional matrix; S4.

2. Input the infrared image feature vector into the dynamic MLP module to obtain an infrared image feature vector after one iteration based on the infrared image feature vector and the second two-dimensional matrix. The infrared image feature vector after one iteration is expressed as: in, represents the infrared image feature vector after one iteration, represents the infrared image feature vector, represents the linear activation function, Representation layer normalization, represents the second two-dimensional matrix; S4.

3. Use the infrared image feature vector after one iteration obtained in S4.2 as the input of the dynamic MLP module in S4.2 in the next iteration, and repeat S4.1 to S4.

3. After performing N2 iterations, select the infrared image feature vector after one iteration in the N2 iterations that meets the second preset condition as the second embedded feature vector.

5. The cross-modal object re-identification method based on dynamic MLP according to claim 4 is characterized in that Selecting the infrared image feature vector after one iteration that meets the second preset condition in the N2 iterations as the second embedded feature vector, including: The infrared image feature vector after an iteration in which both Rank-1 and mAP are the highest or mAP is not the highest but Rank-1 is the highest in N2 iterations is selected as the second embedding feature vector.

6. The cross-modal object re-identification method based on dynamic MLP according to claim 1, characterized in that Updating parameters of the recognition network using the first embedded feature vector and the second embedded feature vector to obtain the trained recognition network includes: Performing a center aggregation loss calculation on the first embedded feature vector and the second embedded feature vector using a first loss function to obtain a first loss; Passing the first embedded feature vector and the second embedded feature vector through a classifier to obtain a first classification result and a second classification result respectively; Using a second loss function to calculate the first classification result and the second classification result respectively to obtain a second loss and a third loss; The first loss, the second loss, and the third loss are back-propagated to update the network parameters of the recognition network until the recognition network converges to obtain a trained recognition network.

7. The cross-modal object re-identification method based on dynamic MLP according to claim 6, characterized in that The first loss function is expressed as: in, represents the feature mean center of the first embedded feature vector, represents the feature mean center of the second embedded feature vector, represents the MSE mean square error loss function, Indicates the first j The first target i The first embedding feature vector, Indicates the infrared image j The first target i The second embedding feature vector, represents the total number of visible light images, Indicates the total number of infrared images.

8. The cross-modal object re-identification method based on dynamic MLP according to claim 6, characterized in that: The second loss function is expressed as: in, represents the total number of first embedded feature vectors or second embedded feature vectors, Indicates the i The first classification result or the first classification result corresponding to the first embedded feature vector i The second classification result corresponding to the second embedded feature vector, Indicates the i an identity tag of a visible light image or infrared image, represents the cross entropy loss function.

Images