A cross-modal pedestrian re-identification method based on cross-scale information interaction

Abstract

The application discloses a cross-modal pedestrian re-identification method based on cross-scale information interaction and belongs to the technical field of computer vision, mainly comprising the following steps: feeding a visible light pedestrian image and an infrared pedestrian image into a cross-modal confusion device, using mature image processing technology to pre-process the two different modal input images, and then obtaining input objects meeting the model requirements; subsequently, a parameter-separated feature extraction network is used to process the input objects to obtain visible light image features and infrared image features; on this basis, a parameter-shared feature extraction network is used in combination with a cross-scale interaction module to further extract modal shared features; meanwhile, global features and local region feature information are extracted from the two different scale features output by the feature extraction network; finally, a multi-loss joint optimization strategy is used according to the obtained features to optimize the model, and finally, the similarity between the query image and the gallery image features is used to determine the query result. The application can significantly improve the accuracy and efficiency of visible light infrared pedestrian re-identification and has a wide application prospect in many fields requiring pedestrian identity recognition, such as security monitoring.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision image retrieval, in particular to a cross-modal pedestrian re-identification model based on cross-scale information interaction, a method and an application thereof. BACKGROUND

[0002] Pedestrian re-identification aims to retrieve the images of the same pedestrian under different cameras from the database, and is widely used in the fields of intelligent security and intelligent life. With the development of deep learning, its accuracy is significantly improved compared with traditional hand-crafted feature extraction method, but there is still room for improvement. Modern cameras are often equipped with visible light and infrared light to ensure night work, but most mainstream pedestrian re-identification models rely on visible light image recognition, and it is difficult to effectively recognize infrared light images, so the cross-modal pedestrian re-identification task appears, and its goal is to realize the mutual retrieval of different modal (visible light and infrared light) pedestrian pictures, but unlike visible light cameras, infrared cameras can work in low light or dark environment, and the imaging principles of the two are different. The former relies on light source and reflected light, and the image contains rich color, detail and other information, while the latter relies on object thermal radiation imaging, and the image does not contain color information, which makes the pictures taken by the two exist modal difference, and in addition to the modal difference between different spectral cameras, it is also affected by many factors such as pedestrian posture change, occlusion, and light change, which makes it difficult for existing models to achieve the actual demand for recognition accuracy. SUMMARY

[0003] In view of the shortcomings of the prior art, the present application provides a cross-modal pedestrian re-identification model and method based on cross-scale information interaction. In order to fully exploit the modal invariant information in a wide range, we introduce a cross-modal confusion and a cross-scale information interaction module based on the constraint of auxiliary information comparison loss. The double modal difference between visible light images and infrared light images, pedestrian posture change and background interference and other influencing factors are overcome, and high-precision cross-modal pedestrian re-identification prediction results are provided, realizing cross-modal pedestrian re-identification.

[0004] To achieve the above purpose, the technical scheme provides a cross-modal pedestrian re-identification method based on cross-scale information interaction, comprising the following steps:

[0005] S1, obtaining a pair of visible light images and infrared images, inputting the visible light images into a cross-modal confusion for pretreatment to generate a pseudo-infrared image, and after expanding the pseudo-infrared image and the infrared image, a training set is constituted;

[0006] S2, constructing a cross-modal pedestrian re-identification network model based on cross-scale information interaction, wherein the cross-modal pedestrian re-identification network model based on cross-scale information interaction adopts an improved Resnet50The network framework comprises a separate parameter feature extraction network and a parameter-shared modal shared feature extraction network,

[0007] The separate parameter feature extraction network adopts two parallel Resnet50 The first-stage feature extraction network in the framework extracts visible light image features and infrared image features respectively,

[0008] The parameter-shared modal shared feature extraction network comprises Resnet50 The second to fifth-stage feature extraction networks in the framework and a cross-scale information interaction module, the cross-scale information interaction module is inserted into the second to fourth-stage feature extraction networks respectively;

[0009] S3, training the cross-modal pedestrian re-identification network model based on cross-scale information interaction based on the training data in the training data set, comprising:

[0010] The visible light image features and the infrared image features are input into the parameter-shared modal shared feature extraction network to obtain multi-scale modal shared feature representations,

[0011] Based on the two scale features output by the fourth to fifth-stage feature extraction networks obtained by the parameter-shared modal shared feature extraction network, the global features and local feature representations of the visible light image and the infrared image of two modal images are obtained after normalization and pooling,

[0012] According to the multi-scale global features and local features obtained by the model, the cross-entropy classification loss, the triplet loss and the cross-modal contrast loss are weighted and back-propagated to update the model parameters of the cross-modal pedestrian re-identification network model, the model parameters comprise Resnet50 The parameters in the parallel first-stage feature extraction network, the shared second to fifth-stage feature extraction network and the cross-scale information interaction module in the framework;

[0013] S4, obtaining a query image and a gallery image inputting the trained cross-modal pedestrian re-identification network model, and obtaining a retrieval result according to the similarity between the two.

[0014] Further, the visible light image therein is input into a cross-modal confusion to generate a pseudo-infrared image, comprising pre-processing according to the following process:

[0015] (19)

[0016] wherein, represents the original infrared image, represents the infrared image after channel alignment processing with the visible light image, transfer represents aligning the format of the single channel repeated three times with the three channels of the visible light image, represents the original visible light image, represents the pseudo-infrared image generated after the cross-modal confusion processing, H , W represents the height and width of the image, represents the image transformation operation adopted by the cross-modal confusion, which includes a weighted grayscale transformation, a cross-channel information confusion transformation, and a spectral dithering, the execution probability of each of the above three transformations is p1, p2, and p3, respectively, and the detailed process of the three transformations is as follows:

[0017] Weighted grayscale transformation: the visible light image is divided into three channels, and then fused using the following formula:

[0018] (20)

[0019] wherein, , , represents the information of the red, green, and blue three channels of the visible light image, , , is a weight factor randomly generated in the range of with a sum of 1, represents the visible light image after the weighted grayscale transformation;

[0020] The cross-channel information confusion transformation of the visible light image is performed according to the following formula:

[0021] (21)

[0022] wherein, , represents that two of the three channels separated from the visible light image are randomly selected as foreground and background, respectively, and then a random rectangle is cut from the foreground and pasted into the background to finally form a single image ;

[0023] The transformation formula of spectral dithering is as follows:

[0024] (22)

[0025] (23)

[0026] wherein, represents that one channel is randomly selected and repeated 3 times to obtain a degenerate spectral image, x chThis represents a single channel of information randomly selected from the three channels of a visible light image, with the information from the other channels then replaced by the selected channel. x ch Thus, a degenerate spectral image is obtained. , Represents the weighting factor, in Randomly select values ​​within the range. ,when When it is 0, That is, to represent a degenerate spectral image, when When it is 1, Represents the original visible light image. This represents a visible light image after spectral transformation.

[0027] Furthermore, the feature extraction network for separating parameters employs two parallel... Resnet50 The first-stage network in the framework extracts visible light image features and infrared image features respectively, using the following formula:

[0028] (twenty four)

[0029] ResnetStage1 express Resnet50 The first-stage feature extraction network in the framework, , These represent visible light image features and infrared image features obtained through a specific modality feature extraction network, respectively.

[0030] Furthermore, the cross-scale information interaction module uses channel-compressed feature maps to generate queries, and uses channel-compressed and spatial-compressed features to generate keys and values. Cross-scale interaction is achieved through multi-head self-attention, and spatial and channel dimensions are restored through fully connected layers and bilinear interpolation. This module includes three components: spatial channel compression, cross-scale interaction, and spatial channel restoration, used to process data obtained through a modality sharing network. ResnetStagei Image features are represented as , where i represents Resnet The output of the i-th stage of the framework, where i can take the value of 2, 3, or 4. Including visible light image features obtained by a separation parameter feature extraction network. and infrared image features It is assembled along the batch dimension and achieves cross-scale information interaction according to the following formula:

[0031] (25)

[0032] (26)

[0033] (27)

[0034] wherein denotes the batch-normalized image feature, wherein Conv denotes the channel compression operation with a convolution kernel of achieves a channel compression ratio of , Pooling denotes the spatial dimension compression operation with an average pooling kernel of achieves a channel compression ratio of , and are flattened and then input into the cross-scale information interaction component which adopts multi-head attention, wherein the flattened serves as the query, while the flattened serves as the key and value, and the output feature of the multi-head attention is denoted as ; the obtained feature is input into the spatial channel restoration component, wherein FC denotes a fully connected layer for restoring the channel dimension, and then the spatial dimension is restored through Sigmoid and Upsample operations to obtain Mask , and Mask and the batch-normalized result are element-wise multiplied to obtain the final output of this module , and the obtained is input into Resnet the i+1 th stage.

[0035] Further, based on the parameter-shared modal-shared feature extraction network, the two scale features output by the fourth and fifth stage feature extraction network are obtained, and after normalization and pooling, the global feature and local feature representation of the visible light image and infrared image of two modal images are obtained, including:

[0036] The two scale features output by the fourth and fifth stage feature extraction network are respectively defined as , and for , the global feature branch is used to perform generalized average pooling to obtain the global feature, and the local feature branch divides the image into multiple equal local blocks along the horizontal direction, and performs generalized average pooling on each equal local block to obtain the local region feature, and the specific formula is as follows:

[0037] (28)

[0038] (29)

[0039] wherein H , W ,C express The height, width, and number of channels of the feature map Channels in the feature map c The upper coordinate is (i, j) The elements, among which , p It is an adjustable parameter. Gem () indicates that a generalized average pooling operation is performed on the feature. Partition This indicates a horizontal partitioning operation that yields equal local feature maps.

[0040] in express Resnet The global features obtained by generalized average pooling from the output of the fifth stage of the framework N Local features, Indicates the process Resnet The output of the fourth stage of the framework and the inserted cross-scale information interaction module, then subjected to generalized average pooling to obtain global features and M Local features.

[0041] Furthermore, based on the multi-scale global and local features obtained by the model, the model parameters of the cross-modal person re-identification network model are updated by weighting and backpropagating cross-entropy classification loss, triplet loss, and cross-modal contrastive loss. The calculation process for each loss is as follows:

[0042] First, features are obtained through a multilayer perceptron. ,right A cross-modal contrastive loss is established as a constraint, and this loss is used... The formula is expressed as follows:

[0043] (30)

[0044] (31)

[0045] (32)

[0046] in, W 1, W 2 represents affine transformation. BN Indicates batch normalization, ReLU It is an activation function. In VI This indicates that the anchor represents visible light features, while the other samples involved in the loss correspond to infrared light features. I vis This represents the set of visible light image samples in the current training batch. Let represent the set of the K least dissimilar infrared samples that have the same ID as visible sample i. is the normalized feature application L2 normalized feature is the feature corresponding to the sample extracted from the set normalized feature ; and is the feature corresponding to the sample extracted from the set , and is the set of all samples in the infrared modality with different IDs from i , and all samples in , and the mean of and

[0047] for and , respectively, the cross-entropy classification loss is calculated to constrain, and the formula is as follows:

[0048] (33)

[0049] where is the probability of the classifier correctly predicting the feature, E denotes expectation;

[0050] for global feature , the cross-entropy classification loss and the triplet loss are calculated respectively, and for local feature , the cross-entropy classification loss is calculated respectively, and the formula is as follows:

[0051] (34)

[0052] (35)

[0053] where is the probability of the classifier correctly predicting the feature, E denotes expectation; denotes the value of the triplet loss, is the distance between the anchor sample and the positive sample, is the distance between the anchor sample and the negative sample, margin is a manually set interval parameter to ensure that there is enough distance difference between the positive and negative samples;

[0054] The final total multi-loss joint constraint function is expressed as follows:

[0055] (36)

[0056] wherein and are hyperparameters used to balance the contribution of each loss function.

[0057] Compared with the prior art, the present application has the following advantages:

[0058] The application provides a cross-modal pedestrian re-identification method based on cross-scale information interaction. The method can fully mine the modal invariant information in a wide range. A cross-modal confusion device and a cross-scale information interaction module are introduced, and a comparison loss constraint based on auxiliary information is used. The method effectively reduces the modal difference between visible light images and infrared light images, pedestrian posture changes, background interference and other influencing factors, and provides a high-precision cross-modal pedestrian re-identification prediction result, and has a wide application prospect in many fields requiring pedestrian identity recognition such as security monitoring. BRIEF DESCRIPTION OF DRAWINGS

[0059] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or the prior art description will be briefly introduced. Obviously, the drawings in the following description are some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0060] Figure 1 A flowchart of a cross-modal pedestrian re-identification method based on cross-scale information interaction in an embodiment of the present application.

[0061] Figure 2 A model architecture diagram of a cross-modal pedestrian re-identification method based on cross-scale information interaction in an embodiment of the present application.

[0062] Figure 3 A principle diagram of a cross-scale information interaction module in an embodiment of the present application.

[0063] Figure 4 A transformation effect diagram of a cross-modal confusion device in an embodiment of the present application. DETAILED DESCRIPTION

[0064] In order to enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor should be within the scope of protection of the present application.

[0065] It should be noted that the terms "first", "second", and the like in the description and claims of the application and the above drawings are used to distinguish similar objects, and do not necessarily indicate a specific order or sequence. It should be understood that the data thus used can be interchanged, where appropriate, so that the embodiments of the application described herein can be carried out in other than the order shown or described herein. Furthermore, the terms "comprise" and "have", and any variations thereof, are intended to cover non-exclusive inclusion, for example, processes, methods, systems, products, or devices that include a series of steps or units without being limited to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products, or devices.

[0066] As shown in Figure 1 The application provides a cross-modal pedestrian re-identification method based on cross-scale information interaction, mainly comprising the following steps:

[0067] S1, a pair of visible light images and infrared images are obtained, and the pair of visible light images and infrared images are input into a cross-modal confusion device for preprocessing, the visible light images are converted into pseudo-infrared images close to the infrared images, so as to promote the model to mine the modal invariant information of the same identity, at the same time, the multi-channel color information existing in the original visible light images is fully utilized, and the transformed pairs of pseudo-infrared images and infrared images are expanded to form a training set. The specific expansion steps include random flipping, padding and Resize, etc.

[0068] Specifically, a plurality of groups of image groups composed of visible light images and infrared light images corresponding to and marked with the same pedestrian category are obtained as a training set. In order to realize the training and testing of the cross-modal pedestrian re-identification model based on cross-scale information interaction, a plurality of groups of image groups composed of visible light images and infrared light images corresponding to and marked with the same pedestrian category are also obtained as a verification set and a test set. The visible light images and infrared light images in the training set are processed to adjust the proportion to 394*192. Since there is a large modal difference between the visible light images and the infrared images, before the data is input into the model, the cross-modal confusion device is first used to preprocess the visible light images, which not only increases the diversity of the samples, but also enables the model to learn more robust feature representations. The processed training set is input into the cross-modal pedestrian re-identification framework for training. The training set is divided into 80 batches for training the cross-modal pedestrian re-identification framework, and the batch processing size of each batch of training set is 48. Specifically, each batch of training set corresponds to 6 different pedestrians, and each pedestrian corresponds to 4 visible light images and 4 infrared light images.

[0069] The visible light image is input into the cross-modal confusion to generate a pseudo-infrared image, including preprocessing according to the following process:

[0070] (37)

[0071] wherein, represents the original infrared image, represents the infrared image after channel alignment processing with the visible light image, and transfer represents aligning the single channel to three channels of the visible light image three times. represents the original visible light image, represents the pseudo-infrared image generated after processing by the cross-modal confusion, H , W represents the height and width of the image, represents the image transformation operation adopted by the cross-modal confusion, which includes weighted grayscale transformation, cross-channel information confusion transformation, and spectral dithering, and the execution probability of each of the above three transformations is p1, p2, and p3, respectively. The detailed process of the three transformations is shown in Figure 4 , including:

[0072] Weighted grayscale transformation: the visible light image is divided into three channels, and then fused using the following formula:

[0073] (38)

[0074] wherein, , , respectively represent the red, green, and blue three channels of the visible light image, , , is a random generated weight factor with a range of and a sum of 1, represents the visible light image after weighted grayscale transformation;

[0075] The cross-channel information confusion transformation is performed on the visible light image according to the following formula:

[0076] (39)

[0077] wherein, , respectively represent randomly selecting two channels from the three channels separated from the visible light image as foreground and background, respectively, and then cutting a random rectangle from the foreground and pasting it into the background to finally form a single image ;

[0078] The formula for transforming spectral jitter is as follows:

[0079] (40)

[0080] (41)

[0081] in, This means randomly selecting one channel and repeating the process three times to obtain a degenerate spectral image. x ch This represents a single channel of information randomly selected from the three channels of a visible light image, with the information from the other channels then replaced by the selected channel. x ch Thus, a degenerate spectral image is obtained. , Represents the weighting factor, in Randomly select values ​​within the range. ,when When it is 0, That is, to represent a degenerate spectral image, when When it is 1, Represents the original visible light image. This represents a visible light image after spectral transformation.

[0082] S2. Construct a cross-modal pedestrian re-identification network model based on cross-scale information interaction. The cross-modal pedestrian re-identification network model based on cross-scale information interaction includes a cross-modal obfuscator, a feature extraction and embedding network, and a multi-loss joint optimization strategy. The cross-modal obfuscator uses techniques such as weighted grayscale, channel mixing, and spectral dithering to process paired visible light images and infrared light images. The feature extraction and embedding network is used to extract image features of the two input modalities.

[0083] The feature extraction and embedding networks in this application include: the backbone network of a cross-modal person re-identification network based on cross-scale information interaction adopts a... ResNet50 The network consists of a deep residual network, an infrared feature extraction network, a visible light feature extraction network, and a fully connected layer, among which parameter separation is employed. Resnet50 The first-stage feature extraction network is used to extract specific modality features. ResNet50The last four stages of the parameter sharing are used as a modal shared feature extraction network, and a cross-scale information interaction module is inserted in the second, third and fourth stages, so that the modal invariant information in a large range is fully mined, and two scale features are obtained, and global features and local features are obtained through a global feature branch and a local feature branch, wherein a generalized average pooling method is used, and the local features are obtained by dividing the features into equal local feature blocks, and a generalized average pooling method is used, and the two scale global features and local features are used for loss constraint, and the model weight is updated through back propagation according to the loss value.

[0084] S3, training the cross-modal pedestrian re-identification network model based on cross-scale information interaction based on the training data in the training data set, comprising: inputting the visible light image features and the infrared image features into the parameter sharing modal shared feature extraction network to obtain multi-scale modal shared feature representation, obtaining two scale features output by the fourth to fifth stage feature extraction network based on the parameter sharing modal shared feature extraction network, obtaining global features and local feature representations of visible light images and infrared images after normalization and pooling, and updating the model parameters of the cross-modal pedestrian re-identification network model by using the cross entropy classification loss, the ternary loss and the cross-modal contrast loss and back propagation based on the multi-scale global features and local features obtained by the model, comprising Resnet50 The first stage feature extraction network in the framework is parallel, the second to fifth stage feature extraction networks are shared, and the parameters in the cross-scale information interaction module are shared.

[0085] In the present application, the random gradient descent method (SGD) is used as the optimizer, the momentum is 0.9, the warm-up method is used to preheat in the first 10 rounds, the learning rate is linearly increased from 0.01 to 0.1, and then the learning rate is reduced to one tenth of the original at the 50th and 80th rounds.

[0086] The training process of the cross-modal pedestrian re-identification network model is described below:

[0087] S301, constructing a model input, inputting a pair of visible light images and infrared images into a cross-modal confusion device to obtain a model input.

[0088] Specifically, a plurality of groups of visible light images and infrared light images corresponding to and labeled as the same person are obtained as a training set. Specifically, the present application obtains a plurality of groups of visible light images and infrared light images corresponding to and labeled as the same person from an existing cross-modal pedestrian re-identification data set as a training set.

[0089] In order to realize the training and testing of the cross-modal pedestrian re-identification model based on cross-scale information interaction, a plurality of groups of visible light images and infrared images corresponding to and labeled with the same pedestrian category are obtained as a verification set and a test set.

[0090] S302, the visible light images in the training set are preprocessed, and the processing mode is shown in formulas (37)-(41).

[0091] At the same time, random flipping, random erasing and other common transformations are applied to the visible light images and infrared images respectively, and a group of visible light image features and a group of infrared image features obtained through the above series of transformations are represented by and respectively, wherein the visible light pictures and infrared pictures of 6 specific identity pedestrians each contain 4 pictures, and are composed of and .

[0092] S303, based on the obtained model input, input it into the parameter-separated specific modal feature extractor to obtain the visible light image features and the infrared image features respectively. The specific modal feature extraction network is constructed, and the Resnet50 framework is used as the first stage feature extraction network. The visible light image is input into the visible light image feature extraction network to obtain the visible light image feature, and the infrared image is input into the visible light image feature extraction network to obtain the visible light image feature:

[0093] (42)

[0094] ResnetStage1 Resnet50 The first stage feature extraction network in the framework is represented by , , respectively.

[0095] S304, based on the obtained modal specific features, input the shared parameter feature extraction network and a plurality of cross-scale information interaction modules to obtain the modal shared features.

[0096] The obtained visible light image features and infrared images are spliced in the batch dimension to obtain the features .

[0097] The obtained is input Resnet50 into the feature extraction network of the second stage of the framework to obtain new i+1 , and the in the new is i before i ​An increment operation was performed on top of that.

[0098] Determine whether the current stage is the second, third, or fourth stage. If so, input it into the cross-scale information interaction module.

[0099] The cross-scale information interaction module generates queries using channel-compressed feature maps and generates keys and values ​​using both channel-compressed and spatial-compressed features. Cross-scale interaction is achieved through multi-head self-attention, and spatial and channel dimensions are restored using fully connected layers and bilinear interpolation. This module consists of three interconnected components: spatial channel compression, cross-scale interaction, and spatial channel restoration. It is used to connect features obtained through a modality sharing network. ResnetStagei Image features are represented as , where i represents Resnet The output of the i-th stage of the framework, where i can take the value of 2, 3, or 4. Including visible light image features obtained by a separation parameter feature extraction network. and infrared image features It is pieced together along the batch dimension. First, the obtained Input space channel compression component:

[0100] (43)

[0101] in Represents the image features after batch normalization, where Conv Indicates adoption The convolution kernel achieves a channel compression ratio of Channel dimension compression operation, Pooling This indicates that a pooling kernel size of 1 is used. The average pooling achieves a channel compression ratio of Compression of spatial dimensions;

[0102] Input the obtained features into the cross-scale information interaction component:

[0103] (44)

[0104] Previously generated and Features are flattened and then input into a cross-scale information interaction component, which employs multi-head attention. The flattened features... Act as a query ( Query ), and flattened Act as a key ( Key ) and value ( Value The output features of multi-head attention are represented as follows: .

[0105] The obtained feature is input into a space channel recovery component:

[0106] (45)

[0107] wherein FC represents a full connection layer for recovering the channel dimension, and then passing through Sigmoid and Upsample operations to recover the spatial dimension to obtain Mask , and element-wise multiplication of Mask and the batch normalization result obtains the final output of this module , which is denoted as .

[0108] After the second to fourth stages and the cross-scale interaction module after each stage, the obtained is input into Resnet50 the fifth stage of the framework to obtain .

[0109] S305, based on the last two scales of features obtained by the shared feature extraction network, global features and regional features of the two modal images are obtained.

[0110] After generalized average pooling, the global feature is obtained, and then is divided into M local blocks with equal size along the height dimension: After generalized average pooling, M local features are obtained:

[0111] (46)

[0112] wherein H , W , C represents the height, width and channel number of the feature map, represents the element in the channel with coordinates c in the feature map, wherein (i,j) , is an adjustable parameter, which is set to 3 in this example. Partition represents a horizontal division operation, obtaining p N local blocks, and Gem represents performing the pooling operation in formula (46) on each local feature to obtain N pooled local features.

[0113] Similarly, after generalized average pooling, the global feature is obtained, and then is divided into M local blocks with equal size along the height dimension: ​The system is horizontally divided into N equal-sized local blocks along the height dimension. For each local block, generalized average pooling is applied to obtain N local features:

[0114] (47)

[0115] Where Partition represents the horizontal partitioning operation, resulting in There are N local blocks in total. Gem represents the pooling operation in formula (47) performed on each local feature to obtain... There are a total of N local features after pooling.

[0116] S306. Based on the obtained global and regional features, the model is optimized using a multi-loss optimization strategy. The query image and the image in the library are input into the trained model, and the retrieval results are output based on the similarity between the two.

[0117] First, features are obtained through a multilayer perceptron. ,right A cross-modal contrastive loss is established as a constraint, and this loss is used... The formula is expressed as follows:

[0118] (48)

[0119] (49)

[0120] (50)

[0121] in, W 1, W 2 represents affine transformation. BN Indicates batch normalization, ReLU It is an activation function. In VI This indicates that the anchor represents visible light features, while the other samples involved in the loss correspond to infrared light features. I vis This represents the set of visible light image samples in the current training batch. Let represent the set of the K least dissimilar infrared samples that have the same ID as visible sample i. Yes application L2 Features obtained after normalization; It refers to a set Features extracted from samples ;and Indicates from set The features corresponding to the samples extracted from them. Indicates that the infrared mode has the same characteristics as ia set of all samples with different IDs, and all samples in the set. Similarly, for the case of anchor as infrared feature, the corresponding loss is The final loss constraint is and the mean of

[0122] For and , cross-entropy classification loss is calculated respectively to constrain, and the formula is as follows:

[0123] (51)

[0124] wherein is the probability of the classifier correctly predicting the feature, E denotes expectation;

[0125] For global feature , cross-entropy classification loss and triplet loss are calculated respectively, and for local feature , cross-entropy classification loss is calculated respectively, and the formula is as follows:

[0126] (52)

[0127] (53)

[0128] wherein is the probability of the classifier correctly predicting the feature, E denotes expectation; denotes triplet loss value, is the distance between the anchor sample and the positive sample, is the distance between the anchor sample and the negative sample, margin is a manually set interval parameter to ensure that there is enough distance difference between the positive sample and the negative sample;

[0129] The final total multi-loss joint constraint function is expressed as follows:

[0130] (54)

[0131] wherein and are hyperparameters for balancing the contribution of each loss function.

[0132] S4, obtaining a query image and a gallery image inputting the trained cross-modal pedestrian re-identification network model, and obtaining a retrieval result according to the similarity between the two.

[0133] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, and are not intended to limit the present application; although the present application has been described in detail with reference to the above embodiments, those skilled in the art should understand that the technical solutions recorded in the above embodiments can be modified, or some or all of the technical features can be replaced by equivalents; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A cross-modal person re-identification method based on cross-scale information interaction, characterized in that, Includes the following steps: S1. Acquire pairs of visible light images and infrared images, input the visible light images into a cross-modal obfuscator for preprocessing to generate pseudo infrared images, and expand the pseudo infrared images and infrared images to form a training set. S2. Construct a cross-modal person re-identification network model based on cross-scale information interaction. The cross-modal person re-identification network model based on cross-scale information interaction adopts an improved... Resnet50 The network framework includes a separate parameter feature extraction network and a parameter-sharing modality-sharing feature extraction network. The separation parameter feature extraction network employs two parallel... Resnet50 The first-stage feature extraction network in the framework extracts features from visible light images and infrared images, respectively. The parameter-sharing modality-sharing feature extraction network includes Resnet50 The framework includes second to fifth stage feature extraction networks and a cross-scale information interaction module, with the cross-scale information interaction module being inserted into the second to fourth stage feature extraction networks, respectively. S3. Train the cross-modal person re-identification network model based on cross-scale information interaction using the training data in the training dataset, including: Visible light image features and infrared image features are input into a parameter-shared modality-shared feature extraction network to obtain multi-scale modality-shared feature representations. The two scale features output from the fourth and fifth stage feature extraction networks based on parameter sharing are normalized and pooled to obtain global and local feature representations of visible light and infrared images. Based on the multi-scale global and local features obtained by the model, the model parameters of the cross-modal person re-identification network model are updated by weighting and backpropagating cross-entropy classification loss, triplet loss, and cross-modal contrastive loss. The model parameters include... Resnet50 The framework includes a parallel first-stage feature extraction network, shared second to fifth-stage feature extraction networks, and parameters in the cross-scale information interaction module. S4. Input the query image and the image from the library into the trained cross-modal pedestrian re-identification network model, and obtain the retrieval results based on the similarity between the two.

2. The cross-modal person re-identification method based on cross-scale information interaction according to claim 1, characterized in that, The visible light image is input into a cross-modal obfuscator for preprocessing to generate a pseudo-infrared image, including preprocessing according to the following procedure: (1) in, Represents the original infrared image. This represents the infrared image after channel alignment with the visible light image. `transfer` indicates that the single channel is repeated three times to align with the three-channel format of the visible light image. Represents the original visible light image. This represents the pseudo-infrared image generated after processing by the cross-modal obfuscator. H , W Indicates the height and width of the image. This indicates the image transformation operations used by the cross-modal obfuscator. These operations include weighted grayscale transformation, cross-channel information obfuscation transformation, and spectral dithering. The execution probabilities for each of these three transformations are p1, p2, and p3, respectively. The detailed processes for these three transformations are as follows: Weighted grayscale transformation: The visible light image is divided into three channels, and then they are merged using the following formula: (2) in, , , These represent the information from the red, green, and blue channels of a visible light image, respectively. , , The range of random generation is Weighting factors whose sum is 1, This represents a visible light image that has undergone weighted grayscale transformation. Perform cross-channel information obfuscation transformation on visible light images according to the following formula: (3) in, , These represent two channels randomly selected from the three channels separated from the visible light image, respectively, as the foreground and background. Then, the foreground is processed... Cut a random rectangle and paste it to the background. In the end, they are composed of a single image. ; The formula for transforming spectral jitter is as follows: (4) (5) in, This means randomly selecting one channel and repeating the process three times to obtain a degenerate spectral image. x ch This represents a single channel of information randomly selected from the three channels of a visible light image, with the information from the other channels then replaced by the selected channel. x ch Thus, a degenerate spectral image is obtained. , Represents the weighting factor, in Randomly select values ​​within the range. ,when When it is 0, That is, to represent a degenerate spectral image, when When it is 1, Represents the original visible light image. This represents a visible light image after spectral transformation.

3. The cross-modal person re-identification method based on cross-scale information interaction according to claim 1, characterized in that, The feature extraction network for separating parameters employs two parallel methods. Resnet50 The first-stage network in the framework extracts visible light image features and infrared image features respectively, using the following formula: (6) ResnetStage1 express Resnet50 The first-stage feature extraction network in the framework, , These represent visible light image features and infrared image features obtained through a specific modality feature extraction network, respectively.

4. A cross-modal person re-identification method based on cross-scale information interaction according to claim 1, characterized in that, The cross-scale information interaction module generates queries using channel-compressed feature maps, and generates keys and values ​​using both channel-compressed and spatial-compressed features. Cross-scale interaction is achieved through multi-head self-attention, and spatial and channel dimensions are restored using fully connected layers and bilinear interpolation. The cross-scale information interaction module comprises three components: spatial channel compression, cross-scale interaction, and spatial channel restoration, used to transmit information through a modality sharing network. ResnetStagei Image features are represented as , where i represents Resnet The output of the i-th stage of the framework, where i can take the value of 2, 3, or 4. Including visible light image features obtained by a separation parameter feature extraction network. and infrared image features It is assembled along the batch dimension and achieves cross-scale information interaction according to the following formula: (7) (8) (9) in Represents the image features after batch normalization, where Conv Indicates adoption The convolution kernel achieves a channel compression ratio of Channel dimension compression operation, Pooling This indicates that a pooling kernel size of 1 is used. The average pooling achieves a channel compression ratio of Spatial dimension compression operation, producing and Features are flattened and then input into a cross-scale information interaction component, which employs multi-head attention. The flattened features... Serving as a query, and flattened Acting as keys and values, the output features of multi-head attention are represented as follows: The obtained features Input space channel recovery component, in which FC This represents a fully connected layer used to recover the channel dimension, which is then processed... Sigmoid and Upsample Operation recovery space dimension obtained Mask and will Mask The results of batch normalization The final output of this module is obtained by multiplying element by element. and will obtain enter Resnet No. i+1 Each stage.

5. The cross-modal person re-identification method based on cross-scale information interaction according to claim 1, characterized in that, The two scale features output by the fourth and fifth stage feature extraction networks based on parameter sharing are normalized and pooled to obtain global and local feature representations of visible light and infrared images, including: The two scale features output by the fourth and fifth stage feature extraction networks are defined as follows: ,for The global feature branch is used to perform generalized average pooling to obtain the global features, and the local feature branch is used to divide it into multiple equal local blocks along the horizontal direction. Generalized average pooling is then performed on each equal local block to obtain the local region features. The specific formula is as follows: (10) (11) in, H , W , C express The height, width, and number of channels of the feature map Channels in the feature map c The upper coordinate is (i, j) The elements, among which , p It is an adjustable parameter. Gem () indicates that a generalized average pooling operation is performed on the feature. Partition This indicates a horizontal partitioning operation that yields equal local feature maps. This represents the global features and N local features obtained by generalized average pooling from the output of the fifth stage. Indicates the process Resnet The output of the fourth stage of the framework and the inserted cross-scale information interaction module, then subjected to generalized average pooling to obtain global features and M Local features.

6. The cross-modal person re-identification method based on cross-scale information interaction according to claim 1, characterized in that, Based on the multi-scale global and local features obtained by the model, the model parameters of the cross-modal person re-identification network model are updated by weighting and backpropagating cross-entropy classification loss, triplet loss, and cross-modal contrastive loss. The calculation process of each loss is as follows: First, features are obtained through a multilayer perceptron. ,right A cross-modal contrastive loss is established as a constraint, and this loss is used... The formula is expressed as follows: (12) (13) (14) in, W 1, W 2 represents affine transformation. BN Indicates batch normalization, ReLU It is an activation function. In VI This indicates that the anchor represents visible light features, while the other samples involved in the loss correspond to infrared light features. I vis This represents the set of visible light image samples in the current training batch. Let represent the set of the K least dissimilar infrared samples that have the same ID as visible sample i. Yes application L2 Features obtained after normalization; It refers to a set Features extracted from samples ;and Indicates from set The features corresponding to the samples extracted from them. Indicates that the infrared mode has the same characteristics as i The set of all samples with different IDs, and For all samples in the dataset, similarly, for the case where the anchor is an infrared feature, the corresponding loss is... The final loss constraint is and The mean, against and The cross-entropy classification loss is calculated separately and constrained as follows: (15) in It is the probability that the classifier correctly predicts the feature. E Expressing expectations; For global features The cross-entropy classification loss and triplet loss are calculated separately for each feature, targeting local features. The cross-entropy classification loss is calculated for each of them, and the formula is expressed as follows: (16) (17) in It is the probability that the classifier correctly predicts the feature. E Expressing expectations; This represents the triplet loss value. It is the distance between the anchor sample and the positive sample. It is the distance between the anchor sample and the negative sample. margin It is a manually set interval parameter to ensure that there is a sufficient distance difference between positive and negative samples; The final overall multi-loss joint constraint function is expressed as follows: (18) in and It is a hyperparameter used to balance the contribution of each loss function.

Images