Pedestrian re-identification model construction method, identification method, equipment and medium

Abstract

The invention discloses a construction method of a pedestrian re-identification model, an identification method, equipment and a medium, and relates to the field of computer vision, and the technical scheme is characterized in that an image set only containing a pedestrian foreground is acquired; the image set comprises a visible light image and an infrared light image; constructing a neural network; wherein the construction of the neural network specifically comprises the following steps: inserting a feature generator between a convolutional layer and a maximum pooling layer of a deep residual network, and respectively connecting a channel feature fusion module to the output of a first bottleneck residual block, a second bottleneck residual block and a third bottleneck residual block of the deep residual network, the spatial feature fusion module and the feature generation module are sequentially connected at the output of the channel feature fusion module of the third bottleneck residual block; and training a neural network by using the image set, and constraining the training process of the neural network in combination with a pre-configured loss function until the loss function converges, thereby obtaining a pedestrian re-identification model with the pedestrian re-identification capability.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of computer vision, more particularly, to a pedestrian re-identification model construction method and device, a pedestrian re-identification method and device, and a medium. BACKGROUND

[0002] With the rapid development of artificial intelligence technology and the concept of intelligent manufacturing entering the public's field of vision, video monitoring devices are also developing in the direction of more intelligence and specialization to ensure the safe and stable operation of various industries and maintain the long-term stability of society. Among many intelligent monitoring technologies, face recognition has received widespread attention because it can quickly locate and identify target identities through facial features. However, due to the low resolution of most monitoring devices, the accuracy of face recognition is limited, affecting the actual application effect of intelligent monitoring in various fields. At the same time, the growing demand for public safety and the continuous increase in the number of monitoring devices have led to a significant increase in video data. Traditional manual monitoring methods require a large amount of manpower and resources, making it difficult to meet the efficient retrieval needs in complex monitoring environments. Therefore, pedestrian re-identification technology has gradually become an important research direction in the fields of computer vision, autonomous driving, and intelligent manufacturing. It can match identities through overall or local features of pedestrians without relying on facial features, providing a more efficient solution for intelligent monitoring.

[0003] The main goal of pedestrian re-identification is to retrieve and identify identities based on pedestrian image features in cross-camera scenarios, that is, given a query object, the pedestrian re-identification technology determines whether the target appears in another scene, is captured by different cameras at different time periods, or is captured by the same camera at different time periods. This technology has wide application prospects, mainly in the following aspects: first, in the field of intelligent transportation and autonomous driving, vehicles can effectively predict pedestrian behavior through the interaction between different angles and different types of cameras, making safe decisions; in service robots and intelligent manufacturing systems, robots often operate in dynamic human-robot interaction environments, and only by accurately tracking and matching staff or visitors can the safety and effectiveness of collaborative operations be ensured.

[0004] In the current traditional pedestrian re-identification task, visible light images and infrared light images cannot be interactively retrieved (interactive retrieval: inputting a visible light modality image can retrieve the corresponding identity of the pedestrian from the infrared image, and vice versa), which prevents all-weather identification and monitoring (all-weather monitoring: all-weather includes daytime and nighttime. If interactive retrieval can be achieved, the corresponding identity retrieval can be performed in the dark based on the images taken during the day, and vice versa). SUMMARY

[0005] The application aims to provide a pedestrian re-identification model construction method, a recognition method, a device and a medium, and solve the problem that current visible light images and infrared light images cannot be interactively searched.

[0006] The above technical objective of the application is achieved by the following technical solutions. In a first aspect, the application provides a pedestrian re-identification model construction method, which comprises the following steps. An image set containing only pedestrian foreground is acquired; the image set comprises visible light images and infrared light images; A neural network is constructed; wherein the construction of the neural network is specifically as follows: a feature generator is inserted between a convolution layer and a max-pooling layer of a deep residual network, a channel feature fusion module is connected to the output of a first bottleneck residual block, a second bottleneck residual block and a third bottleneck residual block of the deep residual network respectively, and a spatial feature fusion module and a feature generation module are connected to the output of the channel feature fusion module of the third bottleneck residual block in sequence. The neural network is trained by using the image set, and the training process of the neural network is constrained in combination with a pre-configured loss function until the loss function converges, so as to obtain a pedestrian re-identification model with pedestrian re-identification capability.

[0007] In an implementation scheme, the feature generator comprises a batch normalization layer and a ReLU activation function layer connected in sequence. The visible light images and the infrared light images are processed in sequence according to a convolution layer, a batch normalization layer, a ReLU activation function layer and a max-pooling layer respectively to obtain initial visible light features and initial infrared light features, the initial visible light features and the initial infrared light features are spliced to obtain a first feature vector; wherein the first feature vector is used as the input of the first bottleneck residual block.

[0008] In an implementation scheme, the first bottleneck residual block is used for feature extraction of the first feature vector to obtain a second feature vector. The channel feature fusion module connected to the first bottleneck residual block is used for performing a global average pooling operation, an adaptive mapping operation and a sigmoid activation function on the first feature vector and the second feature vector in sequence respectively, calculating the weights of the first feature vector and the second feature vector respectively, multiplying the first feature vector and the second feature vector by the respective weights to obtain corresponding third feature vectors and fourth feature vectors. The second bottleneck residual block is configured to perform two times of convolution operation on the third feature vector with a size of 1*1 to obtain two converted third feature vectors, perform one time of convolution operation on the fourth feature vector with a size of 1*1 to obtain a converted fourth feature vector, multiply one converted third feature vector with the converted fourth feature vector, and then perform normalization processing to calculate a channel similarity matrix, multiply another converted third feature vector with the channel similarity matrix to obtain a fifth feature vector of aggregated multi-level feature channel information. The channel feature fusion module connected with the second bottleneck residual block is configured to perform convolution operation on the fifth feature vector with a size of 1*1, and then perform addition operation on the fifth feature vector and the second feature vector to obtain a sixth feature vector. The third bottleneck residual block is configured to perform feature extraction on the sixth feature vector to obtain a seventh feature vector. The channel feature fusion module connected with the third bottleneck residual block is configured to perform, in sequence, global average pooling operation, adaptive mapping operation and sigmoid activation function on the sixth feature vector and the seventh feature vector respectively to calculate weights of the sixth feature vector and the seventh feature vector respectively, multiply the sixth feature vector and the seventh feature vector with the respective weights to obtain an eighth feature vector and a ninth feature vector corresponding thereto. The spatial feature fusion module is configured to perform two times of convolution operation on the eighth feature vector with a size of 1*1 to obtain two converted eighth feature vectors, perform one time of convolution operation on the ninth feature vector with a size of 1*1 to obtain a converted ninth feature vector, multiply one converted eighth feature vector with the converted ninth feature vector, and then perform normalization processing to calculate a spatial similarity matrix, multiply another converted eighth feature vector with the spatial similarity matrix to obtain a tenth feature vector of aggregated multi-level feature channel information and spatial information. The feature generation module includes three convolution layers with a size of 3*3, which are configured to perform feature extraction on the tenth feature vector to capture feature information under different receptive fields, average the feature information extracted by each convolution layer with a size of 3*3 to obtain an eleventh feature vector, and splice the eleventh feature vector with the tenth feature vector to obtain a feature map; wherein each convolution layer adopts different edge padding coefficients and inflation rates.

[0009] In an implementation scheme, the deep residual network further includes a fourth bottleneck residual block configured to extract features in the feature map and crop the features into initial visible light features, initial infrared light features, generated visible light features and generated infrared light features.

[0010] In an implementation scheme, the loss function is a weighted sum function of a first difficult sample triple loss, a cross-entropy loss, a second difficult sample triple loss and an orthogonal loss.

[0011] In an implementation scheme, an expression of the second hard sample triplet loss is: ; wherein, ; wherein, P denote the total number of classes of the initial visible light feature, the initial infrared light feature, the generated visible light feature and the generated infrared light feature respectively, denote the sample center of the i-th class of the initial infrared light feature, denote the sample center of the i-th class of the initial visible light feature, denote the sample center of the i-th class of the generated visible light feature, denote the sample center of the j-th class of the generated visible light feature, denote the sample center of the k-th class of the initial visible light feature, is; ; is, is, is.

[0012] In an implementation scheme, an expression of the orthogonal loss is ; wherein, N denote the batch size, is the inner product of the first feature vector and the eleventh feature vector.

[0013] In a second aspect, the present application provides a pedestrian re-identification method, the method comprising: obtaining a pedestrian image to be identified; inputting the pedestrian image to be identified into a pedestrian re-identification model for identification to obtain a pedestrian re-identification result.

[0014] In a third aspect, the present application provides an electronic device, characterized by comprising a memory and a processor; the memory is used for storing a computer program, the computer program comprising program instructions; the processor is used for executing the program instructions to enable the electronic device to perform the steps of the pedestrian re-identification model construction method according to the first aspect of the present application and the steps of the pedestrian re-identification method according to the second aspect of the present application.

[0015] In a fourth aspect, the present application provides a computer readable storage medium, characterized by comprising a computer program, the computer program being executed by one or more processors to implement the pedestrian re-identification model construction method according to the first aspect of the present application and the pedestrian re-identification method according to the second aspect of the present application.

[0016] Compared with the prior art, the present application has the following beneficial effects: The network structure of the current initial deep residual network is adjusted, a feature generator is inserted between the convolutional layer and the maximum pooling layer of the deep residual network, a channel feature fusion module is connected to the output of the first bottleneck residual block, the second bottleneck residual block and the third bottleneck residual block of the deep residual network respectively, a spatial feature fusion module and a feature generation module are connected to the output of the channel feature fusion module of the third bottleneck residual block in turn, the channel feature information and the spatial feature information of the visible light mode and the infrared light mode are fused through the channel feature fusion module, the spatial feature fusion module and the feature generation module, and finally the feature information of the fused features under different receptive fields is captured by the feature generation module, so that more rich feature representation is obtained, and the problem that the visible light mode and the infrared light mode cannot be interactively searched is solved. Finally, the improved deep residual network is trained by using a loss function, in the loss function, the improved difficult sample triple loss, the cross entropy loss, the difficult sample triple loss and the linear loss are combined to constrain the training process of the network, wherein the improved difficult sample triple loss considers the most difficult sample pair in the training process, and the concept of feature center of the center triple loss is introduced, the influence of the non-most difficult sample pair on the training process is focused on, so that the deep residual network is better guided to perform feature alignment in the cross-modal task, so that the trained deep residual network has the cross-modal pedestrian re-identification capability. BRIEF DESCRIPTION OF DRAWINGS

[0017] The accompanying drawings, which are included to provide a further understanding of the embodiments of the present application and constitute a part of the application, do not constitute a limitation to the embodiments of the present application. In the drawings: Figure 1 A flowchart of a construction method of a pedestrian re-identification model provided for the embodiments of the present application is shown; Figure 2 A network structure diagram of a neural network provided for the embodiments of the present application is shown; Figure 3 A structure diagram of a feature generator provided for the embodiments of the present application is shown; Figure 4 A structure diagram of a channel feature fusion module and a spatial feature fusion module provided for the embodiments of the present application is shown; Figure 5 A structure diagram of a feature generation module provided for the embodiments of the present application is shown; Figure 6 A flowchart of a pedestrian re-identification method provided for the embodiments of the present application is shown. DETAILED DESCRIPTION

[0018] In order to make the objects, technical solutions and advantages of the present application clearer, further detailed description will be given to the present application in combination with embodiments and drawings, and the schematic embodiments of the present application and the description thereof are only used to explain the present application, but not limit the present application.

[0019] It should be noted that the term "include" or "may include" used in various embodiments of the present application indicates the existence of the claimed function, operation or element, and does not limit the addition of one or more functions, operations or elements. In addition, as used in various embodiments of the present application, the terms "include", "have" and their synonyms only mean to indicate specific features, numbers, steps, operations, elements, components or combinations of the foregoing, and should not be understood as first excluding the existence or addition of one or more other features, numbers, steps, operations, elements, components or combinations of the foregoing.

[0020] It should be understood that terms such as "first", "second" are only used for description purposes, and should not be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features. Therefore, the features defined with "first", "second" can explicitly or implicitly include one or more features. In the description of the present application, the meaning of "multiple" is two or more, unless otherwise specifically limited.

[0021] Figure 1 A flowchart of a method for constructing a pedestrian re-identification model according to an embodiment of the present application is shown in FIG. 1. Figure 1 As shown in FIG. 1, the method comprises: S101, obtaining an image set containing only pedestrian foreground; the image set includes visible light images and infrared light images.

[0022] In this embodiment, the open-source image data sets SYSU-MM01 and RegDB are foreground segmented, thereby obtaining an image set containing only pedestrian foreground.

[0023] The image segmentation method is as follows: a segmentation network based on the SAM2 model is used to extract a mask image containing a pedestrian contour. In SAM2, the image in the data set in which the pedestrian is located in the middle of the image is input as the first frame image, and the center point (h / 2, w / 2) of the image is set as the prompt information to assist image segmentation. Then, the pedestrian pixel points of the subsequent frames detected with the coordinate prompt information of the first frame image and the predicted mask as auxiliary information are used to extract the mask of the entire data set visible light pedestrian image.

[0024] Subsequently, the mask image is converted into a binary mask with values of 0 and 1. On this basis, the original image is compared with the extracted binary mask pixel by pixel: if the value of a pixel in the mask is 0, the corresponding pixel in the original image is set to 0, that is, the background pixels are removed; otherwise, if the value of a pixel in the mask is 1, the original value of the pixel in the original image is retained, thereby ensuring the complete retention of the pedestrian foreground information. After this processing, a segmented image containing only the pedestrian foreground information is finally obtained.

[0025] To reduce the color information irrelevant to the pedestrian identity in the visible light mode, which will affect the accuracy of the pedestrian retrieval and matching task, the R, G and B color channels of the visible light image in the data set are weighted by 0.299, 0.587 and 0.114 respectively (the weight is based on the CIE1931 luminosity function standard, and the visible light image converted by the weight is most similar to the infrared image at the pixel level), and the visible light image is converted into a grayscale image. By gradually changing the grayscale rate from 0 to 100% at a step of 10% each time, the remaining images are kept in the original format, and finally the grayscale ratio is determined to be 0.1.

[0026] In S102, a neural network is constructed. The construction of the neural network is specifically as follows: a feature generator is inserted between a convolutional layer and a maximum pooling layer of a deep residual network; a channel feature fusion module is connected to outputs of a first bottleneck residual block, a second bottleneck residual block and a third bottleneck residual block of the deep residual network respectively; and a spatial feature fusion module and a feature generation module are connected to an output of the channel feature fusion module of the third bottleneck residual block in sequence.

[0027] In the embodiment, the network structure of the deep residual network ResNet-50 mainly includes four residual stages, each of which is stacked by a plurality of bottleneck residual blocks with the same output feature map size and channel number. However, before the residual stages, the deep residual network further includes a convolutional layer and an initial maximum pooling layer. The stages are down-sampled (the spatial size is halved and the channel number is doubled) through the first bottleneck residual block. These are common knowledge of those skilled in the art, and of course the network structure of the bottleneck residual block itself is consistent with the prior art, and the embodiment does not make redundant description.

[0028] Please refer to Figure 3The feature generator comprises a batch normalization layer and a ReLU activation function layer connected sequentially. Specifically, a batch normalization layer and a ReLU activation function layer are inserted between the convolutional layer and the max pooling layer, thus forming an initial stage consisting of a convolutional layer, a batch normalization layer, a ReLU activation function layer, and a max pooling layer. This initial stage processes the visible light image and the infrared light image sequentially using the convolutional layer, the batch normalization layer, the ReLU activation function layer, and the max pooling layer to obtain initial visible light features and initial infrared light features. These initial visible light features and initial infrared light features are concatenated to obtain a first feature vector; this first feature vector serves as the input to the first bottleneck residual block.

[0029] Specifically, in cross-modal person re-identification tasks, both infrared and visible light images possess specific features and information. Directly mixing and extracting information from both modalities can lead to the loss of intra-modal features or difficulty in identification. Therefore, to ensure the specificity of features from both modalities as much as possible during initial feature extraction, a feature generator is used. For visible light image input: First, a convolutional layer with a kernel size of 7×7, a stride of 2, and a padding factor of 3 is used to convert the input channel number from 3 to 64 for initial feature extraction. Then, a batch normalization (BN) layer is used to adjust the mean and variance of the output, which helps to accelerate the training process and improve model stability. Next, the ReLU activation function is used to perform a non-linear transformation on the normalized feature map to enhance the model's expressive power. Finally, a max pooling layer is used, where features are downsampled through a max pooling layer with a pooling window of 3×3 and a stride of 2. This helps to reduce the size of the feature map, simplifying the calculation process, while retaining the most salient features. For infrared image input: except for the initial convolutional layer which converts the number of input channels from 1 to 64, the parameters of the remaining layers are consistent with those of the visible light input, so that the feature sizes initially extracted from both inputs are the same. The features extracted separately from the two modalities are concatenated and input into the next stage of the backbone network, namely the first bottleneck residual block.

[0030] like Figure 2 As shown, a hierarchical feature fusion module (HFF) is added to stage 1, the first bottleneck residual block of the ResNet50 network structure. Specifically, as... Figure 4As shown, the multi-level feature fusion module (HFF) is divided into two sub-modules: the channel feature fusion module (HFF-C) and the spatial feature fusion module (HFF-S). The HFF-C module is inserted into the first, second, and third stages of the ResNet50 backbone network to perform channel information fusion of features at different levels; while the HFF-S module is inserted after the third stage, focusing on spatial feature aggregation. Through the feature aggregation method provided in this embodiment, the details and spatial information of low-level features can be effectively transmitted in the network. This not only strengthens the supplementary role of low-level features to high-level features, but also makes the processing of details and global semantic information in images more comprehensive.

[0031] The first bottleneck residual block is used to extract features from the first feature vector to obtain the second feature vector. The channel feature fusion module connected to the first bottleneck residual block is used to perform global average pooling, adaptive mapping and sigmoid activation functions on the first feature vector and the second feature vector respectively, calculate the weights of the first feature vector and the second feature vector, and multiply the first feature vector and the second feature vector by their respective weights to obtain the corresponding third feature vector and fourth feature vector.

[0032] In HFF-C, the first eigenvector is first... Second eigenvector (in, C , H and W These represent the number of channels, the width of the feature, and its height, respectively. (This indicates) the global average pooling operation, adaptive mapping operation, and The activation function generates channel weights, and each weight is multiplied by its corresponding feature vector to obtain the third and fourth feature vectors, thereby enhancing the feature channel information. The expression for the third feature vector is: ; The expression for the fourth eigenvector is: In the formula: This indicates a global average pooling operation. This indicates an adaptive mapping operation. express sigmoid Normalization of functions.

[0033] The second bottleneck residual block is configured to perform two convolution operations with a size of 1*1 on the third feature vector to obtain two converted third feature vectors, perform a convolution operation with a size of 1*1 on the fourth feature vector to obtain a converted fourth feature vector, multiply the converted third feature vector and the converted fourth feature vector, and then perform normalization processing to calculate a channel similarity matrix, and multiply the other converted third feature vector and the channel similarity matrix to obtain a fifth feature vector of aggregated multi-level feature channel information.

[0034] Then, three convolution layers with a size of 1*1 are used 、 、 to convert the first feature vector and the second feature vector into 、 、 . Then, the converted third feature vector and the converted fourth feature vector are used to perform matrix multiplication and normalization operations to calculate a channel similarity matrix softmax , and the specific calculation method is as follows: . .

[0035] The other converted third feature vector and the channel similarity matrix are multiplied to realize channel information aggregation of multi-level features and obtain a fifth feature vector of aggregated multi-level feature channel information. Here, the multiplication operation of the matrix and the vector helps to enhance the attention of the deep residual network to important channel information through the complementarity of low-level features and high-level features, and further improves the expression ability of the features. In the aggregation process, the channel similarity matrix reflects the relationship between the low-level and high-level features, so that the low-level features and the high-level features can effectively share information and enhance the multi-level perception ability of the model during feature fusion.

[0036] The channel feature fusion module connected to the second bottleneck residual block is configured to perform a convolution operation with a size of 1*1 on the fifth feature vector, and then perform an addition operation with the second feature vector to obtain a sixth feature vector.

[0037] In this embodiment, next, to ensure that the size of the fifth feature vector obtained through the above operations is consistent with the size of the second feature vector, a convolution layer with a size of 1*1 is used to process the fifth feature vector. The convolution layer is configured to adjust the size of the fifth feature vector to match the spatial dimension of the second feature vector, so as to ensure effective feature fusion in the subsequent steps and avoid the problem of size mismatch.

[0038] Finally, the features obtained through the above operations are combined with the second feature vector Matrix addition operation is performed to finally obtain the sixth feature vector after channel enhancement The purpose of this operation is to combine the advantages of low-level features and high-level features, so that the final obtained features are more comprehensive and accurate, thereby improving the performance of the network in the cross-modal pedestrian re-identification task. In this way, the channel information is effectively enhanced, so that the model can better learn valuable high-level semantic information in multi-level features while retaining low-level detail information, and finally generate more rich and accurate feature representations. The sixth feature vector The calculation method is as follows: .

[0039] Please refer to Figure 2 The network structure in the stage3 part is composed of a third bottleneck residual block, a channel feature fusion module, a spatial feature fusion module, and a feature generation module. The third bottleneck residual block is used to extract features from the sixth feature vector to obtain the seventh feature vector. The channel feature fusion module connected to the third bottleneck residual block is used to perform global average pooling operation, adaptive mapping operation and sigmoid activation function on the sixth feature vector and the seventh feature vector respectively to calculate the weights of the sixth feature vector and the seventh feature vector. The sixth feature vector and the seventh feature vector are multiplied by the respective weights to obtain the eighth feature vector and the ninth feature vector corresponding thereto.

[0040] Specifically, in the third stage of ResNet50, the number of channels is large and rich in semantic features, but the spatial information is gradually lost. Therefore, the HFF-S is inserted in this stage to aggregate spatial features of the first feature vector and the second feature vector, making up for the lack of high-level deep features in the spatial dimension, thereby improving the ability of the deep residual network to capture local details and spatial structures.

[0041] The spatial feature fusion module is used to perform two 1*1 size convolution operations on the eighth feature vector to obtain two converted eighth feature vectors, and perform a 1*1 size convolution operation on the ninth feature vector to obtain a converted ninth feature vector. After multiplying one of the converted eighth feature vectors and the converted ninth feature vector, normalization processing is performed to calculate a spatial similarity matrix. The other converted eighth feature vector is multiplied by the spatial similarity matrix to obtain a tenth feature vector that aggregates multi-level feature channel information and spatial information.

[0042] The spatial feature fusion module is similar to the channel feature fusion module. First, the converted eighth feature vector and the fourth feature vector space similarity matrix between , i.e., using convolution, matrix multiplication and softmax is calculated by normalization, and the specific calculation method is as follows: It should be noted that the third feature vector and the fourth feature vector in the above are consistent in sign description but different in content.

[0043] Subsequently, the converted third feature vector and the space similarity matrix are multiplied, thereby realizing the spatial information aggregation of multi-level features, and then the convolution layer converts the size of the feature map obtained through the above operation into the size of the second feature vector . Finally, the features obtained through the above operation are added to the channel-strengthened sixth feature vector to obtain the tenth feature vector after multi-level space and channel strengthening, and the calculation method of the tenth feature vector is as follows:

[0044] Finally, the tenth feature vector obtained through the space information and semantic information aggregation operation is transmitted to the next layer of the network for subsequent processing. In this way, the aggregated feature map will provide more rich spatial and semantic information for the subsequent layers, thereby enhancing the expression ability of the model in complex tasks. This step helps to ensure that the network can fully utilize the features of each layer when processing the cross-modal pedestrian re-identification task, thereby improving the final recognition accuracy and robustness.

[0045] In summary, in the multi-level feature fusion module provided in the embodiment, the space and channel similarity matrices are calculated using low-level features and high-level features, and the high-level features are obtained based on the low-level feature space channel information strengthening.

[0046] The embodiment designs a diversified feature generation module to generate multi-scale features through a three-branch convolution structure. As shown in Figure 2 , the next layer of the spatial feature fusion module of the third residual block is the feature generation module, i.e., DFE, which includes three 3*3 size convolution layers for feature extraction of the tenth feature vector to capture feature information under different receptive fields. The feature information extracted by each 3*3 size convolution layer is added and averaged to obtain an eleventh feature vector, and the eleventh feature vector is spliced with the tenth feature vector to obtain a feature map; wherein each convolution layer uses different edge padding coefficients and expansion rates.

[0047] As shown in Figure 5 , the convolution layers of the three branches are , , Different edge padding coefficients and expansion rates are adopted, and are respectively set as (1, 2, 3), so that each branch can capture feature information under different receptive fields. This design allows the network to perform feature extraction at multiple scales, thereby obtaining more rich feature representations.

[0048] After the generation of the features of each branch, the outputs of the three branches are subjected to matrix addition and averaging to form an overall feature. In order to enhance the nonlinear expression ability of the features, the spliced features are subjected to a ReLU activation function to enhance the capture of high-frequency information and complex information. Next, a 1*1 convolution kernel is used to perform convolution operation on the spliced eleventh feature vector. Through this operation, the eleventh feature vector generated is ensured to have the same number of channels as the tenth feature vector f of the original input, while retaining diversified feature information, further enhancing the expression ability of the model and the sensitivity to local details. The calculation method of the generated eleventh feature vector is as follows: .

[0049] Finally, the generated eleventh feature vector and the tenth feature vector f are spliced to obtain the final overall feature map F, which is used for feature extraction in the fourth residual stage stage4.

[0050] In some embodiments, the deep residual network further includes a fourth bottleneck residual block for extracting features from the feature map and cropping the features into initial visible light features, initial infrared light features, generated visible light features, and generated infrared light features.

[0051] S103, the image set is used to train the neural network, and the training process of the neural network is constrained in combination with a pre-configured loss function until the loss function converges, to obtain a pedestrian re-identification model with pedestrian re-identification capability.

[0052] In this embodiment, the loss function is a weighted sum function of the first difficult sample triple loss, the cross-entropy loss, the second difficult sample triple loss, and the orthogonal loss.

[0053] Specifically, the main core of this embodiment is to propose an improved difficult sample triple loss function, i.e., the second difficult sample triple loss. The loss calculates the most difficult sample center pair between different pedestrians in the same modality and different modalities of the same pedestrian, and reduces the Euclidean distance thereof through gradient descent method, thereby reducing the differences between modalities and within modalities.

[0054] Cross-entropy loss is one of the most commonly used loss functions in classification problems, suitable for multi-class classification tasks. In the pedestrian re-identification task, cross-entropy loss is used to measure the difference between the predicted class probability and the true label. The core goal is to maximize the similarity between the model's prediction and the actual label, so that the network learns effective class distinguishing features. Optimization of cross-entropy loss helps improve the classification accuracy of the model, especially in a multi-modal data environment, ensuring matching effects between different modalities.

[0055] Assuming that during training, each batch N contains P pedestrian categories, based on the feature maps output by the deep residual network, the feature maps (such as initial visible light features, initial infrared light features, generated visible light features, and generated infrared light features) are input into the loss function, so that the interface outputs the predicted weight vector of the image as The probability that the feature map belongs to the pedestrian category is:

[0056]

[0057] The first difficult sample triplet loss is a loss function commonly used in metric learning, aiming to improve the model's discrimination ability by optimizing the distance between positive and negative samples. The core idea is to select the most difficult (Euclidean distance farthest) positive sample and the most difficult (Euclidean distance closest) negative sample compared to the anchor sample as the difficult triplet each time. Since the model pays more attention to those sample pairs that are most difficult to distinguish at the current stage during training, it can effectively improve the constraint ability of the model and improve the model's ability to recognize difficult samples.

[0058] Assuming that during training, each batch contains P categories, and each category contains K pedestrian images, the difficult sample triplet loss function formula is as follows: where, represents the anchor sample, represents the negative sample, represents the positive sample, represents a hyperparameter used to control the distance between positive and negative samples.

[0059] The second difficult sample triplet loss considers the most difficult sample pair during training, and also introduces the concept of center triplet loss feature center, focusing on the influence of non-difficult sample pairs on the training process to better guide the model to perform feature alignment in cross-modal tasks. In addition, when calculating the Euclidean distance, the Euclidean distance between different modalities is calculated separately, replacing the traditional way of calculating the overall difficult sample triplet loss, taking into account the differences between modalities.

[0060] Specifically, first, the backbone network fourth stage output features are cropped into initial visible light features , initial infrared light features , generated visible light features , and generated infrared light features , and the sample centers in each feature are calculated respectively, and the specific calculation formula is as follows: , wherein, i represents the number of categories, K represents the number of samples in each category, represents the feature of the sample i in the category j . Through the above calculation method, the initial visible light sample center , the original infrared sample center , the generated visible light sample center , and the generated infrared sample center are obtained.

[0061] Next, in order to reduce the difference between different modalities and avoid that the features of different categories are too close, the intra-class distance is less than the inter-class distance. The distance between the generated visible light features and the original infrared features should be reduced, and the distance between the original visible light features and the generated visible light features should be expanded, and the specific calculation method is as follows: , wherein, P represents the total number of categories of the initial visible light features, the initial infrared light features, the generated visible light features, and the generated infrared light features, represents the sample center of the i-th category of the initial infrared light features, represents the sample center of the i-th category of the initial visible light features, represents the sample center of the i-th category of the generated visible light features, represents the sample center of the j-th category of the generated visible light features, represents the sample center of the k-th category of the initial visible light features, is a hyperparameter.

[0062] Similarly, the distance between the generated infrared features and the original visible light features should also be reduced, and the distance between the original infrared features and the generated visible light features should be expanded, and the specific calculation method is as follows: , wherein, is, is, .

[0063] Based on the above-described L1 and L2, the second difficult sample triple loss is represented as: .

[0064] The orthogonal loss is a loss function used in deep learning to optimize the structure of the feature space, the core idea of which is to make the inner product between different features close to 0, so as to enhance the independence of the features and improve the expression ability of the model. It is commonly used to improve the discriminability of the feature space in neural networks, especially in tasks with high feature dimension and many categories, which can effectively avoid feature redundancy and enhance the discriminability of the model.

[0065] Since the diversified feature generation module adopts a three-branch convolutional structure, in order to enable the features generated by different branches to capture different information and reduce the overlap of elements, the above orthogonal loss function is introduced to constrain the training process, and the specific calculation formula is as follows: wherein, N represents the batch size, is the inner product of the first feature vector and the eleventh feature vector.

[0066] In summary, the loss function adopts the joint training strategy of cross-entropy loss, difficult sample triplet loss (first difficult sample triplet loss), improved difficult sample triplet loss (second difficult sample triplet loss), and orthogonal loss function to constrain the model training process.

[0067] The loss function formula is as follows: wherein, , is a hyperparameter used to balance the weights of the loss functions.

[0068] On the basis of the neural network described in step S102, the preprocessed data set is trained, and in the training process, each batch contains 8 pedestrian identities, each identity has 4 visible light images and 4 infrared light images, and the total training round is 120 batches (epochs). In addition, the training process uses a stochastic gradient descent optimizer (SGD) with a momentum parameter of 0.9, and adopts a learning rate warm-up decay strategy, which is an existing training strategy, and the present embodiment does not make redundant description. In the initial training stage, the learning rate is set to 0.01, and then it is linearly increased to 0.1 at the 10th training cycle (epoch). After entering the 20th epoch, the learning rate is decayed to 0.01, and is further decreased to 0.001 and 0.0001 at the 60th epoch and the 100th epoch, respectively.

[0069] On the basis of the training described in the present embodiment and as a technical means well known to those skilled in the art, after the training is completed, a pedestrian re-identification model with pedestrian re-identification capability can be obtained.

[0070] The embodiment of the present application also provides a pedestrian re-identification method, which comprises: S501, acquire a pedestrian image to be recognized.

[0071] In the embodiment, the image acquisition manner belongs to the common means known by those skilled in the art, and the embodiment does not make redundant description.

[0072] S502, input the pedestrian image to be recognized into a pedestrian re-identification model for recognition to obtain a pedestrian re-identification result.

[0073] In the embodiment, the execution of the corresponding identification task in the trained identification model is a conventional technical means, and the embodiment does not make redundant description.

[0074] The embodiment of the application further provides an electronic device. The electronic device comprises a processor, a memory, a communication interface and at least one communication bus for connecting the processor, the memory and the communication interface. The memory comprises, but is not limited to, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (PROM) or a portable read-only memory (CD-ROM), and is used for storing related instructions and data.

[0075] The communication interface is used for receiving and sending data. The processor can be one or more CPUs, and in the case of one CPU, the CPU can be a single-core CPU or a multi-core CPU. The processor in the electronic device is used to read one or more programs stored in the memory and perform the following operations: acquiring an image set containing only pedestrian foreground; the image set comprises a visible light image and an infrared light image; constructing a neural network; wherein the construction of the neural network is specifically: inserting a feature generator between the convolutional layer and the maximum pooling layer of the deep residual network, connecting a channel feature fusion module to the output of the first bottleneck residual block, the second bottleneck residual block and the third bottleneck residual block of the deep residual network respectively, and connecting a spatial feature fusion module and a feature generation module to the output of the channel feature fusion module of the third bottleneck residual block in sequence; training the neural network by using the image set, and combining a pre-configured loss function to constrain the training process of the neural network until the loss function converges, to obtain a pedestrian re-identification model with pedestrian re-identification capability.

[0076] It should be noted that the specific implementation of each operation can be described in the method embodiment of the above Figure 1 The electronic device can be used to execute a pedestrian re-identification model construction method of the method embodiment of the application, and details are not repeated here.

[0077] The embodiment of the present application further provides a computer readable storage medium, which is a memory device in a computer device and is used for storing programs and data. It can be understood that the computer readable storage medium herein can include an internal storage medium in the computer device and can also include an extended storage medium supported by the computer device. The computer readable storage medium provides a storage space, which stores an operating system of the terminal. In addition, one or more instructions suitable for being loaded and executed by the processor are also stored in the storage space, and the instructions can be one or more computer programs (including program codes). It should be noted that the computer readable storage medium herein can be a high-speed RAM memory or a non-volatile memory such as at least one disk memory. The one or more instructions stored in the computer readable storage medium can be loaded and executed by the processor to implement the corresponding steps of the pedestrian re-identification model construction method in the above embodiment. It should be understood by those skilled in the art that the embodiment of the present application can be provided as a method, a system or a computer program product. Therefore, the present application can be in the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware aspects. Moreover, the present application can be in the form of a computer program product implemented on one or more computer usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer usable program codes.

[0078] The embodiment of the present application further provides a computer program product containing program instructions. The computer program product can be a software or program product containing program instructions, which can be run on a computing device or stored in any available medium. When the computer program product is run on at least one electronic device, the at least one electronic device is caused to perform a pedestrian re-identification model construction method.

[0079] The above specific embodiments further explain the purpose, technical solutions and beneficial effects of the present application. It should be understood that the above description is only a specific embodiment of the present application and is not used to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the protection scope of the present application.

Claims

1. A method for constructing a pedestrian re-identification model, characterized in that the method... The method comprises: acquiring an image set containing only pedestrian foreground; the image set comprises a visible light image and an infrared light image; constructing a neural network; wherein the construction of the neural network is specifically: inserting a feature generator between a convolution layer and a maximum pooling layer of a deep residual network, connecting a channel feature fusion module to the output of a first bottleneck residual block, a second bottleneck residual block and a third bottleneck residual block of the deep residual network respectively, connecting a spatial feature fusion module and a feature generation module to the output of the channel feature fusion module of the third bottleneck residual block in sequence; training the neural network by using the image set, and combining a pre-configured loss function to constrain the training process of the neural network until the loss function converges, to obtain a pedestrian re-identification model with pedestrian re-identification capability. 2.The pedestrian re-identification model construction method of claim 1, wherein, The feature generator comprises a batch normalization layer and a ReLU activation function layer connected in sequence; the visible light image and the infrared light image are processed in sequence according to a convolution layer, a batch normalization layer, a ReLU activation function layer and a maximum pooling layer to obtain an initial visible light feature and an initial infrared light feature, and the initial visible light feature and the initial infrared light feature are spliced to obtain a first feature vector; wherein the first feature vector is used as the input of the first bottleneck residual block. 3.The pedestrian re-identification model construction method of claim 2, wherein, The first bottleneck residual block is used for feature extraction of the first feature vector to obtain a second feature vector. The channel feature fusion module connected with the first bottleneck residual block is used for performing a global average pooling operation, an adaptive mapping operation and a sigmoid activation function on the first feature vector and the second feature vector in sequence to calculate the weights of the first feature vector and the second feature vector respectively, multiplying the first feature vector and the second feature vector by the respective weights to obtain corresponding third feature vector and fourth feature vector. The second bottleneck residual block is used for performing twice 1*1 size convolution operation on the third feature vector to obtain two converted third feature vectors, performing once 1*1 size convolution operation on the fourth feature vector to obtain a converted fourth feature vector, multiplying one converted third feature vector and the converted fourth feature vector, and then performing normalization processing to calculate a channel similarity matrix, multiplying the other converted third feature vector with the channel similarity matrix to obtain a fifth feature vector fused with multi-level feature channel information. The channel feature fusion module connected with the second bottleneck residual block is used for performing 1*1 size convolution operation on the fifth feature vector, and then performing addition operation with the second feature vector to obtain a sixth feature vector. The third bottleneck residual block is used for feature extraction of the sixth feature vector to obtain a seventh feature vector. The channel feature fusion module connected with the third bottleneck residual block is used for performing a global average pooling operation, an adaptive mapping operation and a sigmoid activation function on the sixth feature vector and the seventh feature vector in sequence to calculate the weights of the sixth feature vector and the seventh feature vector respectively, multiplying the sixth feature vector and the seventh feature vector by the respective weights to obtain corresponding eighth feature vector and ninth feature vector. The spatial feature fusion module is configured to perform twice 1*1 convolution operation on the eighth feature vector to obtain two converted eighth feature vectors, perform once 1*1 convolution operation on the ninth feature vector to obtain a converted ninth feature vector, multiply the one converted eighth feature vector and the converted ninth feature vector, and then perform normalization processing to calculate a spatial similarity matrix, and multiply the other converted eighth feature vector and the spatial similarity matrix to obtain a tenth feature vector that fuses multi-level feature channel information and spatial information. The feature generation module includes three 3*3 convolution layers configured to perform feature extraction on the tenth feature vector to capture feature information under different receptive fields, average the feature information extracted by each 3*3 convolution layer to obtain an eleventh feature vector, and splice the eleventh feature vector and the tenth feature vector to obtain a feature map; and each convolution layer adopts different edge padding coefficients and inflation rates.

4. The pedestrian re-identification model construction method according to claim 3, characterized in that, The deep residual network further includes a fourth bottleneck residual block configured to extract features in the feature map and crop the features into initial visible light features, initial infrared light features, generated visible light features, and generated infrared light features.

5. The pedestrian re-identification model construction method according to claim 4, characterized in that, The loss function is a weighted sum function of a first difficult sample triple loss, a cross-entropy loss, a second difficult sample triple loss, and an orthogonal loss.

6. The pedestrian re-identification model construction method of claim 5, wherein, An expression of the second difficult sample triplet loss is: ; wherein, ; wherein, P denotes the total number of classes for the initial visible light features, the initial infrared light features, the generated visible light features, and the generated infrared light features, respectively, denotes the sample center of the i-th class of the initial infrared light features, denotes the sample center of the i-th class of the initial visible light features, denotes the sample center of the i-th class of the generated visible light features, denotes the sample center of the j-th class of the generated visible light features, denotes the sample center of the k-th class of the initial visible light features, is a hyperparameter; ; is, is, is.

7. The pedestrian re-identification model construction method of claim 5, wherein, The expression of the quadrature loss is ; wherein, N denotes the batch size, is the inner product of the first eigenvector and the eleventh eigenvector.

8. A pedestrian re-identification method, characterized in that, The method comprises: Obtaining a pedestrian image to be recognized; Inputting the pedestrian image to be recognized into a pedestrian re-identification model to perform recognition and obtain a pedestrian re-identification result.

9. An electronic device, comprising: comprise a memory and a processor; The memory is configured to store a computer program, and the computer program comprises program instructions; The processor is configured to execute the program instructions to enable the electronic device to perform the steps of the pedestrian re-identification model construction method according to any one of claims 1 to 7 and the steps of the pedestrian re-identification method according to claim 8.

10. A computer-readable storage medium, characterized in that, The computer readable storage medium comprises a computer program, and the computer program, when executed by one or more processors, implements the pedestrian re-identification model construction method according to any one of claims 1 to 7 and the pedestrian re-identification method according to claim 8.

Images