Video infrared-visible light pedestrian re-identification method, device, equipment and medium

Through the dual-stream network and feature aggregation network with weight sharing, the problem of modal differences and space-time information mining in video infrared-visible pedestrian re-identification is solved, and efficient pedestrian re-identification in dark environments is achieved, and the recognition accuracy and stability.

CN118918608APending Publication Date: 2024-11-08YUNNAN POWER GRID CO LTD ELECTRIC POWER RES INST
View PDF 0 Cites 1 Cited by

Patent Information

Application Number
CN202410982598.2
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-22
Publication Date
2024-11-08

AI Technical Summary

Technical Problem

Existing video infrared-visible pedestrian re-identification technology has challenges in the problems of unclear data capture and pose, similar appearance, occlusion, and frame aberration in dark environments. It is difficult to effectively solve modal differences and space-time information mining, and lack a unified framework. Solve these problems.

Method used

Using a dual-stream network with weight sharing, a spatial feature compensation network and a temporal feature aggregation network, preprocessing the video infrared-visible image sequence, maps to the same feature space, extracts cross-modal shared features, supplements and determines spatial information, and aggregates space-time information. Combining cross-entropy loss, triple loss and cross-modal alignment loss for end-to-end training, optimized pedestrian re-identification model.

Benefits of technology

The modal difference between video infrared-visible images is reduced, and robust spatial and temporal feature representations are extracted, which improves the accuracy and efficiency of pedestrian re-identification, and overcomes the difficulties of dark environments and space-time information mining.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118918608A_ABST
    Figure CN118918608A_ABST
Patent Text Reader

Abstract

The embodiment of the invention discloses a video infrared-visible light pedestrian re-identification method. The method comprises the following steps: acquiring a preprocessed video infrared-visible light pedestrian image sequence; constructing a model; a weight sharing double-flow network is adopted, the preprocessed image sequence is mapped to the same feature space, and a cross-modal sharing feature map is obtained; inputting a spatial feature compensation network, and supplementing global features of pedestrians according to the fine-grained features of each frame to determine pedestrian features after spatial information compensation; aggregating the pedestrian features after frame-level spatial information compensation into sequence-level pedestrian features containing spatio-temporal information according to a time feature aggregation network; training in an end-to-end mode, and constructing a loss function optimization model; and carrying out identification by using the optimized model. According to the scheme, robust time and space feature representation can be obtained through the designed space feature compensation network, time feature aggregation network and cross-modal alignment loss, cross-modal difference is reduced, and a cross-modal pedestrian retrieval task is effectively realized.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a method, device, equipment and medium for re-identifying pedestrians using infrared-visible light in videos. Background Art

[0002] Person re-identification (ReID) aims to match pedestrian images acquired from multiple non-overlapping camera views. It plays an important role in intelligent video surveillance and has made great progress in recent years. However, the basic idea in the past was to seek a discriminative embedding space so that the features of the same ID have greater similarity. Recent research has focused on image-based unimodal Re-ID, which captures data through visible light cameras and extracts discriminative features within a single image. Although good results have been achieved, there are still two obstacles to its application in 24-hour intelligent video surveillance. First, visible light cameras cannot capture clear data in dark environments; second, image-based feature extraction cannot effectively solve some challenging problems such as posture, appearance similarity, occlusion, and frame misalignment. Another solution to the above two problems is video-based visible-infrared person re-identification (VVI-ReID), which is a promising ReID technique that uses both visible light and infrared data to extract discriminative representations in video clips. The two main challenges faced by VVI-ReID are modality difference mitigation and spatiotemporal information mining. Modality difference mitigation has been widely studied in cross-modal person re-identification, such as methods based on image translation and methods based on modality-invariant feature learning. Although the motivations are different, these methods can be considered to mitigate modality differences under different learning states. Spatiotemporal information mining is an inevitable problem in video person re-identification. The methods to solve this problem mainly lie in how to aggregate the features of frames in video clips, such as methods based on temporal information aggregation and methods based on complementary information mining. The former regards each video clip as a directed sequence and introduces a module based on recurrent neural network (RNN) or a module based on 3D convolution (3D-CNN) to aggregate the features of all frames and extract temporal features; the latter aims to extract discriminative features by exploring the relationship between unordered frames. Although both cross-modal ReID and video-based ReID have achieved promising results, how to simultaneously address the above two challenges through a unified framework remains an open problem. Summary of the invention

[0003] Based on this, it is necessary to propose a video infrared-visible light pedestrian re-identification method, device, equipment and storage medium to address the above problems.

[0004] To achieve the above-mentioned purpose, the present application provides, in a first aspect, a video infrared-visible light pedestrian re-identification method, the method comprising:

[0005] Preprocessing the acquired video infrared-visible light pedestrian image sequence;

[0006] Constructing a video infrared-visible light pedestrian re-identification model, the model includes a weight-sharing dual-stream network, a spatial feature compensation network, and a temporal feature aggregation network;

[0007] A weight-sharing two-stream network is used to map the preprocessed video infrared-visible pedestrian image sequence into the same feature space to obtain a cross-modal shared feature map.

[0008] Inputting the cross-modal shared feature map into the spatial feature compensation network, supplementing the global features of the pedestrian according to the fine-grained features of each frame to determine the pedestrian features after spatial information compensation;

[0009] Aggregating the pedestrian features after the spatial information compensation at the frame level into sequence-level pedestrian features containing spatiotemporal information according to the temporal feature aggregation network;

[0010] The training is performed in an end-to-end manner, and the video infrared-visible light pedestrian re-identification model is optimized by combining a cross entropy loss function, a triplet loss, and a cross-modal alignment loss;

[0011] The optimized video infrared-visible light pedestrian re-identification model is used to complete the cross-modal pedestrian retrieval task.

[0012] In some implementations, inputting the cross-modal shared feature map into the spatial feature compensation network, and supplementing the global features of the pedestrian according to the fine-grained features of each frame to determine the pedestrian features after spatial information compensation, includes:

[0013] Inputting each frame of the cross-modal shared feature map into the pyramid pooling layer of the spatial feature compensation network to obtain multi-scale features, where the multi-scale features are features of different granularities of the pedestrian image;

[0014] The multi-scale features are subjected to transposed convolution and up-sampled to unify the size of the feature map to be consistent with the size of the cross-modal shared feature map;

[0015] The unified multi-scale features are concatenated in the channel dimension, and the average pooling and fully connected layers are used to reduce the dimension of the concatenated feature map to obtain multi-granular features;

[0016] The dimensions of the multi-granularity features and the features after average pooling of the cross-modal shared feature map are kept consistent, and the multi-granularity features are added to the features after average pooling to determine the pedestrian features after spatial information compensation in each frame.

[0017] In some embodiments, aggregating the pedestrian features after the spatial information compensation at the frame level into sequence-level pedestrian features containing spatiotemporal information according to the temporal feature aggregation network includes:

[0018] Inputting the cross-modal shared feature map of each frame into the temporal feature compensation network and performing a spatial convolution operation, wherein the width and height of the convolution kernel are consistent with the width and height of the cross-modal shared feature map;

[0019] A fully connected layer is used after the spatial convolution operation to reduce the number of channels to 1 and output the importance score of the frame;

[0020] Inputting the pedestrian features of each frame after spatial information compensation into the temporal feature aggregation network;

[0021] The temporal feature aggregation network determines the sequence-level pedestrian features containing spatiotemporal information in each frame according to the pedestrian features after spatial information compensation in each frame and the corresponding importance score of the frame.

[0022] In some embodiments, the temporal feature aggregation network determines the sequence-level pedestrian features of each frame containing spatiotemporal information according to the pedestrian features of each frame after spatial information compensation and the corresponding importance score of the frame, including:

[0023] according to Determine the sequence-level pedestrian features containing spatiotemporal information in each frame, where f seq represents the sequence-level pedestrian features containing spatiotemporal information, T is the number of image frames in a sequence, and A t is the importance score of the t-th frame, t∈[1,T], is the pedestrian feature after spatial information compensation in the tth frame.

[0024] In some embodiments, the end-to-end training is performed to optimize the video infrared-visible light pedestrian re-identification model by combining a cross entropy loss function, a triplet loss, and a cross-modal alignment loss, specifically including:

[0025] According to L=L id +L triplet +L ma Determine the total loss function;

[0026] The video infrared-visible light pedestrian re-identification model is optimized according to the total loss function, where L is the total loss function, L id is the cross entropy loss function, L triplet is the triplet loss, L ma is the cross-modal alignment loss.

[0027] In some embodiments, the cross entropy loss function, triplet loss, and cross-modal alignment loss are determined based on the sequence-level pedestrian features containing spatiotemporal information.

[0028] In some implementations, determining the cross entropy loss function, the triplet loss, and the cross-modal alignment loss according to the sequence-level pedestrian features containing spatiotemporal information specifically includes:

[0029] according to Determine the cross-modal alignment loss, where P represents the number of pedestrian identities, N represents the number of video sequences for each pedestrian, and They represent the visible light feature center and infrared feature center of the i-th pedestrian, respectively, and C i represents the mean of the visible light feature center and infrared feature center of the i-th pedestrian, and denote the jth visible light sequence-level feature of the i-th pedestrian and the jth infrared sequence-level feature of the i-th pedestrian, respectively. m1 and m2 are used to indicate the visible light and infrared modalities, v denotes the visible light modality, r denotes the infrared modality, and α is a hyperparameter used to control the minimum distance between the feature center of the positive sample (same identity) and the feature center of the negative sample (different identity). Represents the feature center of pedestrians with different identities from the i-th pedestrian.

[0030] To achieve the above-mentioned purpose, the second aspect of the present application provides a video infrared-visible light pedestrian re-identification device, the device comprising:

[0031] A preprocessing module, used for preprocessing the acquired video infrared-visible light pedestrian image sequence;

[0032] A model building module, used to build a video infrared-visible light pedestrian re-identification model, the model includes a weight-sharing dual-stream network, a spatial feature compensation network, and a temporal feature aggregation network;

[0033] A cross-modal sharing module is used to map the pre-processed video infrared-visible pedestrian image sequence into the same feature space using a two-stream network with weight sharing to obtain a cross-modal shared feature map;

[0034] A spatial feature compensation module, used to input the cross-modal shared feature map into the spatial feature compensation network, and to supplement the global features of the pedestrian according to the fine-grained features of each frame to determine the pedestrian features after spatial information compensation;

[0035] A temporal feature aggregation module, configured to aggregate the pedestrian features after the spatial information compensation at the frame level into sequence-level pedestrian features containing temporal and spatial information according to the temporal feature aggregation network;

[0036] A model training optimization module, for training in an end-to-end manner, combining a cross entropy loss function, a triplet loss, and a cross-modal alignment loss to optimize the video infrared-visible light pedestrian re-identification model;

[0037] The recognition module is used to apply the optimized video infrared-visible light pedestrian re-identification model to complete the cross-modal pedestrian retrieval task.

[0038] To achieve the above-mentioned object, the third aspect of the present application provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the following steps:

[0039] Preprocessing the acquired video infrared-visible light pedestrian image sequence;

[0040] Constructing a video infrared-visible light pedestrian re-identification model, the model includes a weight-sharing dual-stream network, a spatial feature compensation network, and a temporal feature aggregation network;

[0041] A weight-sharing two-stream network is used to map the preprocessed video infrared-visible pedestrian image sequence into the same feature space to obtain a cross-modal shared feature map.

[0042] Inputting the cross-modal shared feature map into the spatial feature compensation network, supplementing the global features of the pedestrian according to the fine-grained features of each frame to determine the pedestrian features after spatial information compensation;

[0043] Aggregating the pedestrian features after the spatial information compensation at the frame level into sequence-level pedestrian features containing spatiotemporal information according to the temporal feature aggregation network;

[0044] The training is performed in an end-to-end manner, and the video infrared-visible light pedestrian re-identification model is optimized by combining a cross entropy loss function, a triplet loss, and a cross-modal alignment loss;

[0045] The optimized video infrared-visible light pedestrian re-identification model is used to complete the cross-modal pedestrian retrieval task.

[0046] To achieve the above-mentioned purpose, the fourth aspect of the present application provides a computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, the processor executes the following steps:

[0047] Preprocessing the acquired video infrared-visible light pedestrian image sequence;

[0048] Constructing a video infrared-visible light pedestrian re-identification model, the model includes a weight-sharing dual-stream network, a spatial feature compensation network, and a temporal feature aggregation network;

[0049] A weight-sharing two-stream network is used to map the preprocessed video infrared-visible pedestrian image sequence into the same feature space to obtain a cross-modal shared feature map.

[0050] Inputting the cross-modal shared feature map into the spatial feature compensation network, supplementing the global features of the pedestrian according to the fine-grained features of each frame to determine the pedestrian features after spatial information compensation;

[0051] Aggregating the pedestrian features after the spatial information compensation at the frame level into sequence-level pedestrian features containing spatiotemporal information according to the temporal feature aggregation network;

[0052] The training is performed in an end-to-end manner, and the video infrared-visible light pedestrian re-identification model is optimized by combining a cross entropy loss function, a triplet loss, and a cross-modal alignment loss;

[0053] The optimized video infrared-visible light pedestrian re-identification model is used to complete the cross-modal pedestrian retrieval task.

[0054] The embodiments of the present invention have the following beneficial effects:

[0055] The present invention designs an end-to-end video infrared-visible light pedestrian re-identification framework. For the extraction of spatiotemporal information in video sequences, we design a spatial feature compensation network and a temporal feature aggregation network to extract fine-grained features in space and inject them into the global features after average pooling. In terms of time, we design an attention mechanism to emphasize the important frames in the video sequence, use the attention weight to judge the importance of each frame, propose noise frames, aggregate multi-frame features to utilize the motion information of pedestrians, and finally learn robust spatial and temporal feature representations. For the modal difference problem between video infrared-visible light images, we design a modal alignment loss to impose constraints on the features of video infrared-visible light images in the feature space, bring the visible and infrared features of the same pedestrian closer, and push the features of pedestrians of different identities farther away to reduce cross-modal differences, so as to reduce cross-modal differences and achieve accurate and efficient pedestrian re-identification. In addition, the motion information of pedestrians itself also has modal invariance. Therefore, extracting effective spatiotemporal information also helps to overcome the modal difference between infrared-visible light images. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0057] in:

[0058] Figure 1 It is a flowchart of a method for pedestrian re-identification based on infrared-visible light in video in one embodiment;

[0059] Figure 2 is a structural diagram of a video infrared-visible light pedestrian re-identification device in one embodiment;

[0060] Figure 3 It is a flow chart of a video infrared-visible light pedestrian re-identification device in one embodiment;

[0061] Figure 4 A diagram showing a spatial compensation network structure in one embodiment;

[0062] Figure 5 A diagram of a time aggregation network structure in one embodiment;

[0063] Figure 6 is a schematic diagram of the structure of a computer device in one embodiment;

[0064] Figure 7 Schematic diagram of the structure of a computer-readable storage medium in one embodiment. DETAILED DESCRIPTION

[0065] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0066] In the embodiment of the present application, a method for pedestrian re-identification based on video infrared-visible light is provided. Figure 1 , Figure 1 1 is a flow chart of a method for pedestrian re-identification based on infrared-visible light video in one embodiment. The method for pedestrian re-identification based on infrared-visible light video includes steps S1 to S7.

[0067] Step S1, preprocessing the acquired video infrared-visible light pedestrian image sequence;

[0068] Specifically, an infrared-visible light pedestrian image sequence is obtained, and the infrared-visible light pedestrian image sequence in the training sample is preprocessed, including extracting several consecutive frames from the sequence and performing data enhancement operations such as scale normalization, random channel erasing or swapping, and random horizontal flipping and cropping.

[0069] Step S2, constructing a video infrared-visible light pedestrian re-identification model, wherein the model includes a weight-sharing dual-stream network, a spatial feature compensation network, and a temporal feature aggregation network;

[0070] Step S3, using a weight-sharing two-stream network to map the preprocessed video infrared-visible pedestrian image sequence to the same feature space to obtain a cross-modal shared feature map;

[0071] Specifically, in a weight-shared two-stream network, the infrared-visible light pedestrian image sequence is encoded into a common feature space, that is, cross-modal shared features are obtained, where the feature map is obtained before average pooling and the feature vector is obtained after pooling. In this application, the feature map is obtained after encoding in the weight-shared two-stream network and before pooling, that is, the cross-modal shared feature map.

[0072] Step S4, inputting the cross-modal shared feature map into the spatial feature compensation network, supplementing the global features of the pedestrian according to the fine-grained features of each frame to determine the pedestrian features after spatial information compensation;

[0073] Specifically, a spatial feature compensation network is constructed to supplement the pedestrian information with the fine-grained features of each frame, making the pedestrian information richer and more discriminable.

[0074] In some implementations, inputting the cross-modal shared feature map into the spatial feature compensation network, and supplementing the global features of the pedestrian according to the fine-grained features of each frame to determine the pedestrian features after spatial information compensation, includes:

[0075] Inputting each frame of the cross-modal shared feature map into the pyramid pooling layer of the spatial feature compensation network to obtain multi-scale features, where the multi-scale features are features of different granularities of the pedestrian image;

[0076] The multi-scale features are subjected to transposed convolution and up-sampled to unify the size of the feature map to be consistent with the size of the cross-modal shared feature map;

[0077] The unified multi-scale features are concatenated in the channel dimension, and the average pooling and fully connected layers are used to reduce the dimension of the concatenated feature map to obtain multi-granular features;

[0078] The dimensions of the multi-granularity features and the features after average pooling of the cross-modal shared feature map are kept consistent, and the multi-granularity features are added to the features after average pooling to determine the pedestrian features after spatial information compensation in each frame.

[0079] Specifically, the input of the spatial feature compensation network is the feature map F after the two-stream network is encoded and before pooling. t , that is, the cross-modal shared feature map F, the size is [B, T, C, H, W], where H and W are the height and width of the input image, C is the number of channels, B is the size of the batch, T is the number of image frames in a sequence, and the feature map F of the tth frame is t Take the feature map F as an example (t∈[1,T]), t The pyramid pooling layer input into the spatial feature compensation network obtains multi-scale features to represent the features of pedestrian images at different granularities. The multi-scale features are then transposed and convolved to upsample the small-scale feature maps. The feature map size is unified to H, W, and the multi-scale features are concatenated in the channel dimension. The feature map is then reduced in dimension using average pooling and fully connected layers to obtain multi-granular features. Keep Dimension size and feature map F t The feature f after average pooling t The dimensions are consistent, and finally the multi-granularity features And the feature f after average pooling t Add together to get the pedestrian features after spatial information compensation That is, the multi-granularity pedestrian feature is added to the global feature of the pedestrian to ensure that the final feature contains multi-granularity information. The formula expression of the network is as follows:

[0080]

[0081] Among them, SPP represents pyramid pooling operation, TR represents transposed convolution, Represents features of different scales, cat[,] represents feature concatenation operation, Avgpool represents average pooling operation, and FC represents fully connected layer.

[0082] Step S5, aggregating the pedestrian features after the spatial information compensation at the frame level into sequence-level pedestrian features containing spatiotemporal information according to the temporal feature aggregation network;

[0083] Specifically, a temporal feature aggregation network is constructed to aggregate the frame-level features after spatial feature compensation into the features of a video sequence, so as to utilize the motion information of pedestrians in the video sequence and enable the network to learn sequence-level pedestrian features containing temporal and spatial information;

[0084] In some embodiments, aggregating the pedestrian features after the spatial information compensation at the frame level into sequence-level pedestrian features containing spatiotemporal information according to the temporal feature aggregation network includes:

[0085] Inputting the cross-modal shared feature map of each frame into the temporal feature compensation network and performing a spatial convolution operation, wherein the width and height of the convolution kernel are consistent with the width and height of the cross-modal shared feature map;

[0086] A fully connected layer is used after the spatial convolution operation to reduce the number of channels to 1 and output the importance score of the frame;

[0087] Inputting the pedestrian features of each frame after spatial information compensation into the temporal feature aggregation network;

[0088] The temporal feature aggregation network determines the sequence-level pedestrian features containing spatiotemporal information in each frame according to the pedestrian features after spatial information compensation in each frame and the corresponding importance score of the frame.

[0089] In some embodiments, the temporal feature aggregation network determines the sequence-level pedestrian features of each frame containing spatiotemporal information according to the pedestrian features of each frame after spatial information compensation and the importance score of the corresponding frame, including:

[0090] according to Determine the sequence-level pedestrian features containing spatiotemporal information in each frame, where f seq represents the sequence-level pedestrian features containing spatiotemporal information, T is the number of image frames in a sequence, and A t is the importance score of the t-th frame, t∈[1,T], is the pedestrian feature after spatial information compensation in the tth frame.

[0091] Specifically, the input of the temporal feature aggregation network is the feature map F before pooling after dual-stream network encoding and the feature vector f of each frame after spatial feature compensation. spa , the size of the feature map F is [B, T, C, H, W], where H and W are the height and width of the input image, C is the number of channels, B is the batch size, and T is the number of image frames in a sequence. t For example, the feature map F t Input to a spatial convolution, whose convolution kernel width is W and height is H. A fully connected layer is used after the output of the convolution layer to reduce the number of channels to 1, and finally output a score A t ,t∈[1,T], is used to represent the importance score of the tth frame. The network is formulated as follows:

[0092] A t =FC(Conv(Ft ))

[0093]

[0094] Among them, Conv represents the spatial convolution operation, FC represents the fully connected layer, and f seq Represents sequence-level pedestrian features that contain spatiotemporal information.

[0095] Specifically, an importance weight is calculated for each frame, where unimportant frames have low weights and important frames have high weights.

[0096] It should be noted that since video sequences are composed of several consecutive frames, we only calculate the average of several consecutive frames in the temporal feature aggregation network (also known as temporal average pooling in the field). The motion information of pedestrians is the temporal information in the video sequence. Robustness in both space and time means that the features used for testing should be able to obtain spatial and temporal information in the video sequence.

[0097] Step S6, performing training in an end-to-end manner, combining a cross entropy loss function, a triplet loss, and a cross-modal alignment loss to optimize the video infrared-visible light person re-identification model;

[0098] Specifically, a cross-modal alignment loss is constructed to reduce the differences between features of pedestrians of the same identity but in different modalities in the same feature space. The cross entropy loss and triplet loss are combined, and the entire framework is trained in an end-to-end manner.

[0099] In some embodiments, the end-to-end training is performed to optimize the video infrared-visible light pedestrian re-identification model by combining a cross entropy loss function, a triplet loss, and a cross-modal alignment loss, specifically including:

[0100] According to L=L id +L triplet +L ma Determine the total loss function;

[0101] The video infrared-visible light pedestrian re-identification model is optimized according to the total loss function, where L is the total loss function, L id is the cross entropy loss function, L triplet is the triplet loss, L ma is the cross-modal alignment loss.

[0102] It should be noted that the loss is calculated for the final output features here, and the loss function back-propagates to optimize the parameters of the entire video infrared-visible light pedestrian re-identification model framework.

[0103] In some implementations, determining the cross entropy loss function, the triplet loss, and the cross-modal alignment loss according to the sequence-level pedestrian features containing spatiotemporal information specifically includes:

[0104] according to Determine the cross-modal alignment loss, where P represents the number of pedestrian identities, N represents the number of video sequences for each pedestrian, and They represent the visible light feature center and infrared feature center of the i-th pedestrian, respectively, and C i represents the mean of the visible light feature center and infrared feature center of the i-th pedestrian, and denote the jth visible light sequence-level feature of the i-th pedestrian and the jth infrared sequence-level feature of the i-th pedestrian, respectively. m1 and m2 are used to indicate the visible light and infrared modalities, v denotes the visible light modality, r denotes the infrared modality, and α is a hyperparameter used to control the minimum distance between the feature center of the positive sample (same identity) and the feature center of the negative sample (different identity). Represents the feature center of pedestrians with different identities from the i-th pedestrian.

[0105] Specifically, this loss is used to bring the visible and infrared features of the same pedestrian closer together, and push away the features of pedestrians of different identities to reduce cross-modal differences.

[0106] In some implementations, determining the cross entropy loss function, the triplet loss, and the cross-modal alignment loss according to the sequence-level pedestrian features containing spatiotemporal information specifically includes:

[0107] Train in an end-to-end manner, using the SGD optimizer to optimize the entire network;

[0108] The cross entropy loss function is used to optimize the network, and the expression is as follows:

[0109] Among them, y i represents the label of the i-th identity, and Respectively represent the visible light and infrared features of the i-th identity, W represents the common classifier of visible light and infrared modalities, m is used to indicate visible light and infrared modalities, v represents the visible light modality, r represents the infrared modality, B represents the Batch_size size, and this loss function is used to ensure that the learned sequence-level features are identity discriminative.

[0110] It should be noted that the number of identities of infrared and visible light samples is the same, and a classifier is used to classify cross-modal samples with the same identity into the same category.

[0111] In some implementations, determining the cross entropy loss function, the triplet loss, and the cross-modal alignment loss according to the sequence-level pedestrian features containing spatiotemporal information specifically includes:

[0112] The network is optimized using triplet loss, expressed as follows:

[0113]

[0114] Where P represents the number of pedestrian identities, represents the randomly selected anchor point feature, Represents the pedestrian features with the same identity as the anchor point (positive sample), represents the pedestrian features (negative samples) with different identities from the anchor points, α is a hyperparameter used to control the minimum distance between positive samples and negative samples, m is used to indicate visible light and infrared modalities, v represents the visible light modality, and r represents the infrared modality.

[0115] Step S7, applying the optimized video infrared-visible light pedestrian re-identification model to complete the cross-modal pedestrian retrieval task.

[0116] Specifically, the optimized video infrared-visible light pedestrian re-identification model is used to recognize the video infrared-visible light pedestrian image sequence to be identified, completing the cross-modal pedestrian retrieval task.

[0117] The video infrared-visible light pedestrian re-identification method proposed in the present invention is adopted, in which fine-grained features are used to supplement the visual information of pedestrians in space, and the attention weight is used to judge the importance of each frame in time. A noise frame is proposed, and multi-frame features are aggregated to utilize the motion information of pedestrians. Finally, a robust spatial and temporal feature representation is learned, which overcomes the modal difference of infrared and visible light. The modal alignment loss is used to bring the visible and infrared features of the same pedestrian closer, and push away the features of pedestrians of different identities, so as to reduce the cross-modal difference and achieve accurate and efficient pedestrian re-identification.

[0118] In the embodiment of the present application, an infrared-visible light pedestrian re-identification device is provided. Figure 2 and Figure 3 , Figure 2 A structural diagram of an infrared-visible light pedestrian re-identification device in one embodiment, wherein the infrared-visible light pedestrian re-identification device includes: a preprocessing module 201, a model building module 202, a cross-modal sharing module 203, a spatial feature compensation module 204, a temporal feature aggregation module 205, a model training optimization module 206 and a recognition module 207.

[0119] The preprocessing module 201 is configured to preprocess the acquired video infrared-visible light pedestrian image sequence;

[0120] A model building module 202 is configured to build a video infrared-visible light pedestrian re-identification model, wherein the model includes a weight-sharing dual-stream network, a spatial feature compensation network, and a temporal feature aggregation network;

[0121] The cross-modal sharing module 203 is configured to use a weight-sharing dual-stream network to map the pre-processed video infrared-visible pedestrian image sequence to the same feature space to obtain a cross-modal shared feature map;

[0122] The spatial feature compensation module 204 is configured to input the cross-modal shared feature map into the spatial feature compensation network, and supplement the global features of the pedestrian according to the fine-grained features of each frame to determine the pedestrian features after spatial information compensation;

[0123] A temporal feature aggregation module 205 is configured to aggregate the pedestrian features after the spatial information compensation at the frame level into sequence-level pedestrian features containing temporal and spatial information according to the temporal feature aggregation network;

[0124] The model training optimization module 206 is configured to perform training in an end-to-end manner, and optimize the video infrared-visible light pedestrian re-identification model by combining a cross entropy loss function, a triplet loss, and a cross-modal alignment loss;

[0125] The recognition module 207 is configured to apply the optimized video infrared-visible light pedestrian re-recognition model to complete the cross-modal pedestrian retrieval task.

[0126] In some embodiments, the spatial feature compensation module 204 is further configured to input each frame of the cross-modal shared feature map into the pyramid pooling layer of the spatial feature compensation network to obtain multi-scale features, where the multi-scale features are features of different granularities of the pedestrian image;

[0127] The multi-scale features are subjected to transposed convolution and up-sampled to unify the size of the feature map to be consistent with the size of the cross-modal shared feature map;

[0128] The unified multi-scale features are concatenated in the channel dimension, and the average pooling and fully connected layers are used to reduce the dimension of the concatenated feature map to obtain multi-granular features;

[0129] The dimensions of the multi-granularity features and the features after average pooling of the cross-modal shared feature map are kept consistent, and the multi-granularity features are added to the features after average pooling to determine the pedestrian features after spatial information compensation in each frame.

[0130] Specifically, see Figure 4 , Figure 4 FIG. 4 is a diagram showing a spatial compensation network structure in one embodiment.

[0131] In some embodiments, the temporal feature aggregation module 205 is further configured to input each frame of the cross-modal shared feature map into the temporal feature compensation network and perform a spatial convolution operation, wherein the width and height of the convolution kernel are consistent with the width and height of the cross-modal shared feature map;

[0132] A fully connected layer is used after the spatial convolution operation to reduce the number of channels to 1 and output the importance score of the frame;

[0133] Inputting the pedestrian features of each frame after spatial information compensation into the temporal feature aggregation network;

[0134] The temporal feature aggregation network determines the sequence-level pedestrian features containing spatiotemporal information in each frame according to the pedestrian features after spatial information compensation in each frame and the corresponding importance score of the frame.

[0135] In some embodiments, the time feature aggregation module 205 is further configured to Determine the sequence-level pedestrian features containing spatiotemporal information in each frame, where f seq represents the sequence-level pedestrian features containing spatiotemporal information, T is the number of image frames in a sequence, and A t is the importance score of the t-th frame, t∈[1,T], is the pedestrian feature after spatial information compensation in the tth frame.

[0136] Specifically, see Figure 5 , Figure 5 FIG. 4 is a diagram of a time aggregation network structure in one embodiment.

[0137] In some embodiments, the model training optimization module 206 is further configured to id +L triplet +L ma Determine the total loss function;

[0138] The video infrared-visible light pedestrian re-identification model is optimized according to the total loss function, where L is the total loss function, L id is the cross entropy loss function, L triplet is the triplet loss, L ma is the cross-modal alignment loss.

[0139] In some embodiments, the model training optimization module 206 is further configured to determine the cross entropy loss function, triplet loss, and cross-modal alignment loss based on the sequence-level pedestrian features containing spatiotemporal information.

[0140] In some embodiments, the model training optimization module 206 is further configured to Determine the cross-modal alignment loss, where P represents the number of pedestrian identities, N represents the number of video sequences for each pedestrian, and They represent the visible light feature center and infrared feature center of the i-th pedestrian, respectively, and C i represents the mean of the visible light feature center and infrared feature center of the i-th pedestrian, and denote the jth visible light sequence-level feature of the i-th pedestrian and the jth infrared sequence-level feature of the i-th pedestrian, respectively. m1 and m2 are used to indicate the visible light and infrared modalities, v denotes the visible light modality, r denotes the infrared modality, and α is a hyperparameter used to control the minimum distance between the feature center of the positive sample (same identity) and the feature center of the negative sample (different identity). Represents the feature center of pedestrians with different identities from the i-th pedestrian.

[0141] For other details about how the modules in the video infrared-visible light pedestrian re-identification device implement the above technical solution, please refer to the description of the video infrared-visible light pedestrian re-identification method provided above, which will not be repeated here.

[0142] In an embodiment of the present application, a computer device is provided. Figure 6 , Figure 6 3 is a schematic diagram of the structure of a computer device in an embodiment. The device includes a memory 301 and a processor 302. The memory 301 stores a computer program. When the computer program is executed by the processor 302, the processor 302 performs the following steps:

[0143] Preprocessing the acquired video infrared-visible light pedestrian image sequence;

[0144] Constructing a video infrared-visible light pedestrian re-identification model, the model includes a weight-sharing dual-stream network, a spatial feature compensation network, and a temporal feature aggregation network;

[0145] A weight-sharing two-stream network is used to map the preprocessed video infrared-visible pedestrian image sequence into the same feature space to obtain a cross-modal shared feature map.

[0146] Inputting the cross-modal shared feature map into the spatial feature compensation network, supplementing the global features of the pedestrian according to the fine-grained features of each frame to determine the pedestrian features after spatial information compensation;

[0147] Aggregating the pedestrian features after the spatial information compensation at the frame level into sequence-level pedestrian features containing spatiotemporal information according to the temporal feature aggregation network;

[0148] The training is performed in an end-to-end manner, and the video infrared-visible light pedestrian re-identification model is optimized by combining a cross entropy loss function, a triplet loss, and a cross-modal alignment loss;

[0149] The optimized video infrared-visible light pedestrian re-identification model is used to complete the cross-modal pedestrian retrieval task.

[0150] Among them, the processor 302 can also be called a CPU (Central Processing Unit), and the processor 302 may be an integrated circuit chip with signal processing capabilities; the processor 302 can also be a general-purpose processor, DSP (Digital Signal Process), ASIC (Application Specific Integrated Circuit), FPGA (Field Programmable Gate Array) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, among which the general-purpose processor can be a microprocessor or the processor 302 can also be any conventional processor, etc.

[0151] In an embodiment of the present application, a computer readable storage medium is provided. Figure 7 , Figure 7 The structure diagram of a computer-readable storage medium in an embodiment is a schematic diagram, on which a readable computer program 401 is stored; wherein the computer program 401 may be stored in the above storage medium in the form of a software product, and includes a number of instructions for causing a computer device (which may be a personal computer, a server machine, or a network device, etc.) or a processor to perform the following steps:

[0152] Preprocessing the acquired video infrared-visible light pedestrian image sequence;

[0153] Constructing a video infrared-visible light pedestrian re-identification model, the model includes a weight-sharing dual-stream network, a spatial feature compensation network, and a temporal feature aggregation network;

[0154] A weight-sharing two-stream network is used to map the preprocessed video infrared-visible pedestrian image sequence into the same feature space to obtain a cross-modal shared feature map.

[0155] Inputting the cross-modal shared feature map into the spatial feature compensation network, supplementing the global features of the pedestrian according to the fine-grained features of each frame to determine the pedestrian features after spatial information compensation;

[0156] Aggregating the pedestrian features after the spatial information compensation at the frame level into sequence-level pedestrian features containing spatiotemporal information according to the temporal feature aggregation network;

[0157] The training is performed in an end-to-end manner, and the video infrared-visible light pedestrian re-identification model is optimized by combining a cross entropy loss function, a triplet loss, and a cross-modal alignment loss;

[0158] The optimized video infrared-visible light pedestrian re-identification model is used to complete the cross-modal pedestrian retrieval task.

[0159] The aforementioned storage media include: U disk, mobile hard disk, magnetic disk or CD, ROM (Read-Only Memory), RAM (Random Access Memory) and other media that can store program code, or terminal devices such as computers, service machines, mobile phones, and tablets.

[0160] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the program can be stored in a non-volatile computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0161] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0162] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A method for pedestrian re-identification based on infrared-visible light in video, characterized in that: The method includes: Preprocessing the acquired video infrared-visible light pedestrian image sequence; Constructing a video infrared-visible light pedestrian re-identification model, the model includes a weight-sharing dual-stream network, a spatial feature compensation network, and a temporal feature aggregation network; A weight-sharing two-stream network is used to map the preprocessed video infrared-visible pedestrian image sequence into the same feature space to obtain a cross-modal shared feature map. Inputting the cross-modal shared feature map into the spatial feature compensation network, supplementing the global features of the pedestrian according to the fine-grained features of each frame to determine the pedestrian features after spatial information compensation; Aggregating the pedestrian features after the spatial information compensation at the frame level into sequence-level pedestrian features containing spatiotemporal information according to the temporal feature aggregation network; The training is performed in an end-to-end manner, and the video infrared-visible light pedestrian re-identification model is optimized by combining a cross entropy loss function, a triplet loss, and a cross-modal alignment loss; The optimized video infrared-visible light pedestrian re-identification model is used to complete the cross-modal pedestrian retrieval task.

2. The method for pedestrian re-identification based on video infrared-visible light according to claim 1, characterized in that: The step of inputting the cross-modal shared feature map into the spatial feature compensation network and supplementing the global features of the pedestrian according to the fine-grained features of each frame to determine the pedestrian features after spatial information compensation includes: Inputting each frame of the cross-modal shared feature map into the pyramid pooling layer of the spatial feature compensation network to obtain multi-scale features, where the multi-scale features are features of different granularities of the pedestrian image; The multi-scale features are subjected to transposed convolution and up-sampled to unify the size of the feature map to be consistent with the size of the cross-modal shared feature map; The unified multi-scale features are concatenated in the channel dimension, and the average pooling and fully connected layers are used to reduce the dimension of the concatenated feature map to obtain multi-granular features; The dimensions of the multi-granularity features and the features after average pooling of the cross-modal shared feature map are kept consistent, and the multi-granularity features are added to the features after average pooling to determine the pedestrian features after spatial information compensation in each frame.

3. The method for pedestrian re-identification based on video infrared-visible light according to claim 2, characterized in that: The step of aggregating the pedestrian features after the spatial information compensation at the frame level into sequence-level pedestrian features containing spatiotemporal information according to the temporal feature aggregation network includes: Inputting the cross-modal shared feature map of each frame into the temporal feature compensation network and performing a spatial convolution operation, wherein the width and height of the convolution kernel are consistent with the width and height of the cross-modal shared feature map; A fully connected layer is used after the spatial convolution operation to reduce the number of channels to 1 and output the importance score of the frame; Inputting the pedestrian features of each frame after spatial information compensation into the temporal feature aggregation network; The temporal feature aggregation network determines the sequence-level pedestrian features containing spatiotemporal information in each frame according to the pedestrian features after spatial information compensation in each frame and the corresponding importance score of the frame.

4. The method for pedestrian re-identification based on video infrared-visible light according to claim 3, characterized in that: The temporal feature aggregation network determines the sequence-level pedestrian features containing spatiotemporal information of each frame according to the pedestrian features of each frame after spatial information compensation and the importance score of the corresponding frame, including: according to Determine the sequence-level pedestrian features containing spatiotemporal information in each frame, where f seq represents the sequence-level pedestrian features containing spatiotemporal information, T is the number of image frames in a sequence, and A t is the importance score of the t-th frame, t∈[1,T], is the pedestrian feature after spatial information compensation in the tth frame.

5. The method for pedestrian re-identification based on video infrared-visible light according to claim 4, characterized in that: The end-to-end training is performed to optimize the video infrared-visible light pedestrian re-identification model by combining a cross entropy loss function, a triplet loss, and a cross-modal alignment loss, specifically including: According to L=L id +L triplet +L ma Determine the total loss function; The video infrared-visible light pedestrian re-identification model is optimized according to the total loss function, where L is the total loss function, L id is the cross entropy loss function, L triplet is the triplet loss, L ma is the cross-modal alignment loss.

6. The method for pedestrian re-identification based on video infrared-visible light according to claim 5, characterized in that: The cross entropy loss function, triplet loss, and cross-modal alignment loss are determined according to the sequence-level pedestrian features containing spatiotemporal information.

7. The method for pedestrian re-identification based on video infrared-visible light according to claim 6, characterized in that: Determining the cross entropy loss function, triplet loss, and cross-modal alignment loss according to the sequence-level pedestrian features containing spatiotemporal information specifically includes: according to Determine the cross-modal alignment loss, where P represents the number of pedestrian identities, N represents the number of video sequences for each pedestrian, and They represent the visible light feature center and infrared feature center of the i-th pedestrian, respectively, and C i represents the mean of the visible light feature center and infrared feature center of the i-th pedestrian, and denote the jth visible light sequence-level feature of the i-th pedestrian and the jth infrared sequence-level feature of the i-th pedestrian, respectively. m1 and m2 are used to indicate the visible light and infrared modalities. v denotes the visible light modality, r denotes the infrared modality. α is a hyperparameter used to control the minimum distance between the feature center of the positive sample and the feature center of the negative sample. Represents the feature center of pedestrians with different identities from the i-th pedestrian.

8. A video infrared-visible light pedestrian re-identification device, characterized in that: The device includes: A preprocessing module, used for preprocessing the acquired video infrared-visible light pedestrian image sequence; A model building module, used to build a video infrared-visible light pedestrian re-identification model, the model includes a weight-sharing dual-stream network, a spatial feature compensation network, and a temporal feature aggregation network; A cross-modal sharing module is used to map the pre-processed video infrared-visible pedestrian image sequence into the same feature space using a two-stream network with weight sharing to obtain a cross-modal shared feature map; A spatial feature compensation module, used to input the cross-modal shared feature map into the spatial feature compensation network, and to supplement the global features of the pedestrian according to the fine-grained features of each frame to determine the pedestrian features after spatial information compensation; A temporal feature aggregation module, configured to aggregate the pedestrian features after the spatial information compensation at the frame level into sequence-level pedestrian features containing temporal and spatial information according to the temporal feature aggregation network; A model training optimization module, for training in an end-to-end manner, combining a cross entropy loss function, a triplet loss, and a cross-modal alignment loss to optimize the video infrared-visible light pedestrian re-identification model; The recognition module is used to apply the optimized video infrared-visible light pedestrian re-identification model to complete the cross-modal pedestrian retrieval task.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, the processor is caused to perform the steps of the method according to any one of claims 1 to 7.

Citation Information

Cited By

  • Intelligent security video monitoring system for preventing animal attack behaviors

    CN120236243A