An occluded pedestrian re-identification method based on occlusion perception and feature restoration

Abstract

The application relates to the technical field of computer vision, and discloses an occluded pedestrian re-identification method based on occlusion perception and feature restoration, which comprises the following steps: obtaining a test data set containing a retrieval pedestrian image, inputting the retrieval pedestrian image into a feature extraction model, and obtaining N local features of the retrieval pedestrian image; inputting the N local features into a visibility perception model to obtain local visibility scores of the N local features; obtaining feature distances between the retrieval pedestrian image and searched pedestrian images in a gallery according to the local visibility scores; obtaining visible local features of K searched pedestrian images with the smallest feature distances, and completing the occluded local features of the retrieval pedestrian image; and re-searching similar pedestrian images according to the local features and the local visibility scores of the completed retrieval pedestrian image. The application realizes accurate occlusion object perception of occluded pedestrian images under different occlusion scenes and more robust pedestrian re-identification effects.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, and in particular to a method for occluded pedestrian re-identification based on occlusion perception and feature restoration. BACKGROUND

[0002] Person re-identification aims to retrieve the same person through non-overlapping cameras. With the progress of deep learning, person re-identification methods have achieved remarkable performance and have been widely applied to security camera systems. However, in actual scenarios, target pedestrians are often occluded by other pedestrians or static obstacles such as cars, walls, and trees. The interference of people-occluding-people and object-occluding-people and the loss of information of the occluded pedestrians result in a decrease in the recognition effect of the pedestrian re-identification method.

[0003] Existing occluded pedestrian re-identification methods use auxiliary modules such as pose estimation, human parsing, and other offline models to determine the local information of the occluded and non-occluded pedestrians, and to achieve occlusion feature elimination and pedestrian feature alignment. The introduction of auxiliary models increases the additional computational cost, and such methods have the following weaknesses:

[0004] 1) It is ineffective for the occlusion scene of the pedestrian-occluded-pedestrian. Because, unlike object-occluded-pedestrians, the occluded pedestrian and the occluded pedestrian share the same semantic information in the existing occluded pedestrian re-identification method, resulting in the failure of common auxiliary models in the occlusion scene of the pedestrian-occluded-pedestrian, which focuses more on the occluded person than the target person being occluded.

[0005] 2) It is ineffective to use only the non-occluded features of the pedestrian for retrieval. Existing occluded pedestrian re-identification methods usually remove the occluded part of the identified pedestrian information, but the non-occluded pedestrian features may be non-discriminative features or do not contain common local features with the positive sample. In these cases, the similarity of the negative sample pair will be greater than that of the positive sample pair, resulting in retrieval errors. SUMMARY

[0006] Therefore, the present application aims to overcome the problems of existing occluded pedestrian re-identification methods that cannot accurately perceive the occluded pedestrian image in the pedestrian-occluded-pedestrian scene and ineffective retrieval, and to provide an occluded pedestrian re-identification method based on occlusion perception and feature restoration.

[0007] To solve the above problems, the present application provides an occluded pedestrian re-identification method based on occlusion perception and feature restoration, comprising:

[0008] obtaining a test data set containing a retrieval pedestrian image, and inputting the retrieval pedestrian image into a pre-trained feature extraction model to obtain N local features of the retrieval pedestrian image;

[0009] inputting the N local features into a visibility-aware model pre-trained to obtain local visibility scores of the N local features;

[0010] According to the local visibility scores, a feature distance between the retrieved pedestrian image and the searched pedestrian images in the gallery is obtained.

[0011] Obtain the visible local features of the K searched pedestrian images with the smallest feature distance, and complete the occluded local features of the retrieved pedestrian image.

[0012] According to the local features and local visibility scores of the completed retrieved pedestrian image, re-search for similar pedestrian images.

[0013] Optionally, the occluded pedestrian re-identification method based on occlusion awareness and feature restoration further comprises:

[0014] Obtain a complete pedestrian image dataset; the complete pedestrian image dataset contains a plurality of unoccluded pedestrian images;

[0015] According to the complete pedestrian image dataset, simulate occluded pedestrian images under different occlusion scenarios, and generate local occlusion labels of the occluded pedestrian images.

[0016] According to the complete pedestrian image dataset and the simulated occluded pedestrian images, train a feature extraction model; the feature extraction model is used to obtain global features and local features of pedestrian images.

[0017] According to the local features and the local occlusion labels corresponding to the local features, train a visibility-aware model; the visibility-aware model is used to obtain local visibility scores of pedestrian images.

[0018] Optionally, the method of simulating occluded pedestrian images under different occlusion scenarios according to the complete pedestrian image dataset, and generating local occlusion labels of the occluded pedestrian images, comprises:

[0019] After duplicating and shuffling the complete pedestrian image dataset, an occluder dataset is obtained.

[0020] Label the pedestrian images in the complete pedestrian image dataset and the occluder dataset as target pedestrian images and occluder images respectively, and divide the target pedestrian images and the occluder images into N image blocks longitudinally.

[0021] Obtain the image blocks at the specified positions corresponding to each occlusion scenario from the occluder images, label them as occlusion blocks, and then cover the target pedestrian images to generate occluded pedestrian images and local occlusion labels corresponding to each image block.

[0022] Optionally, the occlusion scene includes a pedestrian-occluded pedestrian scene and an object-occluded pedestrian scene; and the covering the target pedestrian image after marking the image blocks corresponding to the specified positions of each occlusion scene as occlusion blocks comprises:

[0023] In the pedestrian-occluded pedestrian scene, the first N-m image blocks are obtained from the occluder image, and the mth to Nth image blocks of the target pedestrian image are covered;

[0024] In the object-occluded pedestrian scene, the image blocks in the upper and lower N / 2 parts of the occluder image are transposed, and a random number of occlusion blocks are selected to cover the image blocks at the corresponding positions in the target pedestrian image.

[0025] Optionally, the training of the feature extraction model according to the complete pedestrian image dataset and the simulated occluded pedestrian image comprises:

[0026] The complete pedestrian image dataset and the simulated occluded pedestrian image are merged to obtain a training dataset containing unoccluded pedestrian images and occluded pedestrian images;

[0027] The feature extraction model comprises a backbone network, a global average pooling branch and a local average pooling branch;

[0028] The unoccluded pedestrian images and the occluded pedestrian images are input into the feature extraction model, the feature map of the pedestrian image is obtained through the backbone network, the global feature is obtained by performing a pooling operation on the feature map through the global average pooling branch, and N local features corresponding to N image blocks are obtained by longitudinally dividing the feature map through the local average pooling branch;

[0029] The parameters of the feature extraction model are optimized according to the global feature, a classification loss function and a triplet loss function, to obtain a trained feature extraction model.

[0030] Optionally, the classification loss function is:

[0031] ,

[0032] In the above formula, is the classification loss function, is the number of images in each batch, is the input pedestrian image is divided into classes of probabilities;

[0033] The triplet loss function is expressed as:

[0034] ,

[0035] In the above formula, is a triplet loss function, is a global feature of the anchor image, is a global feature of the positive sample image, is a global feature of the negative sample image, is the Euclidean distance between the anchor image and the negative sample image, is the Euclidean distance between the anchor image and the positive sample image.

[0036] Optionally, the visibility-aware model is trained according to the local feature and the local occlusion label corresponding to the local feature, comprising:

[0037] A visibility-aware model corresponding to each of the image blocks is obtained; the visibility-aware model comprises three fully connected layers and an activation function layer;

[0038] The local feature corresponding to each of the image blocks is input into the visibility-aware model, and a local visibility score is output through the activation function layer; the local visibility score is:

[0039] ,

[0040] In the above formula, is the local visibility score of the i-th image block in the image predicted by the visibility-aware model, is a softmax activation function, is the weight of the i-th visibility-aware model, is the local feature of the i-th image block in the image; The parameters of the visibility-aware model are optimized using the local visibility score, the local occlusion label, and the local loss function to obtain a trained visibility-aware model.

[0041] Optionally, the local loss function is:

[0042]

[0043] ,

[0044] In the above formula, is a local loss function, is the number of batch training image samples of the visibility-aware model, is the local visibility score of the i-th image block in the i-th image determined according to the local occlusion label, is the local visibility score of the i-th image block in the i-th image predicted by the visibility-aware model, ​​​​​the local visibility score of the i-th image block in the search pedestrian image, the local visibility score of the i-th image block in the search pedestrian image, is a cross-entropy loss function.

[0045] Optionally, the obtaining the feature distance between the search pedestrian image and the searched pedestrian image in the gallery according to the local visibility score comprises:

[0046] obtaining, by the feature extraction model and the visibility perception model, N local visibility scores of each searched pedestrian image in the gallery;

[0047] inputting the N local visibility scores of the search pedestrian image and the searched pedestrian image in the gallery into a visibility score-based feature distance model to obtain the feature distance between the two pedestrian images; the visibility score-based feature distance model is:

[0048] ,

[0049] In the above formula, is the feature distance between the search pedestrian image and the searched pedestrian image , is the local visibility score of the i-th image block in the search pedestrian image , is the local visibility score of the i-th image block in the searched pedestrian image , is the Euclidean distance between the i-th image block in the search pedestrian image and the searched pedestrian image , is the Euclidean distance between the i-th image block in the search pedestrian image and the searched pedestrian image , is the number of image blocks.

[0050] Optionally, after the occluded local feature of the search pedestrian image is completed, the obtaining the K searched pedestrian images with the smallest feature distance further comprises:

[0051] obtaining the local feature and the local visibility score of the completed search pedestrian image; wherein the local feature of the completed search pedestrian image is:

[0052] ,

[0053] In the above formula, is the local feature of the completed search pedestrian image, , are the search pedestrian image and the i-th image block, respectively The first of the retrieved pedestrian images Local features of an image patch , Retrieving pedestrian images and the The first of the retrieved pedestrian images Local visibility score of each image patch The number of pedestrian images selected;

[0054] The local visibility score of the completed retrieved pedestrian image is:

[0055] ,

[0056] In the above formula, To complete the retrieved pedestrian image, the first... Local visibility score of an image patch.

[0057] The occlusion re-identification method based on occlusion perception and feature restoration provided in this invention extracts features from the retrieved pedestrian image using a feature extraction model, and uses a visibility perception model to perceive occlusion in the extracted local features. This improves the recognition accuracy of occluded pedestrian images and the prediction accuracy of local visibility scores, achieving the goal of accurately perceiving occlusions in occluded pedestrian images under different occlusion scenarios. Then, based on the local visibility score obtained from the visibility perception model, the feature distance between the retrieved pedestrian image and the retrieved image is obtained, which can effectively remove the influence of occlusion noise features and improve image retrieval accuracy. Finally, the image patch with high visibility in the retrieved image with the smallest feature distance is used to restore the corresponding image patch with low visibility in the retrieved pedestrian image, making the features of the retrieved pedestrian image richer and more discriminative, and achieving a more robust pedestrian re-identification effect. Attached Figure Description

[0058] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0059] Figure 1 The flowchart of an occlusion re-identification method based on occlusion perception and feature restoration provided by an embodiment of the present invention is shown.

[0060] Figure 2 A flowchart of an occlusion re-identification method based on occlusion perception and feature restoration, provided by another embodiment of the present invention, is shown. DETAILED DESCRIPTION

[0061] The exemplary embodiments will be described in detail herein with reference to the accompanying drawings. In the following description, unless otherwise indicated, like numbers in the different drawings represent similar or identical elements. The following exemplary embodiments described are not meant to represent all embodiments consistent with the present application. Rather, they are merely examples of apparatus and methods consistent with some aspects of the present application as detailed in the appended claims.

[0062] As shown in Figure 1 a flowchart of a method for occluded pedestrian re-identification based on occlusion perception and feature restoration is provided. The method comprises the following steps:

[0063] S10, a test data set containing a search pedestrian image is obtained, and the search pedestrian image is input into a pre-trained feature extraction model to obtain N local features of the search pedestrian image.

[0064] In step S10, the test data set contains one or more search pedestrian images, and the search pedestrian image is an occluded pedestrian image that needs to be identified.

[0065] The feature extraction model can be a widely used IDE (Integrated Development Environment) classification model. The input of the feature extraction model is a pedestrian image (containing an unoccluded pedestrian image and an occluded pedestrian image), and the output is a global feature of the pedestrian image and N local features. The global feature is used for training the feature extraction model, and the local features are used for training the visibility perception model.

[0066] Specifically, for each search pedestrian image of the test data set, the pre-trained feature extraction model is used to extract features of the search pedestrian image to obtain N local features of the search pedestrian image, and the N local features have a one-to-one correspondence with N image blocks divided longitudinally from the search pedestrian image.

[0067] It should be noted that the number N of local features and the number of image blocks can be set according to requirements.

[0068] S20, inputting the N local features into a pre-trained visibility perception model to obtain local visibility scores of the N local features.

[0069] In step S20, each image block corresponds to a pre-trained visibility perception model. As a preferred embodiment, the visibility perception model is a visibility perception model based on a fully connected neural network, which takes a local feature as input and outputs a local visibility score.

[0070] Specifically, for each local feature, an image block corresponding to the local feature is determined first, and then the local feature is occlusion-aware through the visibility-aware model corresponding to the image block to obtain the local visibility score of the local feature (or the image block corresponding to the local feature).

[0071] In step S30, the feature distance between the search pedestrian image and each searched pedestrian image in the gallery is obtained according to the local visibility score.

[0072] In step S30, the gallery is a pre-constructed image gallery containing a plurality of searched pedestrian images.

[0073] Specifically, for each search pedestrian image, the feature distance between the two pedestrian images is calculated by using the N local visibility scores of the search pedestrian image and each searched pedestrian image in the gallery. It can be understood that the smaller the feature distance is, the higher the similarity between the search pedestrian image and the searched pedestrian image is, and the higher the similarity is, the higher the probability of correct matching of the image is.

[0074] It should be noted that the N local visibility scores of the searched pedestrian image are obtained by the feature extraction model and the visibility-aware model.

[0075] As a preferred, the step S30 specifically comprises the following steps:

[0076] First, the N local visibility scores of each searched pedestrian image in the gallery are obtained by the feature extraction model and the visibility-aware model.

[0077] Then, the N local visibility scores of the search pedestrian image and the searched pedestrian image in the gallery are input into the feature distance model based on the visibility score to obtain the feature distance between the two pedestrian images.

[0078] In this embodiment, the searched pedestrian image in the gallery is input into the pre-trained feature extraction model to obtain the N local features of the searched pedestrian image, and the N local features are input into the pre-trained visibility-aware model to obtain the local visibility score of each image block in the searched pedestrian image. Then, the N local visibility scores of the search pedestrian image and the searched pedestrian image in the gallery are input into the feature distance model based on the visibility score, and the common unoccluded local features of the two pedestrian images are obtained by the feature distance model based on the visibility score to remove the noise features of the occlusion, thereby obtaining the feature distance between the two pedestrian images. The feature distance model based on the visibility score is:

[0079] ,

[0080] In the above formula, to retrieve the pedestrian image and the retrieved pedestrian image between the features, to retrieve the pedestrian image the local visibility score of the image block, to retrieve the pedestrian image the local visibility score of the image block, to retrieve the pedestrian image and the retrieved pedestrian image between the features, the Euclidean distance between the image blocks,

[0081] It can be understood that the embodiment uses the visibility score to realize the feature distance calculation of the common unoccluded pedestrian part of the pedestrian image pair, which can avoid the influence of noise features on the similarity value compared to the existing pedestrian re-identification method using Euclidean distance to calculate the similarity between two pedestrian images, and is beneficial to improve the pedestrian search accuracy.

[0082] S40, obtaining the visible local features of the K retrieved pedestrian images with the smallest feature distance, and completing the occluded local features of the retrieved pedestrian image.

[0083] In the embodiment, for each retrieved pedestrian image, after obtaining the feature distance between the retrieved pedestrian image and each retrieved pedestrian image through step S30, first, according to the feature distance, all the retrieved pedestrian images in the image library are sorted from small to large, and K retrieved pedestrian images with the smallest feature distance (i.e. the highest similarity) are obtained to generate a preliminary pedestrian retrieval result, and then the visible local features of the K retrieved pedestrian images are used to complete the occluded local features of the corresponding retrieved pedestrian image to obtain the completed retrieved pedestrian image.

[0084] It should be noted that the number K of the selected retrieved pedestrian images is set according to the needs, and the number K .

[0085] Further, after the step S40, the following steps are further included:

[0086] obtaining the local features and the local visibility score of the completed retrieved pedestrian image. Among them,

[0087] The local features of the completed retrieved pedestrian image are:

[0088] ,

[0089] In the above formula, To complete the local features of the retrieved pedestrian image, , Retrieving pedestrian images and the The first of the retrieved pedestrian images Local features of an image patch , Retrieving pedestrian images and the The first of the retrieved pedestrian images Local visibility score of each image patch The number of pedestrian images selected.

[0090] The local visibility score of the completed pedestrian image can be expressed as:

[0091] ,

[0092] In the above formula, To complete the retrieved pedestrian image, the first... Local visibility score of an image patch.

[0093] S50, based on the local features and local visibility scores of the completed pedestrian images, re-search for similar pedestrian images.

[0094] In this embodiment, for each completed pedestrian image, the feature distance between the completed pedestrian image and each retrieved pedestrian image in the image library is recalculated based on the local features and local visibility score of the completed pedestrian image. The pedestrian image with the highest similarity is then retrieved to obtain the final pedestrian retrieval result. Understandably, the final pedestrian retrieval result includes pedestrian images with the same identity as the retrieved pedestrian image.

[0095] In summary, the occlusion-based pedestrian re-identification method provided in this embodiment extracts features from the retrieved pedestrian image using a feature extraction model and uses a visibility perception model to perceive occlusion in the extracted local features. This improves the recognition accuracy of occluded pedestrian images and the prediction accuracy of local visibility scores, enabling accurate perception of occlusions in pedestrian images under different occlusion scenarios. Then, based on the local visibility score obtained from the visibility perception model, the feature distance between the retrieved pedestrian image and the retrieved image is obtained, effectively removing the influence of occlusion noise features and improving image retrieval accuracy. Finally, the image patch with high visibility in the retrieved image with the smallest feature distance is used to reconstruct the corresponding image patch with low visibility in the retrieved pedestrian image, making the features of the retrieved pedestrian image richer and more discriminative, achieving a more robust pedestrian re-identification effect.

[0096] As shown in Figure 2 the flowchart of another occluded pedestrian re-identification method based on occlusion perception and feature restoration provided by the embodiment of the present application, the method further comprises the following steps:

[0097] S60, obtaining a complete pedestrian image dataset; the complete pedestrian image dataset contains a plurality of unoccluded pedestrian images.

[0098] In the embodiment, the complete pedestrian image dataset can use the existing Occluded-DukeMTMC dataset, which contains 15618 images of 702 different pedestrians in the training dataset, of which 10% is occluded data and 90% is complete pedestrian data, which can be regarded as a complete pedestrian image dataset.

[0099] S70, simulating occluded pedestrian images under different occlusion scenarios according to the complete pedestrian image dataset, and generating local occlusion labels of the occluded pedestrian images.

[0100] Specifically, the complete pedestrian image dataset after copying and shuffling is used as an occlusion dataset, and the pedestrian images in the two datasets are divided into N image blocks, and then according to different occlusion scenarios, the image blocks at specific positions are selected as occlusion blocks to cover part of the original unoccluded pedestrian images, to generate simulated occluded pedestrian images and local occlusion labels corresponding to each image block.

[0101] S80, training a feature extraction model according to the complete pedestrian image dataset and the simulated occluded pedestrian images; the feature extraction model is used to obtain global features and local features of pedestrian images.

[0102] As a preferred, the generated occluded pedestrian images and the original complete pedestrian image dataset are merged to form a training dataset, and the feature extraction model is trained by steps S801 to S804 using the training dataset to obtain a trained feature extraction model.

[0103] S90, training a visibility perception model according to the local features and the local occlusion labels corresponding to the local features; the visibility perception model is used to obtain local visibility scores of pedestrian images.

[0104] As a preferred, the visibility perception model is first trained by steps S901 to S903 using the local features output by the feature extraction model and the local occlusion labels of the image blocks corresponding to the local features, to obtain a trained visibility perception model.

[0105] In summary, the occluded pedestrian re-identification method based on occlusion perception and feature restoration provided in the embodiment trains a feature extraction model by using occluded pedestrian images and complete pedestrian images in different occlusion scenarios, and trains a visibility perception model by using local features obtained by the feature extraction model and simulated local occlusion labels, so that the performance of the feature extraction model and the prediction accuracy of the visibility perception model can be improved, thereby enabling accurate perception of occlusions for occluded pedestrian data in different occlusion scenarios.

[0106] In an optional embodiment, the step S70 can include the following steps:

[0107] S701, after copying and disordering the complete pedestrian image dataset, an occlusion object dataset is obtained.

[0108] S702, the pedestrian images in the complete pedestrian image dataset and the occlusion object dataset are respectively marked as target pedestrian images and occlusion object images, and the target pedestrian images and the occlusion object images are divided into N image blocks in the vertical direction.

[0109] S703, the image blocks at the specified positions corresponding to each occlusion scenario are obtained from the occlusion object images, the target pedestrian images are covered after being marked as occlusion blocks, and occluded pedestrian images and local occlusion labels corresponding to each image block are generated.

[0110] In the embodiment, the occlusion scenarios include pedestrian-occluded-pedestrian scenarios and object-occluded-pedestrian scenarios, at this time, in the step S703, the image blocks at the specified positions corresponding to each occlusion scenario are obtained from the occlusion object images, the target pedestrian images are covered after being marked as occlusion blocks, and the specific steps include the following steps:

[0111] In the pedestrian-occluded-pedestrian scenario, the first N-m image blocks are obtained from the occlusion object images, and the mth to Nth image blocks of the target pedestrian images are covered; in the object-occluded-pedestrian scenario, the image blocks in the upper and lower N / 2 parts of the occlusion object images are transposed, and then a random number of occlusion blocks are selected to cover the image blocks at the corresponding positions in the target pedestrian images.

[0112] It can be understood that the occluded pedestrian image simulation method of the embodiment is based on the spatial relationship of the observed pedestrian occluding the pedestrian and the semantic relationship of the object occluding the pedestrian. The spatial relationship of the pedestrian occluding the pedestrian refers to that in the scene of the pedestrian occluding the pedestrian, the pedestrian as the occlusion data is lower than the occluded pedestrian in the image space. The semantic relationship of the object occluding the pedestrian refers to that in the scene of the object occluding the pedestrian, the semantic information of the object as the occlusion data is context-independent of the body semantic information of the pedestrian. Therefore, in simulating the occluded pedestrian image in the scene of the pedestrian occluding the pedestrian, the first to the Nth image blocks of the target pedestrian image are covered by the first N-m image blocks of the pedestrian image as the occlusion data (i.e. the occlusion object image), which guarantees the occlusion spatial relationship in the scene of the pedestrian occluding the pedestrian. In simulating the occluded pedestrian image in the scene of the object occluding the pedestrian, the image blocks of the upper and lower N / 2 parts of the pedestrian image as the occlusion data (i.e. the occlusion object image) are transposed, a random image block is selected as an occlusion block to cover the corresponding position of the target pedestrian image, and the transposed image blocks are inserted after the target pedestrian image, which is context-independent of the semantic information of the target pedestrian image, and ensures the semantic information in the scene of the object occluding the pedestrian. The above occlusion simulation process covers the image blocks at specific positions in the target pedestrian image to simulate the occluded pedestrian image. The covered image blocks in the target pedestrian image are the occluded part, and the occlusion simulation process also generates the local occlusion label of the corresponding image block, which is used to train the visibility perception model.

[0113] In summary, the occluded pedestrian re-identification method based on occlusion perception and feature restoration provided by the embodiment performs image data augmentation based on the spatial relationship and semantic relationship of the occlusion data, increases the diversity of the occluded pedestrian image, can promote the feature extraction model to learn the occlusion relationship in different occlusion scenes, and pay more attention to the target pedestrian. Meanwhile, the local occlusion label generated by the image data strategy can promote the visibility perception model to accurately predict the visibility score in different occlusion scenes.

[0114] In an optional embodiment, the step S80 can include the following steps:

[0115] S801, merging the complete pedestrian image dataset and the simulated occluded pedestrian image to obtain a training dataset containing unoccluded pedestrian images and occluded pedestrian images.

[0116] S802, obtaining a feature extraction model, wherein the feature extraction model comprises a backbone network, a global average pooling branch and a local average pooling branch.

[0117] S803, input the unoccluded pedestrian image and the occluded pedestrian image into a feature extraction model, obtain a feature map of the pedestrian image through a backbone network, perform a pooling operation on the feature map through a global average pooling branch to obtain a global feature, and perform longitudinal division on the feature map through a local average pooling branch to obtain N local features corresponding to N image blocks.

[0118] S804, optimize parameters of the feature extraction model according to the global feature, the classification loss function and the triplet loss function, and obtain a trained feature extraction model.

[0119] In this embodiment, after obtaining the training data set containing the unoccluded pedestrian image and the occluded pedestrian image, pedestrian images are obtained from the training data set according to a preset batch training size, and batch training is performed on the feature extraction model. In the batch training process, a feature map of the pedestrian image is extracted through the backbone network of the feature extraction model, the extracted feature map is subjected to the global average pooling branch to obtain the global feature of the pedestrian image, and the extracted feature map is subjected to the local average pooling branch to be divided into N local features in the longitudinal direction. The global feature of the pedestrian image is input into the classification loss function and the triplet loss function to constrain the training optimization of the feature extraction model. The classification loss function can be represented as:

[0120]

[0121] In the above formula, is the classification loss function, is the number of images in each batch training, is the input pedestrian image is divided into classes of probabilities;

[0122] The triplet loss function can be represented as:

[0123]

[0124] In the above formula, is the triplet loss function, is the global feature of the anchor image, is the global feature of the positive sample image, is the global feature of the negative sample image, is the Euclidean distance between the anchor image and the negative sample image, is the Euclidean distance between the anchor image and the positive sample image. The anchor image is the image of the target pedestrian, the positive sample image is all other images of the same person as the target pedestrian, and the negative sample image is all other images of different persons from the target pedestrian, i.e., the remaining images in each batch training image except the positive sample image. ​​

[0125] Further, the backbone network of the feature extraction model is a widely used computer vision deep network.

[0126] It can be understood that the embodiment can improve the performance of the feature extraction model by constraining the training optimization of the feature extraction model through the global feature, the classification loss function and the triplet loss function.

[0127] In an optional embodiment, the step S90 can include the following steps:

[0128] S901, obtaining a visibility perception model corresponding to each of the image blocks; the visibility perception model includes three fully connected layers and an activation function layer.

[0129] S902, inputting the local feature corresponding to each of the image blocks into the visibility perception model, and outputting a local visibility score through the activation function layer.

[0130] S903, optimizing the parameters of the visibility perception model by using the local visibility score, the local occlusion label and a local loss function, to obtain a trained visibility perception model.

[0131] In the embodiment, for the training process of each visibility perception model, the feature information of the local feature is further extracted through the fully connected layer of the visibility perception model, and the feature information is input into the activation function layer to obtain a local visibility score. The local visibility score can be expressed as:

[0132] ,

[0133] In the above formula, is the local visibility score of the i-th image block in the image predicted by the visibility perception model, is a softmax activation function, is the weight of the i-th visibility perception model, is the local feature of the i-th image block in the image. Then, the trained visibility perception model is obtained by using the local occlusion label generated in step S10, the local visibility score output by the activation function layer and the local loss function to constrain the training optimization of the visibility perception model. The local loss function is:

[0134] ,

[0135] ,

[0136] In the above formula, is the local loss function, ​​a number of batch training image samples for the visibility perception model, a local visibility score of an i-th image patch in an i-th image determined according to the local occlusion label, a local visibility score of an i-th image patch in an i-th image determined according to the local occlusion label, a local visibility score of an i-th image patch in an i-th image predicted by the visibility perception model, a local visibility score of an i-th image patch in an i-th image predicted by the visibility perception model, a local visibility score of an i-th image patch in an i-th image predicted by the visibility perception model, a local visibility score of an i-th image patch in an i-th image predicted by the visibility perception model, a cross-entropy loss function.

[0137] It should be noted that the optimization of the visibility perception model is irrelevant to the optimization of the feature extraction model, and the backpropagation gradient of the local loss is truncated at the local feature input in the process of optimizing the visibility perception model, that is, the process of optimizing the visibility perception model only learns the visibility score of the input local feature, and does not affect the extraction of the local feature by the feature extraction model.

[0138] It can be understood that the embodiment uses the local visibility score, the local occlusion label and the local loss function to constrain the training process of the visibility perception model, which can improve the prediction accuracy of the visibility perception model.

[0139] It should be noted that in the description of the present application, the terms "first", "second" and the like are only for the purpose of description and cannot be understood as indicating or implying relative importance. In addition, in the description of the present application, unless otherwise specified, the meaning of "a plurality of" is at least two.

[0140] The above is only a preferred embodiment of the present application, and is not intended to limit the present application in any form. Although the present application has been disclosed as above with a preferred embodiment, it is not intended to limit the present application, and any person skilled in the art can make some changes or modifications to the above disclosed technical content without departing from the scope of the technical solution of the present application, and any brief introduction, modification, equivalent change and modification of the above embodiment according to the technical essence of the present application are still within the scope of the technical solution of the present application.

Claims

1. A method for re-identifying occluded pedestrians based on occlusion perception and feature restoration, characterized in that, include: Obtain a test dataset containing images of retrieved pedestrians, and input the retrieved pedestrian images into a pre-trained feature extraction model to obtain N local features of the retrieved pedestrian images; The N local features are input into a pre-trained visibility perception model to obtain the local visibility scores of the N local features. Based on the local visibility score, the feature distance between the retrieved pedestrian image and the retrieved pedestrian image in the image library is obtained; Obtain the K visible local features of the retrieved pedestrian image with the smallest feature distance, and complete the occluded local features of the retrieved pedestrian image; Based on the local features and local visibility scores of the completed pedestrian images, similar pedestrian images are retrieved again. The step of obtaining the feature distance between the retrieved pedestrian image and the retrieved pedestrian image in the image library based on the local visibility score includes: The feature extraction model and the visibility perception model are used to obtain N local visibility scores for each retrieved pedestrian image in the image library. The N local visibility scores of the retrieved pedestrian image and the retrieved pedestrian image in the image library are input into a feature distance model based on visibility scores to obtain the feature distance between the two pedestrian images; the feature distance model based on visibility scores is: ， In the above formula, To retrieve pedestrian images Images of the pedestrians being retrieved The characteristic distance between them To retrieve pedestrian images The Middle Local visibility score of each image patch For the image of the pedestrian being retrieved The Middle Local visibility score of each image patch To retrieve pedestrian images Images of the pedestrians being retrieved The Middle Euclidean distance between image patches The number of image patches; After obtaining the K visible local features of the retrieved pedestrian image with the smallest feature distance and completing the occluded local features of the retrieved pedestrian image, the process further includes: Obtain the local features and local visibility score of the completed pedestrian image; wherein, the local features of the completed pedestrian image are: ， In the above formula, To complete the local features of the retrieved pedestrian image, , Retrieving pedestrian images and the The first of the retrieved pedestrian images Local features of an image patch , Retrieving pedestrian images and the The first of the retrieved pedestrian images Local visibility score of each image patch The number of pedestrian images selected; The local visibility score of the completed retrieved pedestrian image is: ， In the above formula, To complete the retrieved pedestrian image, the first... Local visibility score of an image patch.

2. The occlusion-based pedestrian re-identification method according to claim 1, characterized in that, Before obtaining the test dataset, the method also includes: Obtain a complete pedestrian image dataset; the complete pedestrian image dataset contains multiple unoccluded pedestrian images; Simulate occluded pedestrian images under different occlusion scenarios based on the complete pedestrian image dataset, and generate local occlusion labels for the occluded pedestrian images; A feature extraction model is trained based on the complete pedestrian image dataset and the simulated occluded pedestrian images; the feature extraction model is used to obtain global and local features of pedestrian images; A visibility perception model is trained based on the local features and the corresponding local occlusion labels; the visibility perception model is used to obtain the local visibility score of pedestrian images.

3. The occlusion-based pedestrian re-identification method according to claim 2, characterized in that, The step of simulating occluded pedestrian images under different occlusion scenarios based on the complete pedestrian image dataset and generating local occlusion labels for the occluded pedestrian images includes: After copying and scrambling the complete pedestrian image dataset, an occlusion dataset is obtained; The pedestrian images in the complete pedestrian image dataset and the occlusion dataset are labeled as target pedestrian images and occlusion images, respectively, and the target pedestrian images and the occlusion images are vertically divided into N image blocks; The image block at a specified location corresponding to each occlusion scenario is obtained from the occlusion image, and after being marked as an occlusion block, it is overlaid on the target pedestrian image to generate an occluded pedestrian image and a local occlusion label corresponding to each image block.

4. The occlusion-based pedestrian re-identification method according to claim 3, characterized in that, The occlusion scenarios include pedestrian occlusion scenarios and object occlusion scenarios; The step of obtaining the image patch at a specified location corresponding to each occlusion scenario from the occlusion image, marking it as an occlusion patch, and then covering the target pedestrian image includes: In a scenario where a pedestrian occludes another pedestrian, the first Nm image blocks are obtained from the image of the occluder, and the m-th to N-th image blocks of the target pedestrian image are covered. In a scenario where an object occludes a pedestrian, after transposing the image blocks in the upper and lower N / 2 portions of the occluding object image, a random number of occluding blocks are selected to cover the corresponding image blocks in the target pedestrian image.

5. The occlusion-based pedestrian re-identification method according to claim 2, characterized in that, The step of training a feature extraction model based on the complete pedestrian image dataset and the simulated occluded pedestrian image includes: The complete pedestrian image dataset and the simulated occluded pedestrian images are merged to obtain a training dataset containing both unoccluded and occluded pedestrian images; Obtain a feature extraction model, which includes a backbone network, a global average pooling branch, and a local average pooling branch; The unoccluded pedestrian image and the occluded pedestrian image are input into the feature extraction model. The feature map of the pedestrian image is obtained through the backbone network. The feature map is pooled through the global average pooling branch to obtain global features. The feature map is vertically divided through the local average pooling branch to obtain N local features corresponding to N image blocks. The parameters of the feature extraction model are optimized based on the global features, classification loss function, and triplet loss function to obtain a trained feature extraction model.

6. The occlusion-based pedestrian re-identification method according to claim 5, characterized in that, The classification loss function is: ， In the above formula, For classification loss function, The number of images trained in each batch. For input pedestrian images Divided into The probability of a class; The triplet loss function is expressed as follows: ， In the above formula, The triplet loss function, For global features of the anchor point image, These are global features of positive sample images. These are global features of negative sample images. The Euclidean distance between the anchor image and the negative sample image is given. The distance between the anchor image and the positive sample image is the Euclidean distance.

7. The occlusion-based pedestrian re-identification method according to claim 3, characterized in that, The step of training a visibility perception model based on the local features and the corresponding local occlusion labels includes: Obtain a visibility-aware model corresponding to each of the image patches; the visibility-aware model comprises three fully connected layers and an activation function layer; The local features corresponding to each image patch are input into the visibility perception model, and the local visibility score is output through the activation function layer; the local visibility score is: ， In the above formula, The image predicted by the visibility perception model is the first... Local visibility score of each image patch The softmax activation function is used. For the first Weights of a visibility perception model For the first in the image Local features of an image patch; The parameters of the visibility perception model are optimized using the local visibility score, local occlusion label, and local loss function to obtain a trained visibility perception model.

8. The occlusion-based pedestrian re-identification method according to claim 7, characterized in that, The local loss function is: ， In the above formula, For local loss functions, The number of image samples used in the batch training of the visibility perception model. For the first determined based on the partial occlusion label The first image Local visibility score of each image patch The first prediction of the visibility perception model The first image Local visibility score of each image patch This is the cross-entropy loss function.

Images