Training method and re-identification method for clothes changing pedestrian re-identification model

Abstract

The invention belongs to the technical field of pedestrian re-identification, and discloses a training method and a re-identification method of a clothes-changing pedestrian re-identification model. The training method of the clothes changing pedestrian re-identification model comprises the following steps: S1, constructing a training data set of clothes changing pedestrian re-identification; s2, constructing a pedestrian target detection network, and carrying out two-dimensional target detection; s3, performing pedestrian attribute extraction and elimination, and preliminarily constructing a clothes changing pedestrian re-identification model; and S4, pedestrian attribute feature fusion and pedestrian re-identification are carried out, and a clothes changing pedestrian re-identification model is perfectly constructed. According to the method, pedestrian attribute information is extracted, then pedestrian clothes, clothes colors and other information are shielded by using masks, the interference and influence of the pedestrian clothes on pedestrian re-identification are eliminated, finally, pedestrian attribute feature information with clothes attributes removed is coded, feature fusion is performed on the pedestrian attribute feature information and feature maps of different levels, and the pedestrian re-identification accuracy is improved. Therefore, the re-identification of the clothes changing pedestrian in the monitoring scene is realized.

Description

Technical Field

[0001] The present invention belongs to the technical field of pedestrian re-identification, and in particular, relates to a training method for a clothing-changing pedestrian re-identification model and a re-identification method. Background Art

[0002] Person Re-identification (ReID) is a technology that uses computer vision technology to identify and match the same target person (pedestrian) within the field of view of different cameras.

[0003] Traditional pedestrian re-identification methods are mainly based on the assumption that the tracked pedestrians move within a limited area and a certain time range, and that pedestrians will not change their clothes to produce large changes in appearance features. However, in actual application scenarios, the tracked pedestrians usually undergo large changes in geographical location, and after a certain period of time, their appearance such as clothing also undergoes large changes. Therefore, pedestrian re-identification when pedestrians change clothes has important research and application value. For the task of pedestrian re-identification when changing clothes, the key is to identify the features of pedestrians that are not related to clothing. At present, the main method is to extract the multimodal biometric features of pedestrians, such as contour information, 3D shape features of pedestrians, posture features of pedestrians, gait features, etc., so as to reduce the interference of clothing color. However, these methods usually require the use of additional complex models to extract multimodal biometric features, which requires high manpower and material resources and is difficult to be widely used in current monitoring systems.

[0004] Therefore, there is an urgent need to provide a training method and a re-identification method for a clothing-changing pedestrian re-identification model with higher accuracy and efficiency to solve the above problems.

[0005] In view of this, this application is hereby filed. Summary of the invention

[0006] The technical problem to be solved by the present invention is to overcome the deficiency in the prior art that pedestrian re-identification of people changing clothes cannot reduce the identification interference caused by unimportant features on an economical and efficient basis. The purpose is to provide a training method and a re-identification method for a pedestrian re-identification model of people changing clothes.

[0007] In order to solve the above technical problems, the basic concept of the technical solution adopted by the present invention is: a training method for a clothing-changing pedestrian re-identification model, comprising the steps of:

[0008] S1. Build a training dataset for pedestrian re-identification in surveillance scenarios, and perform data collection and data labeling;

[0009] S2. Build a pedestrian target detection network, take video image data as input, and perform two-dimensional target detection;

[0010] S3, extract and eliminate pedestrian attributes, and preliminarily build a pedestrian re-identification model for changing clothes;

[0011] S4. Carry out pedestrian attribute feature fusion and pedestrian re-identification, and improve the construction of clothing-changing pedestrian re-identification model.

[0012] According to an embodiment of the present invention, step S1 comprises:

[0013] S1.1. Using the open source clothing-changing pedestrian dataset, we collected video image data of the same target person wearing different clothing from different camera perspectives.

[0014] S1.2. Pedestrian ID labeling is performed on video image data taken from different camera angles and wearing different clothing.

[0015] According to an embodiment of the present invention, the data annotation includes pedestrian target detection and pedestrian ID annotation, and the pedestrian category target in the image is annotated using a rectangular box tool, and the rectangular box position P = (X, Y, W, H);

[0016] Among them, (X, Y) represents the coordinate position of the center point of the pedestrian target, and (W, H) represents the length and width of the pedestrian target rectangular box; the same pedestrian is marked with the same ID; for the marking of pedestrian clothing, the same set of clothing is marked with the same clothing ID.

[0017] According to one embodiment of the present invention, the function of the pedestrian attribute extraction and elimination part is to extract the attribute information of the pedestrian using the pedestrian attribute model, and then use a mask to shield the attribute information about the pedestrian's clothing in the pedestrian attribute information, thereby eliminating the features about the pedestrian's clothing in the pedestrian attributes.

[0018] According to an embodiment of the present invention, the pedestrian attributes include: gender, age, clothing, and accessories.

[0019] According to one embodiment of the present invention, the pedestrian target detection network includes: a feature extraction network part and a target detection network part.

[0020] According to one embodiment of the present invention, for the feature extraction part, a convolutional neural network is used, which takes the video image data as input and outputs the feature representation of the image.

[0021] According to an embodiment of the present invention, for the target detection network part, a multi-category detection method is applied to output the category and location information of the target;

[0022] The output is defined as:

[0023] O i =[ci ,x i ,y i ,w i ,h i ]

[0024] O i Denoted as the i-th target, c i Indicates the category of the extracted target, x i ,y i are the center point coordinates of the target rectangle, w i ,h i are the length and width of the target respectively.

[0025] After adopting the above technical solution, the present invention has the following beneficial effects compared with the prior art:

[0026] In the present invention, the pedestrian attribute information is firstly extracted by using a pedestrian attribute detection model, and then the information such as the pedestrian's clothing and clothing color is shielded by a mask, and other pedestrian attribute features are retained to eliminate the interference and influence of pedestrian clothing on pedestrian re-identification. Finally, the pedestrian attribute feature information without clothing attributes is encoded and feature fused with feature maps of different levels to achieve re-identification of pedestrians who change clothes in monitoring scenarios, which has a positive significance for improving the accuracy and efficiency of pedestrian re-identification.

[0027] The specific implementation modes of the present invention are further described in detail below in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] The accompanying drawings are part of the present invention and are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention, but do not constitute an improper limitation of the present invention. Obviously, the drawings described below are only some embodiments. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. In the accompanying drawings:

[0029] Figure 1 The present invention is a flowchart of the steps of a training method for a clothing-changing pedestrian re-identification model in an embodiment of the present invention.

[0030] It should be noted that these drawings and textual descriptions are not intended to limit the conceptual scope of the present invention in any way, but are intended to illustrate the concept of the present invention for those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0031] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. The following embodiments are used to illustrate the present invention but are not used to limit the scope of the present invention.

[0032] In the description of the present invention, it should be noted that the directions or positional relationships indicated by terms such as “upper”, “lower”, “front”, “back”, “left”, “right”, “vertical”, “inside” and “outside” are based on the directions or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific direction, be constructed and operated in a specific direction, and therefore cannot be understood as a limitation on the present invention.

[0033] In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0034] like Figure 1 As shown, the training method of a clothing-changing pedestrian re-identification model of the present invention comprises the steps of:

[0035] S1. Build a training dataset for pedestrian re-identification in surveillance scenarios, and perform data collection and data labeling.

[0036] The step S1 specifically includes:

[0037] S1.1. Use open source clothing-changing pedestrian datasets, such as LTCC, PRCC, Celeb-ReID, etc., to collect video image data of the same target person (pedestrian) wearing different clothes under different camera perspectives;

[0038] S1.2. Pedestrian IDs are annotated for the video image data of people wearing different clothes under different camera viewing angles.

[0039] Specifically, the different camera viewing angles represent different camera installation positions, such as parallel installation, top-down installation, etc.;

[0040] Specifically, for the data collection, by using multiple different cameras to collect data, each pedestrian wears at least two different sets of clothing and is captured by at least two cameras; and there are changes in lighting, viewing angle, pedestrian posture, etc.;

[0041] Specifically, for the data annotation, pedestrian target detection and pedestrian ID annotation are first performed, and the pedestrian category target in the image is annotated using a rectangular box tool, and the rectangular box position P = (X, Y, W, H), where (X, Y) represents the coordinate position of the center point of the pedestrian target, and (W, H) represents the length and width of the pedestrian target rectangular box. The same pedestrian is marked with the same ID; for the annotation of pedestrian clothing, the same set of clothing is marked with the same clothing ID.

[0042] S2. Build a pedestrian target detection network, take video image data as input, and perform two-dimensional target detection.

[0043] By applying two-dimensional object detection, it is possible to locate and identify the target in the two-dimensional image, and determine its category information and position information.

[0044] Specifically, the pedestrian target detection network includes a feature extraction network part and a target detection network part;

[0045] 1) For the feature extraction part, a commonly used backbone network is used, including but not limited to convolutional neural networks such as ResNet, MobileNet, and VGG, with the input being an image (the video image data) and the output being a feature representation of the image;

[0046] 2) For the target detection network part, including but not limited to using commonly used multi-category detection methods such as YOLO, SSD, CenterNet, etc., the output is the category and location information of the target;

[0047] The output is defined as:

[0048] O i =[c i ,x i ,y i ,w i ,h i ]

[0049] O i Denoted as the i-th target, c i Indicates the category of the extracted target, x i ,y i are the center point coordinates of the target rectangular box, w i ,h i are the length and width of the target respectively;

[0050] Specifically, according to the target detection results, the coordinate position of the pedestrian two-dimensional target detection box is obtained, the pedestrian image is extracted from the image, and the width and height of the extracted image are calculated. The longer side is taken and the size is fixed to 256. Then the shorter side (relative to the longer side) is reduced in the same proportion. Finally, the size of the shorter side is expanded to 128, and the insufficient part is filled with pixel value 0. Finally, the size of the pedestrian (target person) image is unified to 256*128.

[0051] S3, build the first part of the clothing-changing pedestrian re-identification model, and perform pedestrian attribute extraction and elimination.

[0052] Specifically, the clothing-changing pedestrian re-identification model includes two parts, namely, pedestrian attribute extraction and elimination part, pedestrian attribute feature fusion and pedestrian re-identification;

[0053] Specifically, the pedestrian attribute extraction and elimination part uses a pedestrian attribute (detection) model to extract pedestrian attribute information, and then uses a mask to shield the attribute information about pedestrian clothing in the pedestrian attribute information, thereby eliminating the features about pedestrian clothing in the pedestrian attributes;

[0054] Specifically, the pedestrian attributes mainly include gender, age, clothing, accessories and other attributes;

[0055] Specifically, the pedestrian attribute recognition network includes two modules: a backbone network and a classification network;

[0056] Specifically, the backbone network is used to extract features of pedestrian images, and the input image size is 256*128*3;

[0057] 256*128 represent the height and width of the image respectively; 3 represents the number of channels, indicating an RGB three-channel image;

[0058] Specifically, the backbone network mainly adopts a convolution combination method such as convolution layer-normalization layer-activation function layer to perform superposition operations of multiple convolution combination layers, including but not limited to using different backbone networks such as ResNet50 and MobileNetV3, and the output is (256 / R)*(128 / R)*C, where R represents the reduction ratio of the image during the feature extraction process, such as R=16, and C represents the number of channels of high-dimensional features obtained through the feature extraction network, such as C=512 or 1024;

[0059] Specifically, for the pedestrian attribute classification network, the fully connected layer is used to perform the classification results of pedestrian attribute features;

[0060] Specifically, for the clothing attribute elimination part, the input pedestrian image x i , output the attribute classification result of the pedestrian Where N represents the number of attribute categories. For each classification result, the result of converting it into a 0-1 binary vector is expressed as The clothing information is masked by setting the clothing attributes therein to 0, thereby eliminating the clothing attributes.

[0061] S4. Construct the second part of the clothing-changing pedestrian re-identification model to perform pedestrian attribute feature fusion and pedestrian re-identification.

[0062] Specifically, the pedestrian attribute feature fusion is to encode the pedestrian attribute information extracted in the above step S3 and fuse it with the pedestrian (feature) image for re-identification of pedestrians who change clothes;

[0063] Specifically, the input includes pedestrian images x i ∈R H×W×C , and the masked pedestrian attribute feature information The network model uses a visual transformer-based network structure for feature extraction and fusion;

[0064] Specifically, first, the pedestrian image x i Converted into a sequence of fixed-size image blocks, represented by N = HxW / [ 2 , where P represents the size of each fixed-size image block, and then it is mapped to a one-dimensional feature representation using a linear mapping method, expressed as Then, the pedestrian category information is added to the one-dimensional feature, expressed as

[0065] Specifically, the visual transformer-based network model includes two stages. The first stage input is the one-dimensional feature representation One is used to model the dependencies between each image block. In addition, Used to extract the underlying feature representation of the global image in the first stage;

[0066] Specifically, for the pedestrian attribute feature information after masking The same linear mapping is performed to map it into a one-dimensional feature representation, which is expressed as and The dimensions are the same, and then an additional attribute information is added to the one-dimensional feature, expressed as

[0067] Specifically, the image sequence and the attribute sequence are concatenated and expressed as As input, it is sent to the second stage of the transformer network for model training. By connecting the image features and attribute description features and training them together, the relationship between the image and the attribute description that eliminates clothing information is learned, and the extraction As the fusion feature of image and attribute description information, the global representation extracted in the first stage is combined The one-dimensional convolutional layer is used to achieve the final re-identification of pedestrians who change clothes;

[0068] Specifically, for the transformer network of the first stage, it is composed of multiple layers of transformer network layers, and n layers can be set (for example, n is set to 5), and the feature dimension of each layer is 512 dimensions;

[0069] Each transformer layer is divided into two sub-layers. The first sub-layer is a multi-head attention module, where the multi-head can be set to 8-head attention; the second sub-layer is a standard feedforward neural network; for each transformer layer, the output of each multi-head attention is residually connected with the input of the current encoder layer, and the output result is sent to the normalization layer, which adopts the layer normalization layer; then the normalized result is sent to the feedforward neural network layer, and then the residual connection calculation and the layer normalization calculation are performed again, and the result is sent to the next transformer layer for repetition;

[0070] Specifically, the transformer network in the second stage is composed of multiple layers of transformer network layers, fully connected layers, and layer normalization layers, which are used to fully extract image and attribute information features.

[0071] S5. Define the loss function for model training.

[0072] Specifically, the loss function includes three aspects:

[0073] The first aspect is the classification and detection box regression loss function of pedestrian target detection, L OD represents the two-dimensional target detection classification loss function, including target classification and position regression. OD =β1L class +β2L bbox ;

[0074] L class Represents the classification loss function of the target, including but not limited to using loss functions such as cross entropy, L bbox is a regression loss function of the target detection box, including but not limited to using a smooth L1 loss function;

[0075] The first aspect is the pedestrian attribute classification loss function. For the pedestrian attribute classification loss function LCLS , using the standard softmax classification loss function;

[0076] The third aspect is the pedestrian re-identification loss function, which includes two aspects:

[0077] The first is the classification loss function, which uses the pedestrian's ID as a training label to determine whether two pedestrians are the same person. The cross entropy loss function is used for calculation, which is expressed as:

[0078]

[0079] in, Represents the ID category of the predicted output, and L represents the true value of the annotation;

[0080] The second is the metric learning loss function, which aims to make the distance between two positive samples (i.e. the same person in different frames) as small as possible and the distance between negative samples as large as possible (different pedestrians); the triplet loss function is used;

[0081] The triplet consists of Anchor (A), Negative (N), and Positive (P), where the base point is A, the positive match is P, and the negative match is N. The purpose of the triplet loss function is to make the distance between features of the same class as small as possible, and the distance between features of different classes as large as possible, which is expressed as:

[0082]

[0083] Represents the distance between two identical pedestrian features, the goal is to continuously reduce; It represents the distance between two different pedestrian features, with the purpose of continuous prompting; α is a constant parameter.

[0084] By applying a training method and a re-identification method for a clothing-changing pedestrian re-identification model provided in the present application, the attribute information of the pedestrian is extracted, and then the information such as the pedestrian's clothing and clothing color is shielded using a mask, and other pedestrian attribute features are retained, and the interference and influence of the pedestrian's clothing on the pedestrian re-identification are eliminated. Finally, the pedestrian attribute feature information without the clothing attributes is encoded, and feature fusion is performed with feature maps of different levels, so as to realize the re-identification of pedestrians who change clothes in monitoring scenarios, which has a positive significance for improving the accuracy and efficiency of the re-identification of pedestrians who change clothes.

[0085] The above description is only a preferred embodiment of the present invention, and does not limit the present invention in any form. Although the present invention has been disclosed as a preferred embodiment as above, it is not used to limit the present invention. Any technician familiar with this patent can make some changes or modifications to equivalent embodiments of equivalent changes by using the technical content suggested above without departing from the scope of the technical solution of the present invention. The implementation scheme in the above embodiment can also be further combined or replaced. However, any simple modification, equivalent change and modification made to the above embodiment based on the technical essence of the present invention without departing from the content of the technical solution of the present invention still falls within the scope of the solution of the present invention.

Claims

1. A training method for a clothing-changing pedestrian re-identification model, characterized in that: Includes steps: S1. Build a training dataset for pedestrian re-identification in surveillance scenarios, and perform data collection and data labeling; S2. Build a pedestrian target detection network, take video image data as input, and perform two-dimensional target detection; S3, extract and eliminate pedestrian attributes, and preliminarily build a pedestrian re-identification model for changing clothes; S4. Carry out pedestrian attribute feature fusion and pedestrian re-identification, and improve the construction of clothing-changing pedestrian re-identification model.

2. The training method of a clothing-changing pedestrian re-identification model according to claim 1, characterized in that: The step S1 comprises: S1.

1. Using the open source clothing-changing pedestrian dataset, we collected video image data of the same target person wearing different clothing from different camera perspectives. S1.

2. Pedestrian ID labeling is performed on video image data taken from different camera angles and wearing different clothing.

3. The training method of a clothing-changing pedestrian re-identification model according to claim 2 is characterized in that: The data annotation includes pedestrian target detection and pedestrian ID annotation, and the pedestrian category target in the image is annotated using a rectangular box tool, and the rectangular box position P = (X, Y, W, H); Among them, (X, Y) represents the coordinate position of the center point of the pedestrian target, and (W, H) represents the length and width of the pedestrian target rectangular box; the same pedestrian is marked with the same ID; for the marking of pedestrian clothing, the same set of clothing is marked with the same clothing ID.

4. The training method of a clothing-changing pedestrian re-identification model according to claim 1, characterized in that: The function of the pedestrian attribute extraction and elimination part is to extract the attribute information of the pedestrian using the pedestrian attribute model, and then use a mask to shield the attribute information about the pedestrian's clothing in the pedestrian attribute information, thereby eliminating the features about the pedestrian's clothing in the pedestrian attributes.

5. The training method of a clothing-changing pedestrian re-identification model according to claim 4 is characterized in that: The pedestrian attributes include: gender, age, clothing, and accessories.

6. The training method of a clothing-changing pedestrian re-identification model according to claim 1, characterized in that: The pedestrian target detection network includes: a feature extraction network part and a target detection network part.

7. The training method of a clothing-changing pedestrian re-identification model according to claim 6, characterized in that: For the feature extraction part, a convolutional neural network is used, which takes the video image data as input and outputs the feature representation of the image.

8. The method for training a clothing-changing pedestrian re-identification model according to claim 6, characterized in that: For the target detection network part, a multi-category detection method is applied to output the category and location information of the target; The output is defined as: O i =[c i ,x i ,y i ,w i ,h i ] O i Denoted as the i-th target, c i Indicates the category of the extracted target, x i ,y i are the center point coordinates of the target rectangle, w i ,h i are the length and width of the target respectively.

9. A method for re-identifying pedestrians who change clothes, characterized in that: A method for training a clothing-changing pedestrian re-identification model according to any one of claims 1 to 8 is applied, Or, a clothes-changing pedestrian re-identification model generated by using a training method for a clothes-changing pedestrian re-identification model according to any one of claims 1 to 8.

Images