Training method of pedestrian re-identification model supporting multiple scenes and re-identification method

Abstract

The invention belongs to the technical field of pedestrian re-identification, and discloses a training method of a multi-scene pedestrian re-identification model and a re-identification method. The training method of the pedestrian re-identification model supporting multiple scenes comprises the steps of constructing a training data set of pedestrian re-identification under multiple scenes; constructing a pedestrian target detection network, and carrying out two-dimensional target detection; multi-scene prompts based on prompts are introduced for joint training, and a multi-scene re-recognition knowledge base is learned; carrying out distillation prompted by a specific scene; and a loss function is defined, model training is carried out, and a pedestrian re-identification model supporting multiple scenes is generated. Through application of the training method of the pedestrian re-recognition model supporting multiple scenes, the pedestrian re-recognition model supporting multiple scenes is generated, manual switching to other types of pedestrian re-recognition models according to scene images is not needed, the pedestrian re-recognition models can be deployed in application equipment more conveniently, and the user experience is improved. And the method has positive significance in improving the pedestrian re-identification efficiency and quality.

Description

Technical Field

[0001] The present invention belongs to the technical field of pedestrian re-identification, and in particular, relates to a training method and a re-identification method of a pedestrian re-identification model supporting multiple scenarios. Background Art

[0002] Person Re-identification (ReID) is a technology that uses computer vision technology to identify and match the same target person (pedestrian) within the field of view of different cameras.

[0003] Traditional pedestrian re-identification methods are mainly based on the movement of the tracked target person in a limited area and a certain time range. Usually, the target person's appearance characteristics will not change significantly. However, in actual application scenarios, the tracked target person usually undergoes a large geographical change, and the appearance characteristics of the pedestrian also change significantly, such as pedestrians changing clothes, pedestrians being blocked, etc. In addition, in order to fully utilize the advantages of data in different modalities, pedestrian re-identification under cross-modal data is proposed, such as pedestrian re-identification under visible light and infrared light modal data. For these different scenarios, pedestrian re-identification methods based on pedestrian re-identification models in specific scenarios are studied, such as pedestrian re-identification when changing clothes, cross-modal data pedestrian re-identification, etc. Since there are few studies on the training methods for generating pedestrian re-identification models that support multi-scenario sharing and they are not mature, in actual applications, it is usually necessary to apply multiple different types of pedestrian re-identification models at the same time. Although each pedestrian re-identification model has good pedestrian re-identification effect in the corresponding specific scenarios, it is difficult to deploy it in actual application devices because it only supports pedestrian re-identification in specific scenarios. In some scenarios, multiple challenges exist at the same time, and different models need to be frequently selected and switched to perform pedestrian re-identification on image data in different scenarios. Such a method is too cumbersome in practical applications and will take up more storage and computing space.

[0004] Therefore, there is an urgent need to provide a training method and a re-identification method that supports multi-scenario pedestrian re-identification models to overcome the above problems.

[0005] In view of this, this application is hereby filed. Summary of the invention

[0006] The technical problem to be solved by the present invention is to overcome the deficiency that the pedestrian re-identification model commonly used in the prior art is only suitable for specific scenarios, and the purpose is to provide a training method and a re-identification method for a pedestrian re-identification model that supports multiple scenarios.

[0007] In order to solve the above technical problems, the basic concept of the technical solution adopted by the present invention is: a training method for a pedestrian re-identification model supporting multiple scenes, characterized in that it includes the steps of:

[0008] S1. Build a training dataset for person re-identification in various scenarios, and perform data collection and data annotation;

[0009] S2. Build a pedestrian target detection network, take video image data as input, and perform two-dimensional target detection;

[0010] S3, introduce multi-scene prompts based on prompts for joint training and learn the multi-scene re-identification knowledge base;

[0011] S4, distillation of specific scene prompts;

[0012] S5. Define the loss function and perform model training to generate a pedestrian re-identification model that supports multiple scenarios.

[0013] According to an embodiment of the present invention, step S1 comprises:

[0014] S1.1. Using open source person re-identification datasets in different scenarios, we can collect video image data of the same target person from different camera angles in different types of scenarios.

[0015] S1.2. Pedestrian ID labeling is performed on video image data from different camera angles.

[0016] According to an embodiment of the present invention, the different types of scenes include: regular scenes, pedestrian changing clothes scenes, pedestrian obscured scenes, image blur scenes, and different modal data scenes.

[0017] According to an embodiment of the present invention, the data annotation includes pedestrian target detection and pedestrian ID annotation, and the pedestrian category target in the image is annotated using a rectangular box tool, and the rectangular box position P = (X, Y, W, H);

[0018] Where (X, Y) represents the coordinate position of the center point of the pedestrian target, (W, H) represents the length and width of the pedestrian target rectangle, and the same pedestrian is marked with the same ID.

[0019] According to an embodiment of the present invention, before step S3, the method further includes:

[0020] The feature knowledge of different types of scenes is extracted from the pedestrian re-identification datasets in different scenes to form a multi-scene re-identification knowledge base.

[0021] According to one embodiment of the present invention, the pedestrian target detection network includes: a feature extraction network part and a target detection network part.

[0022] According to one embodiment of the present invention, for the feature extraction network part, a convolutional neural network is used, which takes the video image data as input and outputs a feature representation of the image.

[0023] According to an embodiment of the present invention, for the target detection network part, a multi-category detection method is applied to output the category and location information of the target;

[0024] The output is defined as:

[0025] O i =[c i ,x i ,y i ,w i ,h i ]

[0026] O i Denoted as the i-th target, c i Indicates the category of the extracted target, x i ,y i are the center point coordinates of the target rectangle, w i ,h i are the length and width of the target respectively.

[0027] After adopting the above technical solution, the present invention has the following beneficial effects compared with the prior art:

[0028] By applying the training method of a pedestrian re-identification model supporting multiple scenes provided by the present invention, a pedestrian re-identification model supporting multiple scenes is generated. There is no need to manually switch to other types of pedestrian re-identification models according to scene images. By learning a universal pedestrian re-identification model supporting multiple scenes from data of multiple scenes, it can be more conveniently deployed to application devices, which has positive significance for improving the efficiency and quality of pedestrian re-identification.

[0029] The specific implementation modes of the present invention are further described in detail below in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] The accompanying drawings are part of the present invention and are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention, but do not constitute an improper limitation of the present invention. Obviously, the drawings described below are only some embodiments. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. In the accompanying drawings:

[0031] Figure 1 A diagram showing the steps of a method for generating a pedestrian re-identification model supporting multiple scenarios in an embodiment of the present invention.

[0032] It should be noted that these drawings and textual descriptions are not intended to limit the conceptual scope of the present invention in any way, but are intended to illustrate the concept of the present invention for those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0033] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. The following embodiments are used to illustrate the present invention but are not used to limit the scope of the present invention.

[0034] In the description of the present invention, it should be noted that the directions or positional relationships indicated by terms such as “upper”, “lower”, “front”, “back”, “left”, “right”, “vertical”, “inside” and “outside” are based on the directions or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific direction, be constructed and operated in a specific direction, and therefore cannot be understood as a limitation on the present invention.

[0035] In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0036] like Figure 1 As shown, the present invention provides a training method for a pedestrian re-identification model supporting multiple scenarios, and the training method comprises the steps of:

[0037] S1. Build a training dataset for person re-identification in various scenarios and perform data collection and labeling.

[0038] The step S1 specifically includes:

[0039] S1.1. Using open source person re-identification datasets in different scenarios, we can collect video image data of the same target person from different camera angles in different types of scenarios.

[0040] S1.2. Perform pedestrian ID tagging on the video image data under the different camera angles.

[0041] Specifically, the different types of scenes include, but are not limited to, conventional scenes (pedestrians are not obstructed, do not change clothes, and the image is clear), pedestrian changing clothes scenes (changing clothes scenes), pedestrian obstructed scenes, image blurred scenes, different modal data scenes, and other scene types.

[0042] Specifically, for the above different types of scenarios, different open source datasets can be selected according to the different types of scenarios;

[0043] For conventional scenarios, datasets such as Market-1501 and MSMT17 can be used;

[0044] For pedestrians changing clothes, datasets such as PRCC and Celeb-ReID can be used;

[0045] For scenes with blurred image pixels, datasets such as MLR-CUHK03 can be used;

[0046] For occluded scenes, datasets such as Occluded-Duke can be used;

[0047] For cross-modal data scenarios, datasets such as SYSU-mm01 can be used.

[0048] Specifically, for the data collection, by using multiple different cameras to collect data, data of different scenes such as different clothing, occlusion changes, and lighting changes of the same target person are constructed;

[0049] Specifically, the data annotation includes pedestrian target detection and pedestrian ID annotation. The pedestrian category target in the image is annotated using a rectangular box tool, and the rectangular box position P = (X, Y, W, H), where (X, Y) represents the coordinate position of the center point of the pedestrian target, and (W, H) represents the length and width of the pedestrian target rectangular box. The same pedestrian is annotated with the same ID;

[0050] Specifically, for the multi-scene person re-identification dataset, it is expressed as D = {x i ,y i ,s i}, where x i ,y i Represents pedestrian images and corresponding IDs, s i Indicates the corresponding scenes, such as occlusion, changing clothes, etc.

[0051] S2. Construct a pedestrian target detection network, and use the video image data as input to perform two-dimensional target detection.

[0052] By applying two-dimensional object detection, it is possible to locate and identify the target in the two-dimensional image, and determine its category information and position information.

[0053] Specifically, the pedestrian target detection network includes: a feature extraction network part and a target detection network part;

[0054] 1) For the feature extraction part, a commonly used backbone network is used, including but not limited to convolutional neural networks such as ResNet, MobileNet, and VGG, with the input being an image (the video image data) and the output being a feature representation of the image;

[0055] 2) For the target detection network part, including but not limited to using commonly used multi-category detection methods such as YOLO, SSD, CenterNet, etc., the output is the category and location information of the target;

[0056] The output is defined as:

[0057] O i =[c i ,x i ,y i ,w i ,h i ]

[0058] O i Denoted as the i-th target, c i Indicates the category of the extracted target, x i ,y i are the center point coordinates of the target rectangular box, w i ,h i are the length and width of the target respectively;

[0059] Specifically, according to the target detection results, the coordinate position of the pedestrian two-dimensional target detection box is obtained, the pedestrian image is extracted from the image, the width and height of the extracted image are calculated, and the longer side is taken. The size of the longer side is fixed to 256, and then the shorter side (relative to the longer side) is reduced in the same proportion. Finally, the size of the shorter side is expanded to 128, and the insufficient part is filled with pixel value 0. Finally, the pedestrian (target person) image size is unified to 256*128.

[0060] S3. Introduce prompt-based multi-scene prompts for joint training and learn a multi-scene re-ID knowledge base.

[0061] Specifically, the role of this stage is to extract feature knowledge of different types of scenes from data sets of different scenes to form a multi-scene re-ID knowledge base; the multi-scene re-ID knowledge base is learned by introducing a cue-based multi-scene joint training process, in which the cue information of a specific scene can adaptively encode the specific scene knowledge, and the features extracted by the backbone network are general features that are independent of the scene;

[0062] Specifically, the prompt information for a specific scene is expressed as P = {Pj |j=1,2,3,...,S}, where S represents the number of all scenes, P j Indicates the N additional prompts included in the jth scene; S=4, N=2 can be set to form a prompt information pool;

[0063] Specifically, for the cue-based multi-scene joint training, a visual transformer-based network structure is used for learning. For each input image x i , and its corresponding specific scene prompt s i Trained together as labels;

[0064] Specifically, for the input image x i And its corresponding specific scene prompts i , using the general visual transformer processing process, firstly, data preprocessing is performed, including block-based feature embedding, adding category information, etc.; the preprocessed image information is represented as {e [CLS] ; E i}, where E i That is, it represents the feature sequence after the input image is converted, e [CLS] represents the category information; then the scene prompt information is concatenated with the image information, expressed as {e [CLS] ; E i ;P si}, as the input of the visual transformer network;

[0065] Specifically, for the visual transformer network, it is composed of multiple layers of transformer network layers, and n layers can be set; for example, n can be set to 5, and the feature dimension of each layer is 512 dimensions;

[0066] Each transformer layer is divided into two sub-layers. The first sub-layer is a multi-head attention module, where the multi-head is represented by m-heads. For example, m can be set to 8.

[0067] The second sublayer is a standard feedforward neural network. In each transformer layer, the output of each multi-head attention is residually connected with the input of the current encoder layer, and the output result is sent to the normalization layer, which uses a layer normalization layer. The normalized result is then sent to the feedforward neural network layer, and then the residual connection calculation and layer normalization calculation are performed again, and the result is sent to the next transformer layer for repetition.

[0068] Specifically, by utilizing the visual transformer network, a global feature representation f is obtained for subsequent person re-identification.

[0069] The stage represented by step S3 is to learn a multi-scene re-identification library based on pedestrian re-identification data of multiple scenes, adopt a variety of specific scene prompt information, adaptively learn different specific scene knowledge under the prompt of scene labels, and learn the distinctive feature scene knowledge that describes different specific scenes. It plays an important role in multi-scene pedestrian re-identification.

[0070] S4. Distillation of scene-specific cues.

[0071] In the stage represented by step S4, different knowledge captured by scene-specific prompts in the multi-scene re-identification library is adaptively unified into the student model, which has multiple prompts and can effectively handle pedestrian re-identification in various scenarios.

[0072] Specifically, when using the model for person re-identification prediction, it is assumed that the scene label of the image is known, and then the global feature representation is used for person re-identification; however, in actual scenarios, it is difficult to obtain scene labels for some specific scene data. In addition, some scene images may contain multiple scene features, such as blur and pedestrian occlusion at the same time. Therefore, it is difficult to achieve the optimal person re-identification result by only using the knowledge of a specific scene.

[0073] Specifically, therefore, the knowledge distillation of the multi-scene re-identification knowledge base is adopted to obtain a general knowledge, and this process is represented as the distillation of specific scene prompts;

[0074] Specifically, in model distillation, the learning model is defined as a general branch model, the teacher model is the multi-scene re-identification knowledge base in the above steps, and the student model has the same backbone network structure as the teacher model. The difference lies in the selection process of the prompt knowledge.

[0075] Specifically, randomly initialize M general prompt information P v , unlike the above-mentioned selection of cues based on scene labels, this student model applies all common cues to any image in any scene. The input of the corresponding visual transformer network is defined as {e [CLS] ; E i ;P v};

[0076] Specifically, during the model distillation process, the teacher model, i.e., the multi-scene re-identification knowledge base in the above step, is frozen, the parameters remain unchanged, and the output includes the global feature f and the predicted probability output p; for the student model, the output is defined as the global feature f′ and the predicted probability output p′;

[0077] Specifically, the global feature representation output by the teacher model is used as supervision to guide the global feature output f′ of the student model.

[0078] S5. Define the loss function for model training;

[0079] Specifically, the loss function includes three aspects:

[0080] 1) The first aspect is the classification and detection box regression loss function of pedestrian target detection, L OD represents the two-dimensional target detection classification loss function, including target classification and position regression. OD =β1L class +β2L bbox

[0081] L class Represents the classification loss function of the target, including but not limited to using loss functions such as cross entropy, L bbox is a regression loss function of the target detection box, including but not limited to using a smooth L1 loss function;

[0082] 2) The second aspect is the loss function of the model training in step S3, which includes two aspects. One is the classification loss function, which uses the pedestrian ID as the training label to determine whether two pedestrians are the same person. The cross entropy loss function is used for calculation, which is expressed as:

[0083]

[0084] in, Represents the ID category of the predicted output, and L represents the true value of the annotation;

[0085] The second is the metric learning loss function, which aims to make the distance between two positive samples (i.e. the same target person in different frames) as small as possible and the distance between negative samples as large as possible (different pedestrians); the triplet loss function is adopted; the triplet consists of Anchor (A), Negative (N), and Positive (P), the base point is A, the positive match is P, and the negative match is N. The purpose of the triplet loss function is to make the distance between features of the same class as small as possible, and the distance between features of different classes as large as possible, which is expressed as:

[0086]

[0087] Represents the distance between two identical pedestrian features, the goal is to continuously reduce, It represents the distance between two different pedestrian features. The purpose is to provide continuous reminders. α is a constant parameter.

[0088] 3) The third aspect is to prompt the model training loss function of the distillation stage, which includes three aspects: the classification loss function, the metric learning loss function, and the distillation loss function. The first two loss functions are the same as the model training loss function of the first stage. For the distillation loss function, it is expressed as:

[0089]

[0090] in, Among them, f i 、f j Represents the scene s extracted using the multi-scene re-identification knowledge base i 、s j The characteristic graph of i , f′ j Represents the feature map of the learning model for .

[0091] The present invention also provides a pedestrian re-identification method supporting multiple scenarios, which uses the training method of the above-mentioned pedestrian re-identification model supporting multiple scenarios, or uses the pedestrian re-identification model supporting multiple scenarios generated by the training method of the pedestrian re-identification model supporting multiple scenarios to perform pedestrian re-identification.

[0092] The present invention provides a training method for a pedestrian re-identification model that supports multiple scenes, which generates a pedestrian re-identification model that supports multiple scenes. There is no need to manually switch to other types of pedestrian re-identification models according to scene images. By learning a universal pedestrian re-identification model that supports multiple scenes from data of multiple scenes, it can be more conveniently deployed to application devices, which has positive significance for improving the efficiency and quality of pedestrian re-identification.

[0093] The above description is only a preferred embodiment of the present invention, and does not limit the present invention in any form. Although the present invention has been disclosed as a preferred embodiment as above, it is not used to limit the present invention. Any technician familiar with this patent can make some changes or modifications to equivalent embodiments of equivalent changes by using the technical content suggested above without departing from the scope of the technical solution of the present invention. The implementation scheme in the above embodiment can also be further combined or replaced. However, any simple modification, equivalent change and modification made to the above embodiment based on the technical essence of the present invention without departing from the content of the technical solution of the present invention still falls within the scope of the solution of the present invention.

Claims

1. A training method for a pedestrian re-identification model supporting multiple scenes, characterized in that: The training method comprises the steps of: S1. Build a training dataset for person re-identification in various scenarios, and perform data collection and data annotation; S2. Build a pedestrian target detection network, take video image data as input, and perform two-dimensional target detection; S3, introduce multi-scene prompts based on prompts for joint training and learn the multi-scene re-identification knowledge base; S4, distillation of specific scene prompts; S5. Define the loss function and perform model training to generate a pedestrian re-identification model that supports multiple scenarios.

2. The method for training a person re-identification model supporting multiple scenarios according to claim 1, characterized in that: The step S1 comprises: S1.

1. Using open source person re-identification datasets in different scenarios, we can collect video image data of the same target person from different camera angles in different types of scenarios. S1.

2. Pedestrian ID labeling is performed on video image data from different camera angles.

3. The method for training a multi-scenario person re-identification model according to claim 2, characterized in that: The different types of scenes include: regular scenes, pedestrian changing clothes scenes, pedestrian obstructed scenes, image blur scenes, and different modal data scenes.

4. The method for training a multi-scenario person re-identification model according to claim 2, characterized in that: The data annotation includes pedestrian target detection and pedestrian ID annotation, and the pedestrian category target in the image is annotated using a rectangular box tool, and the rectangular box position P = (X, Y, W, H); Where (X, Y) represents the coordinate position of the center point of the pedestrian target, (W, H) represents the length and width of the pedestrian target rectangle, and the same pedestrian is marked with the same ID.

5. The method for training a multi-scenario person re-identification model according to claim 2, characterized in that: Before step S3, the following steps are also included: The feature knowledge of different types of scenes is extracted from the pedestrian re-identification datasets in different scenes to form a multi-scene re-identification knowledge base.

6. The method for training a multi-scenario person re-identification model according to claim 4, characterized in that: The pedestrian target detection network includes: a feature extraction network part and a target detection network part.

7. The method for training a multi-scenario person re-identification model according to claim 6, characterized in that: For the feature extraction network part, a convolutional neural network is used, which takes the video image data as input and outputs the feature representation of the image.

8. The method for training a multi-scenario person re-identification model according to claim 6, characterized in that: For the target detection network part, a multi-category detection method is applied to output the category and location information of the target; The output is defined as: O i =[c i ,x i ,y i ,w i ,h i ] O i Denoted as the i-th target, c i Indicates the category of the extracted target, x i ,y i are the center point coordinates of the target rectangle, w i ,h i are the length and width of the target respectively.

9. A pedestrian re-identification method supporting multiple scenes, characterized in that: A method for training a multi-scenario person re-identification model according to any one of claims 1 to 8 is applied, Or, a pedestrian re-identification model supporting multiple scenarios is generated by applying the training method of the pedestrian re-identification model supporting multiple scenarios according to any one of claims 1-8.

Images