An attribute-aware domain-expanded person re-identification method and system

Abstract

The present invention discloses an attribute-aware domain expansion person re-identification method and system, which obtains an image to be identified; identifies the image to be identified using a trained recognition model, and determines the recognition result based on the output of the recognition model. The recognition model includes a backbone network, which is configured to extract a basic feature map based on the image to be identified, a local feature extractor, a global feature extractor, and a visual feature construction module, which is configured to construct a comprehensive visual feature based on the global visual feature and the local visual feature; a classifier, which is configured to perform attribute prediction based on the local semantic attribute feature and the global semantic attribute feature to output the probability corresponding to each attribute. The embodiment of the present invention realizes the simultaneous extraction of visual features and semantic attribute features of people in the video through the recognition model, thereby improving the accuracy of the two tasks of attribute recognition and person re-identification.

Description

Technical Field

[0001] The present invention relates to the field of image analysis technology, and in particular to an attribute-aware domain expansion pedestrian re-identification method and system. Background Art

[0002] Long-term historical experience in the security field shows that human image data collected by cameras is an effective clue for investigation. Extracting biometric features based on video data and conducting personal identification can greatly improve the accuracy and effectiveness of public security handling. Traditional video analysis is to extract facial features to conduct identity recognition. However, it is difficult to capture frontal, unobstructed, and clear faces in real-life snapshot data, which makes personal identity recognition and trajectory tracking difficult. Therefore, multi-granular multi-dimensional pedestrian full-body feature extraction and comparison become very important. Through the extraction and comparison of pedestrian multi-dimensional features, it is determined whether the target pedestrian appearing in a camera at a certain moment has appeared in the surveillance video of another camera at another moment, so as to quickly discover and continuously track key personnel. The pedestrian feature extraction algorithm currently used in the industry is usually a deep network that extracts visual features (feature vectors) and semantic attribute features (features such as clothing style and color) separately, which occupies more computing resources and has poor real-time performance. Summary of the invention

[0003] The embodiments of the present invention provide an attribute-aware domain expansion person re-identification method and system for simultaneously extracting visual features and semantic attribute features of people in a video, thereby improving the accuracy of both attribute recognition and person re-identification tasks, and realizing fast and accurate pedestrian retrieval using pedestrian features fused with multi-information.

[0004] The present disclosure provides an attribute-aware domain expansion person re-identification method, including:

[0005] Obtain an image to be recognized;

[0006] Recognize the image to be recognized using a trained recognition model, and determine a recognition result based on an output of the recognition model;

[0007] Wherein, the recognition model includes:

[0008] A backbone network is configured to extract a basic feature map based on the image to be recognized;

[0009] A local feature extractor, configured to determine local feature information at different positions based on the basic feature map;

[0010] A global feature extractor configured to determine global feature information based on the basic feature map;

[0011] A local visual feature extraction head configured to extract local visual features based on the local feature information;

[0012] A local semantic attribute feature extraction head is configured to extract local semantic attribute features based on the local feature information;

[0013] A global visual feature extraction head, configured to extract global visual features based on the global feature information;

[0014] A global semantic attribute feature extraction head is configured to extract global semantic attribute features based on the global feature information;

[0015] A visual feature construction module, configured to construct a comprehensive visual feature based on the global visual feature and the local visual feature;

[0016] The classifier is configured to perform attribute prediction based on the local semantic attribute features and the global semantic attribute features to output the probability corresponding to each attribute.

[0017] In some embodiments, the backbone network is constructed based on a convolutional neural network (CNN).

[0018] In some embodiments, the local feature extractor and the global feature extractor are constructed based on convolution flows of different scales respectively; and

[0019] Downsampling is configured in the global feature extractor, and the feature map size of the local feature extractor is larger than the feature map size of the global feature extractor.

[0020] In some embodiments, the recognition model includes several local feature extractors.

[0021] In some embodiments, the training process of the recognition model includes:

[0022] Acquire a training data set, wherein the training data set includes pedestrian re-identification data and pedestrian attribute recognition data and corresponding labels, and the pedestrian re-identification data and pedestrian attribute recognition data may be independent of each other;

[0023] Training the recognition model based on the training data set;

[0024] During training, a mask is set in the loss function calculation so that the loss function of each task focuses on the sample loss of its own task; and

[0025] The recognition model is trained based on the weights corresponding to the loss functions of each task.

[0026] In some embodiments, at least one of the local visual feature extraction head, the local semantic attribute feature extraction head, the global visual feature extraction head, and the global semantic attribute feature extraction head is constructed based on an attention mechanism;

[0027] The workflow of the attention mechanism includes:

[0028] Based on the input feature map, the weight distribution on the channel attention is obtained through pooling and sigmoid function and multiplied with the input feature map;

[0029] Based on the result of multiplication with the feature map, the weight distribution at the spatial position is obtained through convolution and sigmoid function and multiplied with the result of multiplication with the feature map to obtain an output feature map.

[0030] The embodiments of the present disclosure also provide an attribute-aware domain expansion pedestrian re-identification system, including a processor, wherein the processor is configured to execute the attribute-aware domain expansion pedestrian re-identification method described in each embodiment of the present disclosure.

[0031] The embodiments of the present disclosure further provide a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the attribute-aware domain expansion pedestrian re-identification method of each embodiment of the present disclosure are implemented.

[0032] The embodiment of the present invention realizes the simultaneous extraction of visual features and semantic attribute features of people in the video through the recognition model, thereby improving the accuracy of the two tasks of attribute recognition and pedestrian re-identification.

[0033] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented according to the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present invention. Moreover, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0035] Figure 1 This is a schematic diagram of the network structure of an embodiment of the present disclosure;

[0036] Figure 2 Schematic diagram of the loss function for data label supervision guidance in the prior art;

[0037] Figure 3A schematic diagram of a data label supervision guidance loss function for the disclosed method;

[0038] Figure 4 Schematic diagram of the attention mechanism of the visual and semantic attribute feature extraction head of the disclosed method. DETAILED DESCRIPTION

[0039] The exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0040] The disclosed embodiment proposes an attribute-aware domain expansion pedestrian re-identification method, such as Figure 1 As shown, including:

[0041] The image to be identified is obtained. Specifically, the image to be identified can be input into a neural network, and then the image to be identified is identified using a trained recognition model, and a recognition result is determined based on the output of the recognition model. The recognition model used in the pedestrian re-identification method disclosed in the present invention may include:

[0042] The backbone network is configured to extract a basic feature map based on the image to be recognized. The extracted basic feature map is used as an input for a subsequent local feature extractor and a global feature extractor. In some embodiments, the backbone network can be constructed based on a convolutional neural network (CNN).

[0043] The local feature extractor is configured to determine local feature information at different positions based on the basic feature map.

[0044] The global feature extractor is configured to determine the global feature information based on the basic feature map. The local feature extractor and the global feature extractor can extract the corresponding local feature information and global feature information.

[0045] A local visual feature extraction head configured to extract local visual features based on the local feature information;

[0046] A local semantic attribute feature extraction head is configured to extract local semantic attribute features based on the local feature information;

[0047] A global visual feature extraction head, configured to extract global visual features based on the global feature information;

[0048] The global semantic attribute feature extraction head is configured to extract global semantic attribute features based on the global feature information. The method disclosed in the present invention further performs feature extraction through the corresponding local feature extraction head and the global feature extraction head.

[0049] A visual feature construction module, configured to construct a comprehensive visual feature based on the global visual feature and the local visual feature;

[0050] The classifier is configured to perform attribute prediction based on the local semantic attribute features and the global semantic attribute features to output the probability corresponding to each attribute.

[0051] Visual features are constructed based on the features extracted by the corresponding local feature extraction head and the global feature extraction head, and attribute prediction is performed based on the local semantic attribute features and the global semantic attribute features through the classifier to output the probability corresponding to each attribute. This realizes the simultaneous extraction of visual features and semantic attribute features of people in the video, thereby improving the accuracy of the two tasks of attribute recognition and pedestrian re-identification.

[0052] In some embodiments, the local feature extractor and the global feature extractor are constructed based on convolution flows of different scales, respectively; and the global feature extractor is configured with downsampling, and the feature map size of the local feature extractor is larger than the feature map size of the global feature extractor. For example, a global branch and a local branch can be set after the backbone network. The global branch introduces downsampling to increase the receptive field of the model, focusing on global features, and can effectively capture a wide range of feature information; the feature map size of the local branch can be twice the size of the global feature map. By connecting convolution flows of multiple scales in parallel, the combined extraction of global and local features is realized, which not only captures features with a larger range, but also pays attention to fine-grained detail information. Then, the channel and spatial attention mechanisms are combined to enable the model to automatically pay attention to more important features, thereby improving the accuracy of pedestrian retrieval.

[0053] In some embodiments, the recognition model includes several local feature extractors, that is, there can be multiple local feature extractors, thereby forcing the network to learn detailed features of different positions (top, middle, bottom) of the image and extract fine-grained information.

[0054] Traditional person re-identification model training relies on specialized re-identification datasets, such as Figure 2As shown, for each pedestrian sample in the data set, there need to be multiple sample pictures of the same person under different shooting angles, which brings difficulties to the data set collection and annotation. Therefore, the number of samples of pedestrian re-identification data sets currently disclosed in the industry is relatively small, and the scene in a single data set is single. A separate attribute data set only has semantic labels but lacks pedestrian identity (ID) labels, and there are no sample pictures of the same person under multiple different perspectives, so it is impossible to train a pedestrian re-identification model. When existing pedestrian re-identification methods train models that support semantic attribute reasoning on pedestrian re-identification data sets, they need to additionally annotate a large amount of attribute information on the basis of traditional pedestrian re-identification data sets, which is time-consuming and labor-intensive. Moreover, the pedestrian re-identification models trained on these data sets generally have cross-domain problems, that is, the model trained on a certain data set only performs well on that data set, but the performance will drop significantly when it is migrated to other data sets. In some embodiments of the method disclosed herein, the training process of the recognition model includes: obtaining a training data set, such as Figure 3 As shown, the training data set includes pedestrian re-identification data and pedestrian attribute recognition data and corresponding labels, and the pedestrian re-identification data and pedestrian attribute recognition data can be independent of each other. The recognition model is trained based on the training data set;

[0055] During the training process, a mask is set in the loss function calculation so that the loss function of each task focuses on the sample loss of its own task; and the recognition model is trained based on the weights corresponding to the loss function of each task.

[0056] In this example, a training data set of 64 samples is used as an example. The traditional method training data set contains 64 samples and corresponding identity label attribute labels. The loss function is adjusted through training to complete the model training. Figure 3 As shown in the figure, the training set can be divided into 32 image identity label masks and attribute labels and 32 image identity label attribute label masks, that is, independent "pedestrian re-identification datasets" and "pedestrian attribute recognition datasets". Unlike traditional methods, it is mandatory that each training sample has samples of the same person from different perspectives in the dataset for training and learning visual features, and attribute features are annotated for learning semantic attribute features.

[0057] In the specific training process, by setting the mask in the loss function calculation, the loss function of different tasks only focuses on the sample loss of their own tasks, and then the loss functions of different tasks are combined with different weights to jointly guide the reverse deduction of the network. The model structure and training method of the disclosed method can be used for domain fusion, which can greatly alleviate the cross-domain problem. It solves the problem that the traditional method is only effective on the training sample set, but in other scenarios, due to different shooting angles, scenes, resolutions, etc., the actual reasoning and retrieval of pedestrians has a low accuracy.

[0058] A separate person re-identification dataset has pedestrian identity annotations, but the scene is single and the amount of data is small, while a separate attribute recognition dataset has more scenes and a larger number, but it cannot annotate pedestrian identities. If you simply use multiple pedestrian re-identification datasets to train the model, it will result in too many types of pedestrian identity labels, which will have a negative impact on model training. The method disclosed in this disclosure utilizes the commonality of pedestrian re-identification and attribute recognition tasks in extracting basic image features. By setting a mask in the calculation of the loss function, it is possible to train two types of datasets simultaneously without additional annotation, thereby expanding the scene domain that the model is adapted to and alleviating traditional cross-domain problems.

[0059] In some embodiments, at least one of the local visual feature extraction head, the local semantic attribute feature extraction head, the global visual feature extraction head, and the global semantic attribute feature extraction head is constructed based on an attention mechanism. The workflow of the attention mechanism includes:

[0060] Based on the input feature map, the weight distribution on the channel attention is obtained through pooling and sigmoid function and multiplied with the input feature map;

[0061] Based on the result of multiplication with the feature map, the weight distribution at the spatial position is obtained through convolution and sigmoid function and multiplied with the result of multiplication with the feature map to obtain an output feature map.

[0062] The method disclosed in the present invention also uses an attention mechanism when constructing the pedestrian visual and semantic attribute feature extraction head. Figure 4 As shown, the attention mechanism used by the method of the present invention combines channel attention and spatial attention. First, the weight distribution on the channel is obtained through pooling and sigmoid, and multiplied with the input feature map to guide the model to pay attention to more important channels; then, the feature map after the channel attention is taken as input, and the weight distribution in the spatial position is obtained through convolution and sigmoid operations, and then the weight is multiplied on the feature map after the channel attention, so as to realize automatic attention to the feature maps of important channels and space.

[0063] The method disclosed in this disclosure adds pedestrian comprehensive visual feature vectors for pedestrian comparison and retrieval, which is more accurate and robust. This is because high-dimensional feature vectors can capture and express more detailed features, rather than just coarse-grained features such as "male wearing a white short-sleeved shirt and black trousers".

[0064] The use of multi-task learning to share backbone feature extraction can achieve the extraction of more detailed information about each pedestrian without increasing the reasoning time. The present invention achieves end-to-end joint training and reasoning of two tasks. On the one hand, the backbone features are shared to save computing power and time, thereby improving real-time performance. On the other hand, since the information of the two tasks is combined at the same time to guide network training, the accuracy of pedestrian re-identification and attribute recognition is greatly improved.

[0065] The existing training set requires that each sample has both an identity label and an attribute label, which brings two problems: First, it is impossible to reuse the existing public data sets, and the identity and attribute information need to be annotated at the same time. Each person needs to have multiple pictures from different angles in the data set, and the collection and annotation are time-consuming and laborious; if you choose to additionally annotate the attribute information on the existing pedestrian re-identification data set, it will be affected by the small number of existing pedestrian re-identification and the single scene. The model has low robustness and is not accurate enough in reasoning in new scenes. The method disclosed in the present invention does not require additional labeling of any samples, which can greatly alleviate the problem of inaccurate cross-domain recognition in the prior art. The present invention uses the mask in the loss function calculation to train two types of data sets at the same time, expands the scene domain adapted to the model, and alleviates the traditional cross-domain problem.

[0066] The embodiments of the present disclosure also provide an attribute-aware domain expansion pedestrian re-identification system, including a processor, wherein the processor is configured to execute the attribute-aware domain expansion pedestrian re-identification method described in each embodiment of the present disclosure.

[0067] The embodiments of the present disclosure further provide a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the attribute-aware domain expansion pedestrian re-identification method of each embodiment of the present disclosure are implemented.

[0068] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0069] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0070] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for enabling a terminal (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0071] The embodiments of the present invention are described above in conjunction with the accompanying drawings, but the present invention is not limited to the above-mentioned specific implementation modes, which are merely illustrative rather than restrictive. Under the guidance of the present invention, ordinary technicians in this field can also make many forms without departing from the scope of protection of the present invention and the claims, all of which are within the protection of the present invention.

Claims

1. An attribute-aware domain-expanded person re-identification method, characterized in that: include: Obtain an image to be recognized; Recognize the image to be recognized using a trained recognition model, and determine a recognition result based on an output of the recognition model; Wherein, the recognition model includes: A backbone network is configured to extract a basic feature map based on the image to be recognized; A local feature extractor, configured to determine local feature information at different positions based on the basic feature map; A global feature extractor configured to determine global feature information based on the basic feature map; A local visual feature extraction head configured to extract local visual features based on the local feature information; A local semantic attribute feature extraction head is configured to extract local semantic attribute features based on the local feature information; A global visual feature extraction head, configured to extract global visual features based on the global feature information; A global semantic attribute feature extraction head is configured to extract global semantic attribute features based on the global feature information; A visual feature construction module, configured to construct a comprehensive visual feature based on the global visual feature and the local visual feature; The classifier is configured to perform attribute prediction based on the local semantic attribute features and the global semantic attribute features to output the probability corresponding to each attribute.

2. The attribute-aware domain expansion person re-identification method according to claim 1, characterized in that: The backbone network is constructed based on convolutional neural network (CNN).

3. The attribute-aware domain expansion person re-identification method according to claim 1, characterized in that: The local feature extractor and the global feature extractor are constructed based on convolution flows of different scales respectively; and Downsampling is configured in the global feature extractor, and the feature map size of the local feature extractor is larger than the feature map size of the global feature extractor.

4. The attribute-aware domain expansion person re-identification method according to claim 3, characterized in that: The recognition model includes several local feature extractors.

5. The attribute-aware domain expansion person re-identification method according to claim 1, characterized in that: The training process of the recognition model includes: Acquire a training data set, wherein the training data set includes pedestrian re-identification data and pedestrian attribute recognition data and corresponding labels, and the pedestrian re-identification data and pedestrian attribute recognition data may be independent of each other; Training the recognition model based on the training data set; During training, a mask is set in the loss function calculation so that the loss function of each task focuses on the sample loss of its own task; and The recognition model is trained based on the weights corresponding to the loss functions of each task.

6. The attribute-aware domain expansion person re-identification method according to claim 1, characterized in that: At least one of the local visual feature extraction head, the local semantic attribute feature extraction head, the global visual feature extraction head, and the global semantic attribute feature extraction head is constructed based on an attention mechanism; The workflow of the attention mechanism includes: Based on the input feature map, the weight distribution on the channel attention is obtained through pooling and sigmoid function and multiplied with the input feature map; Based on the result of multiplication with the feature map, the weight distribution at the spatial position is obtained through convolution and sigmoid function and multiplied with the result of multiplication with the feature map to obtain an output feature map.

7. An attribute-aware domain-expanded person re-identification system, characterized in that: It includes a processor configured to execute the attribute-aware domain expansion pedestrian re-identification method as described in any one of claims 1-6.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the attribute-aware domain expansion pedestrian re-identification method according to any one of claims 1 to 6.

Images