A pedestrian re-identification system and method based on a convolutional neural network

Abstract

The application discloses a pedestrian re-identification system and method based on a convolutional neural network, wherein the system comprises: a feature vector generation model which obtains pedestrian feature vectors of a pedestrian image containing a target pedestrian and multiple to-be-inquired images containing pedestrians respectively according to the pedestrian image and the multiple to-be-inquired images; the feature vector generation model comprises an image input module, a deep feature extraction module, a pedestrian local region extraction module and a feature alignment transformation splicing module, and the deep feature extraction module is a deep convolutional neural network; a similarity calculation module calculates similarities between the pedestrian feature vector of the pedestrian image and pedestrian feature vectors of the multiple to-be-inquired images respectively; and a pedestrian re-identification module arranges the multiple to-be-inquired images in descending order according to the sizes of the similarities, and takes a to-be-inquired image with a similarity greater than a preset similarity threshold value as a pedestrian re-identification result. The application can realize end-to-end pedestrian feature extraction and is suitable for cross-camera identification in multiple scenes.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of pedestrian re-identification, in particular to a pedestrian re-identification system and method based on a convolutional neural network. BACKGROUND

[0002] Public security is the premise of social order stability and national economic development, and is also the basic guarantee for protecting the safety of citizens' person and property and promoting the harmonious development of economy and society. With the development of economy and computer technology, in recent years, China has increased support for the public security field, and a large number of monitoring devices have been installed in crowded public places to prevent and respond to public security incidents.

[0003] A large amount of monitoring data helps to solve problems such as security incident tracing, suspect tracking, and missing person recovery, but traditional manual operation cannot extract effective information from a large amount of monitoring data, and using computers to solve these problems has become an inevitable trend, therefore, in recent years, pedestrian re-identification technology has been widely concerned.

[0004] Pedestrian re-identification (Person Re-identification, abbreviated as Re-ID) is also known as pedestrian re-identification, which is a key sub-problem of image retrieval. Pedestrian re-identification is to use computer vision algorithms to match cross-device pedestrian images or videos, that is, given a query image, retrieve the same pedestrian in the image library of different monitoring devices. Pedestrian re-identification has great application prospects in criminal investigation, intelligent security, intelligent transportation, intelligent retail and other fields.

[0005] Traditional pedestrian re-identification methods mainly use color, texture, edge, shape and salient region features designed by humans, combined with a metric criterion to achieve small-scale data sets, and project the manually designed features into a metric space with discriminative ability. In this metric space, the distance between pedestrian images with the same label is closer, and the distance between pedestrian images with different labels is farther apart, and the core idea is to make the distance between positive sample pairs smaller than the minimum distance between negative sample pairs.

[0006] However, manually designed features usually need to be specially designed according to the characteristics of the data itself, and have strong human subjective components; manually designed features depend on the data set they face, and usually perform well on a certain data set, but have poor generalization ability on other data sets, and need to redesign the features. SUMMARY

[0007] The present application aims to provide a pedestrian re-identification system and method based on a convolutional neural network to solve the technical problem of re-designing identification features for different data sets in the prior art and difficulty in identifying pedestrians in different cross-border scenarios.

[0008] The application can be achieved by the following technical solutions.

[0009] A pedestrian re-identification system based on a convolutional neural network, comprising:

[0010] A feature vector generation model is configured to obtain pedestrian feature vectors of a pedestrian image containing a target pedestrian and a plurality of query images containing pedestrians, respectively, based on the pedestrian image and the plurality of query images, the feature vector generation model comprising an image input module for inputting images, a deep feature extraction module for extracting deep semantic features of the images, a pedestrian local region extraction module for extracting pedestrian local region features, and a feature alignment transformation splicing module for obtaining pedestrian feature vectors based on the deep semantic features and the pedestrian local region features, the deep feature extraction module being a deep convolutional neural network.

[0011] A similarity calculation module is configured to calculate similarities between the pedestrian feature vector of the pedestrian image and the pedestrian feature vectors of the plurality of query images, respectively.

[0012] A pedestrian re-identification module is configured to arrange the plurality of query images in descending order according to the similarities, and add query images with a similarity greater than a preset similarity threshold to a pedestrian re-identification result set, the pedestrian re-identification result set being a result of pedestrian re-identification.

[0013] Optionally, the deep feature extraction module comprises:

[0014] a data input layer, a first convolutional combination layer, a feature extraction and sampling layer, a first channel splicing layer, and a second convolutional combination layer connected in sequence, the first convolutional combination layer and the second convolutional combination layer each comprising a convolutional layer, a batch normalization layer, and an activation layer.

[0015] The data input layer obtains image data input by the image input module and preprocessed, the first convolutional combination layer extracts features from the image data to obtain first features, the feature extraction and sampling layer sequentially extracts features, down-samples, and up-samples the first features to obtain second features, the first channel splicing layer splices channels of the second features to obtain third features, and the second convolutional combination layer extracts features from the third features to obtain deep semantic features.

[0016] Optionally, the feature extraction and sampling layer comprises:

[0017] a feature extraction and down-sampling layer and a feature extraction and up-sampling layer, the feature extraction and down-sampling layer being connected to the first convolutional combination layer and the feature extraction and up-sampling layer.

[0018] The feature extraction and down-sampling layer performs feature extraction and down-sampling on the first feature, and the feature extraction and up-sampling layer performs feature extraction and up-sampling on the down-sampled feature.

[0019] Optionally, the feature extraction and down-sampling layer comprises a first feature extraction and down-sampling sub-layer, a second feature extraction and down-sampling sub-layer, a third feature extraction and down-sampling sub-layer and a fourth feature extraction and down-sampling sub-layer connected in sequence, the first feature extraction and down-sampling sub-layer comprises:

[0020] a first dense layer and a first down-sampling transition layer, the first dense layer is connected with the first convolution combination layer and the first down-sampling transition layer respectively, and the first down-sampling transition layer is connected with the first dense layer and the second feature extraction and down-sampling sub-layer respectively;

[0021] The first dense layer comprises a first number of dense units, the dense unit comprises a first combination unit and a second combination unit, the first combination unit comprises a first convolution layer, a first batch normalization layer and a first activation layer, and the second combination unit comprises a second convolution layer, a second batch normalization layer and a second activation layer.

[0022] The configuration of the first convolution layer is (1*1, 1, 0, 64), which means that the first convolution layer selects a convolution kernel with a size of 1*1, the number of convolution kernels is 64, the step is 1, and full 0 padding is used.

[0023] The configuration of the second convolution layer is (3*3, 1, 1, 16), which means that the second convolution layer selects a convolution kernel with a size of 3*3, the number of convolution kernels is 16, the step is 1, and full 1 padding is used.

[0024] The first down-sampling transition layer is composed of a third convolution layer and a pooling layer connected in series, the configuration of the third convolution layer is (1*1, 1, 0, 128), which means that the third convolution layer selects a convolution kernel with a size of 1*1, the number of convolution kernels is 128, the step is 1, and full 0 padding is used.

[0025] The configuration of the pooling layer is (2*2, 2), which means that the pooling layer selects a pooling kernel with a size of 2*2 and a step of 2.

[0026] Optionally, the second feature extraction and down-sampling sub-layer comprises:

[0027] a second dense layer and a second down-sampling transition layer, the second dense layer is connected with the first down-sampling transition layer and the second down-sampling transition layer respectively, and the second down-sampling transition layer is connected with the second dense layer and the third feature extraction and down-sampling sub-layer respectively.

[0028] The second dense layer includes a second number of the dense units.

[0029] The second down-sampling transition layer is composed of a fourth convolutional layer and the pooling layer in series, and the fourth convolutional layer is configured as (1*1, 1, 0, 192), which means that the fourth convolutional layer selects a convolutional kernel with a size of 1*1, the number of convolutional kernels is 192, the step is 1, and full 0 padding is used.

[0030] Optionally, the third feature extraction and down-sampling sub-layer includes:

[0031] The third dense layer and the third down-sampling transition layer, the third dense layer is connected with the second down-sampling transition layer and the third down-sampling transition layer respectively, and the third down-sampling transition layer is connected with the third dense layer and the fourth feature extraction and down-sampling sub-layer respectively;

[0032] The third dense layer includes a third number of the dense units.

[0033] The third down-sampling transition layer is composed of a fifth convolutional layer and the pooling layer in series, and the fifth convolutional layer is configured as (1*1, 1, 0, 256), which means that the fifth convolutional layer selects a convolutional kernel with a size of 1*1, the number of convolutional kernels is 256, the step is 1, and full 0 padding is used.

[0034] Optionally, the fourth feature extraction and down-sampling sub-layer includes:

[0035] The fourth dense layer and the fourth down-sampling transition layer, the fourth dense layer is connected with the third down-sampling transition layer and the fourth down-sampling transition layer respectively, and the fourth down-sampling transition layer is connected with the fourth dense layer and the feature extraction and up-sampling layer respectively;

[0036] The fourth dense layer includes a fourth number of the dense units.

[0037] The fourth down-sampling transition layer is composed of a sixth convolutional layer and the pooling layer in series, and the sixth convolutional layer is configured as (1*1, 1, 0, 256), which means that the sixth convolutional layer selects a convolutional kernel with a size of 1*1, the number of convolutional kernels is 256, the step is 1, and full 0 padding is used.

[0038] Optionally, the feature extraction and up-sampling layer includes:

[0039] The fifth dense layer and the up-sampling transition layer, the fifth dense layer is connected with the fourth down-sampling transition layer and the up-sampling transition layer respectively, and the up-sampling transition layer is connected with the fifth dense layer and the first channel splicing layer respectively;

[0040] The fifth dense layer includes a fifth number of the dense units.

[0041] The up-sampling transition layer includes a seventh convolutional layer, and the up-sampling multiple is 2; the configuration of the seventh convolutional layer is (1*1, 1, 0, 256), which indicates that the seventh convolutional layer selects a convolutional kernel with a size of 1*1, the number of the convolutional kernel is 256, the step is 1, and all 0 padding is used.

[0042] Optionally, the feature alignment transformation splicing module includes:

[0043] The feature alignment layer, the linear transformation layer and the second channel splicing layer are connected with each other, the feature alignment layer is connected with the second convolutional combination layer and the linear transformation layer, the linear transformation layer is connected with the feature alignment layer and the second channel splicing layer;

[0044] The feature alignment layer generates a corresponding local region mask according to the pedestrian local region feature, and the deep layer semantic feature is multiplied with the corresponding points of the local region mask and then subjected to a max-pooling operation to obtain a first feature vector, wherein the pedestrian local region feature is obtained by the pedestrian local region extraction module;

[0045] The linear transformation layer performs linear transformation on the first feature vector to obtain a second feature vector;

[0046] The second channel splicing layer performs channel splicing operation on the second feature vector to obtain a pedestrian feature vector.

[0047] The application further provides a pedestrian re-identification method based on a convolutional neural network, which includes:

[0048] The pedestrian image containing a target pedestrian and a plurality of query images containing pedestrians are respectively input into the trained feature vector generation model to obtain pedestrian feature vectors of the pedestrian image and each query image, wherein the feature vector generation model includes an image input module for inputting an image, a deep feature extraction module for extracting deep layer semantic features of the image, a pedestrian local region extraction module for extracting pedestrian local region features, and a feature alignment transformation splicing module for obtaining a pedestrian feature vector according to the deep layer semantic features and the pedestrian local region features, and the deep feature extraction module is a deep convolutional neural network.

[0049] The similarity between the target pedestrian feature vector and the pedestrian feature vector corresponding to each query image is calculated respectively.

[0050] The query images are arranged in descending order according to the similarity, and the query images with a similarity greater than a preset similarity threshold are added to a pedestrian re-identification result set, and the pedestrian re-identification result set is the result of pedestrian re-identification.

[0051] This invention provides a pedestrian re-identification system and method based on a convolutional neural network. The system includes: a feature vector generation model, used to obtain pedestrian feature vectors for the pedestrian image and each query image based on a pedestrian image containing a target pedestrian and multiple query images containing pedestrians. The feature vector generation model includes an image input module for inputting images, a deep feature extraction module for extracting deep semantic features of the images, a pedestrian local region extraction module for extracting local region features of pedestrians, and a feature alignment transformation and concatenation module for obtaining pedestrian feature vectors based on deep semantic features and local region features of pedestrians. The deep feature extraction module is a convolutional neural network with increased network depth. A similarity calculation module is used to calculate the similarity between the pedestrian feature vector of the pedestrian image and the pedestrian feature vector of each query image. A pedestrian re-identification module is used to sort the query images in descending order according to the similarity, and add the query images with similarity greater than a preset similarity threshold to a pedestrian re-identification result set. The pedestrian re-identification result set is the result of pedestrian re-identification.

[0052] In view of this, the beneficial effects of the present invention are as follows:

[0053] The deep feature extraction module of this invention is a deep convolutional neural network. Due to the increased network depth, it can obtain more abstract high-level semantic features. At the same time, the pedestrian local region features extracted by the pedestrian local region extraction module retain the shallow fine-grained features, and it also has a good recognition effect on small pedestrian local region features. The feature alignment transformation and splicing module combines high-level semantic features and shallow fine-grained features to obtain the final pedestrian feature vector, realizing end-to-end pedestrian feature extraction. Based on the pose alignment method, it can evaluate pedestrian features from various local regions of the pedestrian, and can be applied to cross-camera recognition in multiple scenarios. Attached Figure Description

[0054] Figure 1 This is a schematic diagram of the system structure of the present invention;

[0055] Figure 2 This is a schematic diagram of the deep feature extraction module in an embodiment of the present invention;

[0056] Figure 3 This is a schematic diagram of the structure of the dense layer in an embodiment of the present invention;

[0057] Figure 4 This is a schematic diagram of the structure of the dense unit in an embodiment of the present invention;

[0058] Figure 5 This is a schematic diagram of the structure of the first and second combined units in an embodiment of the system of the present invention;

[0059] Figure 6 a flowchart of the method of the present application;

[0060] Figure 7 a flowchart of the method of the present application;

[0061] Figure 8 a flowchart of the method of the present application; DETAILED DESCRIPTION

[0062] Terminology:

[0063] Person Re-identification (Re-ID for short) is also known as person re-identification, which is a key sub-problem of image retrieval. It is to match the images or videos of pedestrians across devices by using computer vision algorithms, that is, given a query image, the same pedestrian is retrieved in the image library of different monitoring devices. Due to its great application prospect in intelligent security and video monitoring, person re-identification has become a research focus in the field of computer vision.

[0064] The embodiment of the present application provides a person re-identification system and method based on a convolutional neural network, to solve the technical problems that different data sets need to be re-designed in the prior art, and it is difficult to identify pedestrians in different cross-border scenes.

[0065] In order to facilitate the understanding of the present application, the present application will be described more fully below with reference to the accompanying drawings. The preferred embodiments of the present application are shown in the drawings. However, the present application can be realized in many different forms and is not limited to the embodiments described herein. On the contrary, these embodiments are provided to make the disclosure of the present application more thorough and comprehensive.

[0066] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present application belongs. The terms used in the specification of the present application are only for the purpose of describing specific embodiments and are not intended to limit the present application. The term "and / or" used herein includes any and all combinations of one or more related listed items.

[0067] Please refer to Figure 1 The following is an embodiment of the person re-identification system based on the convolutional neural network of the present application, which includes:

[0068] The feature vector generation model 11 is configured to obtain a pedestrian feature vector of a pedestrian image containing a target pedestrian and a plurality of pedestrian query images containing pedestrians respectively, and the feature vector generation model comprises an image input module configured to input an image, a deep feature extraction module configured to extract deep semantic features of the image, a pedestrian local region extraction module configured to extract pedestrian local region features, and a feature alignment transformation splicing module configured to obtain the pedestrian feature vector based on the deep semantic features and the pedestrian local region features.

[0069] The similarity calculation module 22 is configured to calculate the similarity between the pedestrian feature vector of the pedestrian image and the pedestrian feature vector of each pedestrian query image respectively.

[0070] The pedestrian re-identification module 33 is configured to arrange the pedestrian query images in descending order according to the similarity, and add the pedestrian query images with a similarity greater than a preset similarity threshold to a pedestrian re-identification result set, wherein the pedestrian re-identification result set is the result of pedestrian re-identification.

[0071] The feature vector generation model in the embodiment is configured to obtain a pedestrian feature vector of a pedestrian image containing a target pedestrian and a plurality of pedestrian query images containing pedestrians respectively. Specifically, the feature vector generation model comprises an image input module, a deep feature extraction module, a pedestrian local region extraction module, and a feature alignment transformation splicing module.

[0072] The image input module inputs image data for which the feature vector is to be calculated, and in a preferred embodiment, one pedestrian image containing a specific pedestrian, i.e., a target pedestrian, and a plurality of pedestrian query images are input. The plurality of pedestrian query images form a search image database, and each pedestrian query image obtained through pedestrian detection contains one or more pedestrians. The pedestrians contained in each pedestrian query image may or may not include the target pedestrian.

[0073] The deep feature extraction module in the embodiment is a convolutional neural network with increased network depth, and the deep feature extraction module extracts features from the input pre-processed image data to obtain deep semantic features of the image data.

[0074] In a preferred embodiment, the deep feature extraction module in the embodiment comprises:

[0075] a data input layer, a first convolutional combination layer, a feature extraction and sampling layer, a first channel splicing layer, and a second convolutional combination layer connected in sequence; the first convolutional combination layer and the second convolutional combination layer each comprise a convolutional layer, a batch normalization layer, and an activation layer.

[0076] The data input layer obtains image data input by the image input module and preprocessed, the first convolution combination layer extracts features from the image data to obtain first features, the feature extraction and sampling layer sequentially extracts features, down-samples and up-samples the first features to obtain second features, the first channel splicing layer splices channels of the second features to obtain third features, and the second convolution combination layer extracts features from the third features to obtain deep semantic features.

[0077] In the embodiment, the feature extraction and sampling layer includes:

[0078] The feature extraction and down-sampling layer is connected with the first convolution combination layer and the feature extraction and up-sampling layer, respectively; the feature extraction and down-sampling layer extracts features and down-samples the first features, and the feature extraction and up-sampling layer extracts features and up-samples the down-sampled features.

[0079] In the embodiment, the feature extraction and down-sampling layer includes a first feature extraction and down-sampling sub-layer, a second feature extraction and down-sampling sub-layer, a third feature extraction and down-sampling sub-layer and a fourth feature extraction and down-sampling sub-layer connected in sequence, and the first feature extraction and down-sampling sub-layer includes:

[0080] The first dense layer is connected with the first convolution combination layer and the first down-sampling transition layer, respectively, and the first down-sampling transition layer is connected with the first dense layer and the second feature extraction and down-sampling sub-layer, respectively.

[0081] In the preferred embodiment, the first dense layer includes six dense units, and each dense unit includes a first combination unit and a second combination unit. The first combination unit includes a first convolution layer, a first batch normalization layer and a first activation layer, and the second combination unit includes a second convolution layer, a second batch normalization layer and a second activation layer. The first convolution layer is configured as (1*1, 1, 0, 64), which means that the first convolution layer selects a convolution kernel with a size of 1*1, the number of convolution kernels is 64, the step is 1, and full 0 padding is used. The second convolution layer is configured as (3*3, 1, 1, 16), which means that the second convolution layer selects a convolution kernel with a size of 3*3, the number of convolution kernels is 16, the step is 1, and full 1 padding is used. The first down-sampling transition layer is composed of a third convolution layer and a pooling layer connected in series. The third convolution layer is configured as (1*1, 1, 0, 128), which means that the third convolution layer selects a convolution kernel with a size of 1*1, the number of convolution kernels is 128, the step is 1, and full 0 padding is used. The pooling layer is configured as (2*2, 2), which means that the pooling layer selects a pooling kernel with a size of 2*2, and the step is 2.

[0082] It is worth mentioning that the dense unit in the embodiment includes two combined units, i.e., a first combined unit and a second combined unit, each of which includes a convolution layer, a batch normalization layer and an activation layer, wherein the configuration of the convolution layer in the first combined unit is (1*1, 1, 0, 64), and the configuration of the convolution layer in the second combined unit is (3*3, 1, 1, 16). In a preferred embodiment, the activation layer adopts a ReLU activation function.

[0083] In the embodiment, the second feature extraction and downsampling sub-layer includes:

[0084] The second dense layer and the second downsampling transition layer, the second dense layer is connected with the first downsampling transition layer and the second downsampling transition layer respectively, and the second downsampling transition layer is connected with the second dense layer and the third feature extraction and downsampling sub-layer respectively;

[0085] The second dense layer includes a second number of dense units, and in a preferred embodiment, the second dense layer includes 8 dense units; the second downsampling transition layer is composed of a fourth convolution layer and a pooling layer in series, and the configuration of the fourth convolution layer is (1*1, 1, 0, 192), which means that the fourth convolution layer selects a convolution kernel with a size of 1*1, the number of convolution kernels is 192, the step is 1, and full 0 padding is used.

[0086] In the embodiment, the third feature extraction and downsampling sub-layer includes:

[0087] The third dense layer and the third downsampling transition layer, the third dense layer is connected with the second downsampling transition layer and the third downsampling transition layer respectively, and the third downsampling transition layer is connected with the third dense layer and the fourth feature extraction and downsampling sub-layer respectively;

[0088] The third dense layer includes a third number of dense units, and in a preferred embodiment, the third dense layer includes 12 dense units; the third downsampling transition layer is composed of a fifth convolution layer and a pooling layer in series, and the configuration of the fifth convolution layer is (1*1, 1, 0, 256), which means that the fifth convolution layer selects a convolution kernel with a size of 1*1, the number of convolution kernels is 256, the step is 1, and full 0 padding is used.

[0089] In the embodiment, the fourth feature extraction and downsampling sub-layer includes:

[0090] The fourth dense layer and the fourth downsampling transition layer, the fourth dense layer is connected with the third downsampling transition layer and the fourth downsampling transition layer respectively, and the fourth downsampling transition layer is connected with the fourth dense layer and the feature extraction and upsampling layer respectively;

[0091] The fourth dense layer includes a fourth number of dense units, and preferably, the fourth dense layer includes 8 dense units.

[0092] In this embodiment, the feature extraction and up-sampling layer includes:

[0093] The fifth dense layer and the up-sampling transition layer, the fifth dense layer is connected with the fourth down-sampling transition layer and the up-sampling transition layer respectively, and the up-sampling transition layer is connected with the fifth dense layer and the first channel concatenation layer respectively.

[0094] The fifth dense layer includes a fifth number of dense units, and preferably, the fifth dense layer includes 8 dense units; and the up-sampling transition layer includes a seventh convolutional layer, and the up-sampling multiple is 2, and the configuration of the seventh convolutional layer is (1*1, 1, 0, 256), which means that the seventh convolutional layer selects a convolutional kernel with a size of 1*1, the number of convolutional kernels is 256, the step is 1, and full 0 padding is used.

[0095] In this embodiment, the feature alignment and transformation concatenation module includes:

[0096] The feature alignment layer, the linear transformation layer and the second channel concatenation layer, the feature alignment layer is connected with the second convolutional combination layer and the linear transformation layer respectively, and the linear transformation layer is connected with the feature alignment layer and the second channel concatenation layer respectively.

[0097] The feature alignment layer generates a corresponding local region mask according to the pedestrian local region feature, and performs a maximum pooling operation on the deep semantic feature multiplied by the corresponding points of the local region mask to obtain a first feature vector, and the pedestrian local region feature is obtained according to the pedestrian local region extraction module.

[0098] The linear transformation layer performs linear transformation on the first feature vector to obtain a second feature vector.

[0099] The second channel concatenation layer performs a channel concatenation operation on the second feature vector to obtain a pedestrian feature vector.

[0100] It is worth noting that in the first feature extraction and down-sampling sub-layer, the second feature extraction and down-sampling sub-layer, the third feature extraction and down-sampling sub-layer, the fourth feature extraction and down-sampling sub-layer and the feature extraction and up-sampling layer in this embodiment, the configurations of all dense units are the same. The pooling layer in this embodiment is a maximum pooling layer, and the configuration is (2*2, 2), which means that the pooling layer selects a pooling kernel with a size of 2*2, and the step is 2.

[0101] In the embodiment, the pedestrian local region extraction module extracts the pedestrian posture in the pedestrian image by using the trained deep learning model. In the preferred embodiment, first, the PoseNet pedestrian posture extraction model is used to obtain specific information of 17 key points of the human skeleton, and each key point information includes the horizontal coordinate, the vertical coordinate and the confidence of the key point; according to the key point information of the pedestrian, the head, the upper torso, the lower torso, the thigh, the calf and the shoe of the pedestrian are obtained as the six local region features of the pedestrian.

[0102] In the embodiment, the similarity calculation module calculates the similarity between the pedestrian feature vector of the pedestrian image and the pedestrian feature vector of each to-be-queried image. In the preferred embodiment, the cosine distance can be used to evaluate the similarity between the feature vectors, and the preset similarity threshold can be 0.5.

[0103] The embodiment proposes a CNN-based posture alignment pedestrian re-identification method for cross-camera pedestrian re-identification in multiple scenes. The method divides the pedestrian features by region through posture alignment, and uses the feature vectors corresponding to each part of the pedestrian after region division as the feature vectors for measuring the similarity of the pedestrian. The method realizes end-to-end pedestrian feature extraction, and the posture alignment-based method can evaluate the pedestrian features from each part of the pedestrian, which can be applied to cross-camera identification in multiple scenes.

[0104] The deep feature extraction module in the embodiment is a deep convolutional neural network. Due to the increased depth of the network, more abstract high-level semantic features can be obtained. At the same time, the pedestrian local region features extracted by the pedestrian local region extraction module also retain the shallow fine-grained features, which have good recognition effect on small pedestrian local region features. The feature alignment transformation and splicing module combines high-level semantic features and shallow fine-grained features to obtain the final pedestrian feature vector, realizes end-to-end pedestrian feature extraction, and the posture alignment-based method can evaluate the pedestrian features from each local region of the pedestrian, which can be applied to cross-camera identification in multiple scenes.

[0105] Please refer to Figure 7 , the following is another embodiment of a pedestrian re-identification method based on a convolutional neural network. The pedestrian image containing the target pedestrian and the plurality of to-be-queried image sets generated under multiple cameras are input into the trained feature vector generation model to obtain the pedestrian feature vectors corresponding to the pedestrian image and each to-be-queried image, respectively. The similarity between the pedestrian feature vector of the pedestrian image and each to-be-queried image is calculated, and it is judged whether the similarity corresponding to each to-be-queried image is greater than the preset similarity threshold. If the similarity corresponding to a to-be-queried image is greater than the preset similarity threshold, it is judged that the pedestrian in the to-be-queried image is the target pedestrian. The to-be-queried image is added to the pedestrian re-identification result set, and the pedestrian re-identification result set is the result of pedestrian re-identification and is output.

[0106] It is worth noting that in the embodiment, CxHxW is used to represent the size of the feature vector (feature map / image) in the calculation process, where C, H, and W represent the number of channels, the image height, and the image width of the feature vector (feature map), respectively. When C = 1, the size of the feature vector is represented by HxW.

[0107] Referring to Figure 8 , the process of generating a feature vector in the embodiment is as follows:

[0108] (1) input the image for which the feature vector needs to be calculated, for example, input the pedestrian image containing the target pedestrian and multiple images containing pedestrians to be queried;

[0109] (2) pre-process 1 and pre-process 2 are performed on the input image respectively, and the input image is resized to 3x384x128 and 3x256x192 respectively through resize operation. The image size of 3x384x128 indicates that the number of channels of the image is 3, the image height is 384 pixels, and the image width is 128 pixels;

[0110] (3) the deep semantic features of pedestrians are extracted by using the self-designed pedestrian feature extraction network ReNet (i.e., a deep feature extraction module). ReNet is a convolutional neural network with increased network depth, and the network structure is as shown in Figure 2 The size of the image output by the network is 256x24x8.

[0111] (4) the key point information of pedestrians is extracted by using the public pedestrian pose extraction algorithm PoseNet, and the output size is 17x3, where 17 represents the number of extracted pedestrian key points, and 3 represents the key point information, which is the horizontal coordinate, the vertical coordinate, and the confidence, respectively;

[0112] (5) perform pedestrian feature alignment (pose alignment), which mainly includes mask alignment pooling operation: first, use the key point information output by PoseNet to obtain the local region mask corresponding to the head, upper torso, lower torso, thigh, calf, and shoe of the pedestrian; then perform corresponding mask alignment pooling operation on the ReNet output feature map.

[0113] Specifically, the process of mask alignment pooling operation is as follows: first, generate a feature map with the same size as the ReNet output feature map, with 1 channel and internal element value of 0; then, set the region element value of the local region mask corresponding to the local region to 1 according to the obtained local region; finally, multiply the ReNet output feature map with the local region mask corresponding to each channel, and then perform maximum pooling.

[0114] In the embodiment, the feature vector is generated by Figure 2As can be known from the network structure explanation shown, the ReNet output feature map size is 256x24x8, and 6 feature vectors are obtained after mask alignment pooling, and the size of each feature vector is 1x256; each local region feature (head, upper torso, lower torso, thigh, calf, shoe) of the pedestrian corresponds to a feature vector.

[0115] (6) Linear transformation operation is performed, the linear transformation is a convolution operation with 6 convolution kernels being 1*1, a step being 1, a padding value being 0 and an output channel number being 128, and 6 feature vectors with the size of 1x128 are obtained respectively;

[0116] (7) Channel splicing operation is performed, the above 6 feature vectors with the size of 1x128 are subjected to channel splicing operation, and finally a feature vector with the size of 1x768 is output.

[0117] The pedestrian feature extraction network provided in the embodiment is formed in a dense set and is based on the dense structure of the dense set, and the network depth is increased; while deeper high-level semantic features are extracted, the information of shallow fine-grained features is retained, and the feature extraction for small layout regions of the human body is enhanced.

[0118] Referring to Figure 2 The deep feature extraction module in the pedestrian re-identification system based on the convolutional neural network embodiment of the application comprises:

[0119] (1) a data input layer Data, and the input image size is 3x384x128;

[0120] (2) a first convolution combination layer CBR1, configured as (3x3, 2, 1, 64), and the output feature map size is 64x192x64;

[0121] (3) a first feature extraction and downsampling sub-layer DBG1&TDL1, comprising a first dense layer DBG1 and a first downsampling transition layer TDL1. The first dense layer DBG1 is composed of 6 dense units, and the output feature map size is 160x192x64; the first downsampling transition layer TDL1 is composed of a first convolution layer and a pooling layer in series, the first convolution layer is configured as (1x1, 1, 0, 128), and the pooling layer is configured as (2x2, 2), and the output feature map size is 128x96x32.

[0122] (4) The second feature extraction and down-sampling sub-layer DBG2&TDL2 includes a second dense layer DBG2 and a second down-sampling transition layer TDL2. The second dense layer DBG2 is composed of 8 dense units, and the size of the output feature map is 256x96x32; the second down-sampling transition layer TDL2 is composed of a second convolution layer and a pooling layer in series, the configuration of the second convolution layer is (1x1, 1, 0, 192), and the configuration of the pooling layer is (2x2, 2), and the size of the output feature map is 192x48x16.

[0123] (5) The third feature extraction and down-sampling sub-layer DBG3&TDL3 includes a third dense layer DBG3 and a third down-sampling transition layer TDL3. The third dense layer DBG3 is composed of 12 dense units, and the size of the output feature map is 384x48x16; the third down-sampling transition layer TDL3 is composed of a third convolution layer and a pooling layer in series, the configuration of the third convolution layer is (1x1, 1, 0, 256), and the configuration of the pooling layer is (2x2, 2), and the size of the output feature map is 256x24x8.

[0124] (6) The fourth feature extraction and down-sampling sub-layer DBG4&TDL4 includes a fourth dense layer DBG4 and a fourth down-sampling transition layer TDL4. The fourth dense layer DBG4 is composed of 8 dense units, and the size of the output feature map is 384x24x8; the fourth down-sampling transition layer TDL4 is composed of a fourth convolution layer and a pooling layer in series, the configuration of the fourth convolution layer is (1x1, 1, 0, 256), and the configuration of the pooling layer is (2x2, 2), and the size of the output feature map is 256x12x4.

[0125] (7) The feature extraction and up-sampling layer DBG5&TUL1 includes a fifth dense layer DBG5 and an up-sampling transition layer TUL1. The fifth dense layer DBG5 is composed of 8 dense units, and the size of the output feature map is 384x12x4; the up-sampling transition layer TUL1 includes a seventh convolution layer, the configuration of the seventh convolution layer is (1x1, 1, 0, 256), the up-sampling layer multiple is 2, and the size of the output feature map is 256x24x8.

[0126] (8) The first channel splicing layer Concat1, and the size of the output feature map is 512x24x8.

[0127] (9) The second convolution combination layer CBR2, the configuration of the convolution layer is (3x3, 1, 1, 256), and the size of the output feature map is 256x24x8.

[0128] In the embodiment, DBG represents a dense layer, such as Figure 3As shown in the figure, each dense layer is composed of a plurality of dense units connected in series, and the output of each dense unit is subjected to a channel splicing operation with the input of the next dense unit.

[0129] As shown in the figure, each dense unit includes two combination units, that is, [1x1 CBR, 3x3 CBR]; as shown in the figure, Figure 4 each CBR is a combination unit, which includes three parts: a convolution layer (Conv), a batch normalization layer (BN) and an activation layer (Relu). Figure 5

[0130] In this embodiment, the training process of the deep feature extraction module is as follows:

[0131] (1) Collect and make pedestrian re-identification sample data, and assume that a pedestrian data set includes m pedestrian IDs, and a total of n cameras, wherein the data set is distributed as follows:

[0132] Training set (Train): including t pedestrian IDs, and each pedestrian ID includes all the data of the pedestrian collected under p (1<p≤n) cameras;

[0133] Query set (query): including all pedestrian IDs except the t pedestrian IDs in the training set, and for each pedestrian ID, at least one pedestrian data collected under p (1<p≤n) cameras is selected as query data;

[0134] Gallery set (gallery): including pedestrian IDs same as the query set, and the data under each pedestrian ID is all the data except the query data.

[0135] (2) Design a pedestrian feature extraction network, and define a pedestrian re-identification loss function, the formula is as follows:

[0136]

[0137] wherein represents the prediction cross-entropy loss of the pth component, v p represents the visual attribute of the pth local region, and when the local region is not visible, the loss calculation is not performed.

[0138] (3) Set the training strategy, and adjust the learning rate, the size of a single training batch, and the optimization strategy to iteratively train the sample to the loss function convergence, and obtain the final pedestrian re-identification feature extraction model.

[0139] The experimental results on a certain data set collected by the pedestrian re-identification system based on the convolutional neural network provided in the embodiment of the application are as follows:

[0140] ​

[0141] From the experimental results, it can be seen that the ReNet network provided in the embodiment has better effects on multiple evaluation indexes of a certain pedestrian data set. The ReNet network provided in the embodiment is based on a dense structure, increases the depth of the network, and obtains more abstract high-level semantic features; on the other hand, the shallow fine-grained features are also retained, and good recognition effects are obtained for small component regions.

[0142] AP (Average Precision) and mAP (mean Average Precision). AP measures the goodness (effectiveness) of the judgment result of the model on a single class, which is the average of the accuracy of each correct prediction. mAP measures the average value of AP of the model on all classes.

[0143] CMC (Cumulative Match Characteristic). The CMC curve (Cumulative Match Characteristic Curve) has a horizontal axis representing Rank-n and a vertical axis representing Accuracy. CMC represents that the first k gallery samples contain the query image, so CMC1 represents that the first 1 gallery sample contains the query image, and CMC5 represents that the first 5 gallery samples contain the query.

[0144] Please refer to Figure 6 , the following is an embodiment of a pedestrian re-identification method based on a convolutional neural network, comprising:

[0145] S100: input a pedestrian image containing a target pedestrian and a plurality of query images containing pedestrians into a trained feature vector generation model respectively, and obtain pedestrian feature vectors of the pedestrian image and each query image, respectively, the feature vector generation model comprising an image input module for inputting an image, a deep feature extraction module for extracting deep semantic features of an image, a pedestrian local region extraction module for extracting pedestrian local region features, and a feature alignment transformation splicing module for obtaining pedestrian feature vectors according to deep semantic features and pedestrian local region features, the deep feature extraction module being a convolutional neural network with increased network depth;

[0146] S200: calculate the similarity between the pedestrian feature vector of the pedestrian image and the pedestrian feature vector of each query image, respectively;

[0147] S300: determine the query image with the highest similarity as the query result of the target pedestrian.S300: arranging the to-be-query images in descending order according to the sizes of the similarities, and adding the to-be-query image whose similarity is greater than a preset similarity threshold to a pedestrian re-identification result set, wherein the pedestrian re-identification result set is a result of the pedestrian re-identification.

[0148] The pedestrian re-identification method based on the convolutional neural network provided in the embodiment has a deep convolutional neural network as the deep feature extraction module. Since the depth of the network is increased, more abstract high-level semantic features can be obtained. Meanwhile, the local region features of the pedestrian extracted by the pedestrian local region extraction module also retain the shallow fine-grained features, and have good recognition effect on small local region features of the pedestrian. The feature alignment transformation splicing module combines the high-level semantic features and the shallow fine-grained features to obtain the final pedestrian feature vector, realizes end-to-end pedestrian feature extraction, and the posture alignment-based mode can evaluate the pedestrian features from each local region of the pedestrian, and can be applied to cross-camera recognition in multiple scenes.

[0149] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described system, device and unit can refer to the corresponding processes in the foregoing method embodiments, which will not be described herein.

[0150] In the embodiments provided in the present application, it should be understood that the disclosed system, device and method can be implemented in other manners. For example, the device embodiments described above are only schematic; the division of the units is only a logical function division, and there can be another division manner in actual implementation; for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed mutual couplings or direct couplings or communication connections can be indirect couplings or communication connections through some interfaces, devices or units, and can be electrical, mechanical or in other forms.

[0151] The units described as separate components can or can not be physically separate, and the components displayed as units can or can not be physical units, i.e., can be located in one place, or can be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purposes of the embodiments.

[0152] In addition, each functional unit in the various embodiments of the present application can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit. The integrated unit can be implemented in the form of hardware, or in the form of software functional units.

[0153] The integrated unit, if implemented in the form of a software function unit and sold or used as an independent product, can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application or the entire or part of the technical solutions that essentially contribute to the prior art can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in the various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0154] The above embodiments are only used to illustrate the technical solutions of the present application, but not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, it should be understood by those skilled in the art that the technical solutions recorded in the foregoing embodiments can be modified, or some technical features can be replaced by equivalents; and these modifications or replacements do not make the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A person re-identification system based on convolutional neural networks, characterized in that, include: A feature vector generation model is used to obtain pedestrian feature vectors for the pedestrian image and each query image based on a pedestrian image containing a target pedestrian and multiple query images containing pedestrians. The feature vector generation model includes an image input module for inputting images, a deep feature extraction module for extracting deep semantic features of the images, a pedestrian local region extraction module for extracting local region features of pedestrians, and a feature alignment transformation and concatenation module for obtaining pedestrian feature vectors based on deep semantic features and local region features of pedestrians. The deep feature extraction module is a deep convolutional neural network. The similarity calculation module is used to calculate the similarity between the pedestrian feature vector of the pedestrian image and the pedestrian feature vector of each query image. The pedestrian re-identification module is used to sort the query images in descending order according to the similarity, and add the query images with similarity greater than a preset similarity threshold to the pedestrian re-identification result set, which is the result of pedestrian re-identification; the deep feature extraction module includes: a data input layer, a first convolutional combination layer, a feature extraction and sampling layer, a first channel stitching layer, and a second convolutional combination layer connected in sequence; both the first and second convolutional combination layers include a convolutional layer, a batch normalization layer, and an activation layer; the data input layer acquires image data input by the image input module and preprocessed; the first convolutional combination layer extracts features from the image data to obtain a first feature; the feature extraction and sampling layer sequentially extracts, downsamples, and upsamples the first feature to obtain a second feature; the first channel stitching layer stitches channels into the second feature to obtain a third feature; and the second convolutional combination layer extracts features from the third feature to obtain deep semantic features; The feature extraction and sampling layer includes a feature extraction and downsampling layer and a feature extraction and upsampling layer. The feature extraction and downsampling layer is connected to the first convolutional combination layer and the feature extraction and upsampling layer, respectively. The feature extraction and downsampling layer performs feature extraction and downsampling on the first feature, and the feature extraction and upsampling layer performs feature extraction and upsampling on the downsampled feature.

2. The pedestrian re-identification system based on convolutional neural networks according to claim 1, characterized in that, The feature extraction and downsampling layer includes a first feature extraction and downsampling sub-layer, a second feature extraction and downsampling sub-layer, a third feature extraction and downsampling sub-layer, and a fourth feature extraction and downsampling sub-layer connected in sequence. The first feature extraction and downsampling sub-layer includes: A first dense layer and a first downsampling transition layer, wherein the first dense layer is connected to the first convolutional combination layer and the first downsampling transition layer respectively, and the first downsampling transition layer is connected to the first dense layer and the second feature extraction and downsampling sub-layer respectively; Wherein, the first dense layer includes a first number of dense units, the dense unit includes a first combination unit and a second combination unit, the first combination unit includes a first convolutional layer, a first batch normalization layer and a first activation layer, and the second combination unit includes a second convolutional layer, a second batch normalization layer and a second activation layer. The configuration of the first convolutional layer is (1*1,1,0,64), which means that the first convolutional layer selects a convolutional kernel with a size of 1*1, the number of convolutional kernels is 64, the stride is 1, and it is padded with all zeros; The configuration of the second convolutional layer is (3*3,1,1,16), which means that the second convolutional layer selects a convolutional kernel with a size of 3*3, the number of convolutional kernels is 16, the stride is 1, and it is padded with all 1s; The first downsampling transition layer is composed of a third convolutional layer and a pooling layer connected in series. The configuration of the third convolutional layer is (1*1,1,0,128), which means that the third convolutional layer selects a convolutional kernel with a size of 1*1, the number of convolutional kernels is 128, the stride is 1, and it is filled with all zeros. The pooling layer is configured as (2*2,2), which means that the pooling layer selects a pooling kernel with a size of 2*2 and a step size of 2.

3. The person re-identification system based on convolutional neural networks according to claim 2, characterized in that, The second feature extraction and downsampling sublayer includes: The second dense layer and the second downsampling transition layer are respectively connected to the first downsampling transition layer and the second downsampling transition layer, and the second downsampling transition layer is respectively connected to the second dense layer and the third feature extraction and downsampling sub-layer. The second dense layer comprises a second number of the dense units; The second downsampling transition layer is composed of a fourth convolutional layer and the pooling layer connected in series. The fourth convolutional layer is configured as (1*1,1,0,192), which means that the fourth convolutional layer selects a convolutional kernel with a size of 1*1, the number of convolutional kernels is 192, the stride is 1, and it is filled with all zeros.

4. The pedestrian re-identification system based on convolutional neural networks according to claim 3, characterized in that, The third feature extraction and downsampling sublayer includes: The third dense layer and the third downsampling transition layer are respectively connected to the second downsampling transition layer and the third downsampling transition layer. The third downsampling transition layer is respectively connected to the third dense layer and the fourth feature extraction and downsampling sub-layer. The third dense layer comprises a third number of the dense units; The third downsampling transition layer is composed of the fifth convolutional layer and the pooling layer connected in series. The configuration of the fifth convolutional layer is (1*1,1,0,256), which means that the fifth convolutional layer selects a convolutional kernel with a size of 1*1, the number of convolutional kernels is 256, the stride is 1, and it is filled with all zeros.

5. The pedestrian re-identification system based on convolutional neural networks according to claim 4, characterized in that, The fourth feature extraction and downsampling sublayer includes: The fourth dense layer and the fourth downsampling transition layer are respectively connected to the third downsampling transition layer and the fourth downsampling transition layer, and the fourth downsampling transition layer is respectively connected to the fourth dense layer and the feature extraction and upsampling layer; The fourth dense layer comprises a fourth number of the dense units; The fourth downsampling transition layer is composed of the sixth convolutional layer and the pooling layer connected in series. The sixth convolutional layer is configured as (1*1,1,0,256), which means that the sixth convolutional layer selects a convolutional kernel with a size of 1*1, the number of convolutional kernels is 256, the stride is 1, and it is filled with all zeros.

6. The pedestrian re-identification system based on convolutional neural networks according to claim 5, characterized in that, The feature extraction and upsampling layer includes: The fifth dense layer and the upsampling transition layer are respectively connected to the fourth downsampling transition layer and the upsampling transition layer, and the upsampling transition layer is respectively connected to the fifth dense layer and the first channel splicing layer; The fifth dense layer comprises a fifth number of the dense units; The upsampling transition layer includes a seventh convolutional layer with an upsampling factor of 2. The configuration of the seventh convolutional layer is (1*1,1,0,256), which means that the seventh convolutional layer selects a convolutional kernel with a size of 1*1, the number of convolutional kernels is 256, the stride is 1, and it is padded with all zeros.

7. The pedestrian re-identification system based on convolutional neural networks according to any one of claims 1-6, characterized in that, The feature alignment transformation and splicing module includes: The system comprises a feature alignment layer, a linear transformation layer, and a second channel splicing layer. The feature alignment layer is connected to the second convolutional combination layer and the linear transformation layer, respectively. The linear transformation layer is connected to the feature alignment layer and the second channel splicing layer, respectively. The feature alignment layer generates a corresponding local region mask based on the pedestrian local region features. The deep semantic features are multiplied with the local region mask channel by channel and then max pooling is performed to obtain the first feature vector. The pedestrian local region features are obtained from the pedestrian local region extraction module. The linear transformation layer performs a linear transformation on the first feature vector to obtain the second feature vector; The second channel splicing layer performs channel splicing operation on the second feature vector to obtain the pedestrian feature vector.

8. A person re-identification method based on convolutional neural networks, characterized in that, include: The pedestrian image containing the target pedestrian and multiple query images containing pedestrians are respectively input into a trained feature vector generation model to obtain pedestrian feature vectors for the pedestrian image and each query image. The feature vector generation model includes an image input module for input images, a deep feature extraction module for extracting deep semantic features of the image, a pedestrian local region extraction module for extracting local features of the pedestrian, and a feature alignment transformation and concatenation module for obtaining pedestrian feature vectors based on deep semantic features and local features of the pedestrian. The deep feature extraction module is a deep convolutional neural network. The deep feature extraction module includes a data input layer, a first convolutional combination layer, a feature extraction and sampling layer, a first channel concatenation layer, and a second convolutional combination layer connected in sequence. Both the first and second convolutional combination layers include convolutional layers and batch normalization layers. The system includes an activation layer; the data input layer acquires preprocessed image data input by the image input module; the first convolutional combination layer extracts features from the image data to obtain a first feature; the feature extraction and sampling layer sequentially extracts, downsamples, and upsamples the first feature to obtain a second feature; the first channel concatenation layer concatenates the second feature to obtain a third feature; and the second convolutional combination layer extracts features from the third feature to obtain deep semantic features. The feature extraction and sampling layer includes a feature extraction and downsampling layer and a feature extraction and upsampling layer, which are respectively connected to the first convolutional combination layer and the feature extraction and upsampling layer. The feature extraction and downsampling layer extracts and downsamples the first feature, and the feature extraction and upsampling layer extracts and upsamples the downsampled feature. Calculate the similarity between the pedestrian feature vector of the pedestrian image and the pedestrian feature vector of each query image; The images to be queried are sorted in descending order according to the similarity. The images to be queried with a similarity greater than a preset similarity threshold are added to the pedestrian re-identification result set, which is the result of pedestrian re-identification.

Images