Pedestrian group representation method based on member progressive relationship

Abstract

The present invention discloses a pedestrian group representation method based on member progressive relationship, the method comprising: obtaining pedestrian pictures and extracting pedestrian features; performing linear projection dimensionality reduction on pedestrian features to obtain hidden codes of pedestrian features; introducing group representation vectors into the hidden codes of pedestrian features and performing padding to obtain a padded feature sequence; combining the group representation vectors in the padded feature sequence with individual appearance information to obtain group individual representation; combining the group representation vectors in the padded feature sequence with inter-individual correlation information to obtain group correlation representation; fusing group individual representation and group correlation representation to obtain multi-level group representation. By using the present invention, the feature information of the group can be better represented, thereby improving the accuracy of pedestrian re-identification. As a pedestrian group representation method based on member progressive relationship, the present invention can be widely used in the field of computer information processing.

Description

Technical Field

[0001] The present invention relates to the field of computer information processing, and in particular to a pedestrian group representation method based on member progressive relations. Background Art

[0002] Group re-identification technology refers to finding the picture with the highest overlap rate with the people in the picture to be found from another set of pictures, given a picture containing people to be found. There are two traditional methods: one is to match pictures based on manual feature extraction, and the other is to establish group features through convolutional neural networks and graph neural networks. The method based on manual feature extraction is not stable enough, resulting in poor results, and the representation ability is limited in large-scale search. The method based on convolutional neural networks and graph neural networks has the problems of slow calculation speed, slow network model convergence, poor results, and can only consider the correlation between local features. Summary of the invention

[0003] In order to solve the above technical problems, the purpose of the present invention is to provide a pedestrian group characterization method based on member progressive relationship, which can better characterize the characteristic information of the group, thereby improving the accuracy of pedestrian re-identification.

[0004] The first technical solution adopted by the present invention is: a pedestrian group characterization method based on member progressive relationship, comprising the following steps:

[0005] Get pedestrian pictures and extract pedestrian features;

[0006] Perform linear projection dimensionality reduction on pedestrian features to obtain the latent code of pedestrian features;

[0007] The implicit encoding of pedestrian features introduces a group representation vector and fills it in to obtain a feature sequence after filling.

[0008] The group representation vector in the filled feature sequence is combined with the individual appearance information to obtain the group individual representation;

[0009] The group representation vector in the filled feature sequence is combined with the inter-individual association information to obtain the group association representation;

[0010] The group individual representation and group association representation are fused to obtain a multi-level group representation.

[0011] Furthermore, the step of obtaining a pedestrian image and extracting pedestrian features specifically includes:

[0012] Get the pedestrian images in the group and resize them to get pedestrian images of uniform size;

[0013] Input pedestrian images of uniform size into the Resnet-50 network and output feature maps;

[0014] The feature map is processed by average pooling to obtain pedestrian features.

[0015] Furthermore, the step of introducing the group representation vector into the implicit encoding of the pedestrian features and performing padding to obtain the padded feature sequence specifically includes:

[0016] Add the group representation vector before the hidden code of pedestrian features to obtain a preliminary feature sequence;

[0017] Based on the filling feature, the preliminary feature sequence is padded to a preset length to obtain a padded feature sequence;

[0018] Set the pedestrian mask according to the feature interval.

[0019] Furthermore, the step of combining the group representation vector in the filled feature sequence with the individual appearance information to obtain the group individual representation specifically includes:

[0020] The padded feature sequence is learned by three multi-layer perceptrons to obtain the corresponding features Q1, K1 and V1;

[0021] Calculate the similarity of each feature with other features;

[0022] The similarity is transformed into softmax and multiplied with feature V1 to obtain the features with high similarity between each feature and other features;

[0023] The steps of perceptron learning and similarity calculation are repeated until a preset number of times is reached to obtain the similarity information of the first group of different subspaces;

[0024] The similar information of the first group of different subspaces is concatenated to obtain the individual representation of the group.

[0025] Furthermore, it also includes calculating the association information between individuals. The specific steps are as follows:

[0026] Add the individual representation of the group to the filled feature sequence and calculate the residual to obtain the first feature result;

[0027] Regularizing the first feature result and calculating the tensor based on a multi-layer perceptron to obtain a second feature result;

[0028] The second feature result is added to the first feature result and the residual is calculated to obtain the inter-individual association information.

[0029] Furthermore, the step of combining the group representation vector in the filled feature sequence with the inter-individual association information to obtain the group association representation specifically includes:

[0030] The inter-individual correlation information is passed through three multi-layer perceptrons to obtain the corresponding feature sequences Q2, K2 and V2;

[0031] Calculate the similarity between the group representation vector and other individual association information;

[0032] The similarity is transformed into softmax and multiplied with the feature sequence V2 to obtain the features with high similarity to the association information between individuals in the group representation vector;

[0033] The steps of perceptron learning and similarity calculation are repeated until a preset number of times is reached to obtain similar information of a second group of different subspaces;

[0034] The similar information of the second group of different subspaces is concatenated to obtain the group association representation.

[0035] Furthermore, the step of fusing the group individual representation and the group association representation to obtain a multi-level group representation specifically includes:

[0036] The group individual representation and the individual association representation are spliced ​​together, and then fused through a multi-layer perceptron to obtain the fusion result;

[0037] The fusion result is regularized and residually connected with the previous group individual representation to output a multi-level group representation.

[0038] Furthermore, it also includes:

[0039] Get the image to be tested and find similar images in pedestrian images based on the group association representation.

[0040] The beneficial effect of the method of the present invention is that the present invention calculates the association between pedestrian features, combines different levels of association information as group representation, and sets pedestrian masks to avoid association calculations between invalid features, thereby being able to better represent the feature information of the group and thus improving the re-identification accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 It is a flowchart of the steps of a pedestrian group characterization method based on member progressive relationship of the present invention;

[0042] Figure 2 It is a structural block diagram of a pedestrian group representation system based on member progressive relationship of the present invention. DETAILED DESCRIPTION

[0043] The present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. The step numbers in the following embodiments are only provided for the convenience of explanation and description, and the order between the steps is not limited in any way. The execution order of each step in the embodiment can be adaptively adjusted according to the understanding of those skilled in the art.

[0044] like Figure 1 As shown, the present invention provides a pedestrian group characterization method based on member progressive relationship, the method comprising the following steps:

[0045] S1, obtain pedestrian images and extract pedestrian features;

[0046] S1.1, obtaining pedestrian images in the group and resizing them to obtain pedestrian images of uniform size;

[0047] Specifically, obtain all pedestrian images {P1, P2, P3…, Pn} in a group, and limit the size of each pedestrian image to 256*128;

[0048] S1.2. Input pedestrian images of uniform size into the Resnet-50 network and output feature maps;

[0049] Specifically, these pedestrian pictures are input into Resnet-50, and the output of the last convolutional layer is taken as the feature map to obtain a feature map sequence, each of which has a size of 2048*7*1.

[0050] S1.3. Process the feature map by average pooling to obtain pedestrian features.

[0051] Specifically, these feature maps are average pooled to a size of 2048*4*1. Let the sequence be x={x1, x2, x3…x n}, the obtained feature map is average pooled to 2048*1, and then changed into a 2048-dimensional vector as the feature of a single pedestrian.

[0052] S2, perform linear projection dimensionality reduction on pedestrian features to obtain the latent code of pedestrian features;

[0053] Specifically, each feature map is passed through the reshape function to obtain a feature vector with a length of 2048*4, and finally through a fully connected layer to reduce the dimension to a feature vector with a length of 128.

[0054] S3, introduce the group representation vector into the hidden code of pedestrian features and fill it up to obtain the filled feature sequence;

[0055] S3.1. Add the group representation vector before the hidden code of pedestrian features to obtain a preliminary feature sequence;

[0056] Specifically, a 128-dimensional learnable feature vector is added before the hidden code of the pedestrian features. The cls vector is used to learn the group representation. The entire sequence is represented as {cls, F1, F2, F3…Fn}, which is used as the input for the subsequent group representation learning.

[0057] S3.2, padding the preliminary feature sequence to a preset length based on the padding feature to obtain a padded feature sequence;

[0058] Specifically, since the number of people in each group varies, and the length of the output sequence needs to be fixed and equal to the number of blocks of the deep self-attention transformer network, we need to use padding features to make the feature sequences of equal length, so the input feature sequence is input = {cls, F1, F2, F3...Fn, pad1, pad2...pad k}, Assuming that the number of blocks in the deep self-attention transformer network is N, the length of the feature sequence is also N, and the entire input size is N*128.

[0059] S3.3. Set a pedestrian mask according to the feature interval.

[0060] Specifically, the pedestrian mask mask = {0, 0, 0 ... 1, 1, 1} is set at the same time, with n + 1 zeros in front, corresponding to cls to F n The features between them are meaningful for group characterization, while the following k 1s correspond to k pad features, which are meaningless for characterizing the group.

[0061] S4, combining the group representation vector in the filled feature sequence with the individual appearance information to obtain the group individual representation;

[0062] S4.1, the padded feature sequence is learned by three multi-layer perceptrons to obtain the corresponding features Q1, K1 and V1;

[0063] S4.2, calculate the similarity between each feature and other features;

[0064] Specifically, the similarity calculation formula is as follows:

[0065]

[0066] where d k Represents the length of the input feature sequence. The similarity between all features and the filled features will be calculated as infinitesimal to avoid the impact of the filled features.

[0067] S4.3, perform softmax transformation on the similarity and multiply it with feature V to obtain a feature with a high similarity between each feature and other features;

[0068] S4.4, looping the steps of perceptron learning and similarity calculation until a preset number of times is reached, and obtaining the similarity information of the first group of different subspaces;

[0069] Specifically, steps S4.1-S4.3 are executed four times to obtain similar information of four different subspaces.

[0070] S4.5. Concatenate the similar information of the first group of different subspaces to obtain the individual representation of the group.

[0071] Specifically, similar information in the four subspaces is concatenated together, and then reduced to 128 dimensions through a multi-layer perceptron, and the output is N 128-dimensional tensors. The first tensor is the group individual representation, which has integrated the individual's apparent characteristics, and the remaining N-1 tensors contain the association information between individuals.

[0072] S5, combining the group representation vector in the filled feature sequence with the inter-individual association information to obtain the group association representation;

[0073] S5.1, pass the inter-individual correlation information through three multi-layer perceptrons to obtain the corresponding feature sequence Q2, feature sequence K2 and feature sequence V2;

[0074] S5.2, calculating the similarity between the group representation vector and the association information between other individuals;

[0075] Specifically, three multi-layer perceptrons are used to learn different feature sequences Q, K, and V. The similarity between the cls vector and other individual association information is calculated as follows. Similarly, pedestrian masks are used to eliminate the influence of the filling features:

[0076]

[0077] S5.3, perform softmax transformation on the similarity and multiply it with the feature sequence V2 to obtain features with high similarity to the association information between individuals in the group representation vector;

[0078] Specifically, the similarity obtained by S5.2 is softmax transformed and then multiplied with the V2 feature learned from the cls vector to obtain the features in the cls feature that have a higher similarity with the inter-individual association information.

[0079] S5.4, looping the steps of perceptron learning and similarity calculation until a preset number of times is reached, and obtaining a second set of similar information in different subspaces;

[0080] Specifically, steps S5.1-S5.3 are executed four times to obtain similar information of four different subspaces.

[0081] S5.5. Concatenate similar information of the second group of different subspaces to obtain group association representation.

[0082] Specifically, similar information in the four subspaces is spliced ​​together and then reduced to 128 dimensions through a multi-layer perceptron. The output group association representation incorporates the association information between individuals.

[0083] S6. Fuse the group individual representation and the group association representation to obtain a multi-level group representation.

[0084] S6.1, concatenate the group individual representation and the individual association representation, and then fuse them through a multi-layer perceptron to obtain a fusion result;

[0085] S6.2. Regularize the fusion result and make a residual connection with the previous group individual representation to output a multi-level group representation.

[0086] S7, obtaining the image to be tested and finding similar images in pedestrian images according to the group association representation.

[0087] As a further preferred embodiment of the method, between steps S4 and S5, the process further includes calculating the association information between individuals. The specific steps are as follows:

[0088] Add the individual representation of the group to the filled feature sequence and calculate the residual to obtain the first feature result;

[0089] Regularizing the first feature result and calculating the tensor based on a multi-layer perceptron to obtain a second feature result;

[0090] The second feature result is added to the first feature result and the residual is calculated to obtain the inter-individual association information.

[0091] Specifically, the result obtained in step S4.5 is added to the input feature sequence in S3.2 for residual calculation, and is added to the result obtained after regularization processing and then calculated by a multi-layer perceptron to obtain an N*128-dimensional tensor for residual calculation.

[0092] Reference Figure 2 , a pedestrian group representation system based on member progressive relationship, including:

[0093] Feature extraction module, which obtains pedestrian images and extracts pedestrian features;

[0094] The encoding module performs linear projection dimensionality reduction on pedestrian features to obtain the hidden code of pedestrian features;

[0095] The module is introduced to introduce the group representation vector into the latent code of pedestrian features and fill it in to obtain the filled feature sequence;

[0096] The individual representation module combines the group representation vector in the filled feature sequence with the individual appearance information to obtain the group individual representation;

[0097] The association representation module combines the group representation vector in the filled feature sequence with the association information between individuals to obtain the group association representation;

[0098] The fusion module fuses the group individual representation and the group association representation to obtain a multi-level group representation.

[0099] As a further preferred embodiment of the system, it also includes:

[0100] The recognition module is used to obtain the image to be tested and find similar images in the pedestrian images based on the group association representation

[0101] The contents of the above method embodiments are all applicable to the present system embodiments. The functions specifically implemented by the present system embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0102] A pedestrian group representation device based on member progressive relationship:

[0103] at least one processor;

[0104] at least one memory for storing at least one program;

[0105] When the at least one program is executed by the at least one processor, the at least one processor implements the pedestrian group characterization method based on member progressive relationship as described above.

[0106] The contents of the above method embodiments are all applicable to the present device embodiments. The functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0107] A storage medium stores processor-executable instructions, wherein the processor-executable instructions are used to implement a pedestrian group characterization method based on member progressive relationship as described above when executed by the processor.

[0108] The contents of the above method embodiments are all applicable to the present storage medium embodiments. The functions specifically implemented by the present storage medium embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0109] The above is a specific description of the preferred implementation of the present invention, but the invention is not limited to the embodiments. Those skilled in the art may make various equivalent modifications or substitutions without violating the spirit of the present invention. These equivalent modifications or substitutions are all included in the scope defined by the claims of this application.

Claims

1. A pedestrian group representation method based on member progressive relationship, characterized in that: The following steps are involved: Get pedestrian pictures and extract pedestrian features; Perform linear projection dimensionality reduction on pedestrian features to obtain the latent code of pedestrian features; The implicit encoding of pedestrian features introduces a group representation vector and fills it in to obtain a feature sequence after filling. The group representation vector in the filled feature sequence is combined with the individual appearance information to obtain the group individual representation; The group representation vector in the filled feature sequence is combined with the inter-individual association information to obtain the group association representation; The group individual representation and group association representation are integrated to obtain a multi-level group representation; The step of combining the group representation vector in the filled feature sequence with the individual appearance information to obtain the group individual representation specifically includes: The padded feature sequence is learned by three multi-layer perceptrons to obtain the corresponding features Q1, K1 and V1; Calculate the similarity of each feature with other features; The similarity is transformed into softmax and multiplied with feature V to obtain the features with high similarity between each feature and other features; The steps of perceptron learning and similarity calculation are repeated until a preset number of times is reached to obtain the similarity information of the first group of different subspaces; The similar information of the first group of different subspaces is concatenated to obtain the individual representation of the group; It also includes calculating the association information between individuals. The specific steps are as follows: Add the individual representation of the group to the filled feature sequence and calculate the residual to obtain the first feature result; Regularizing the first feature result and calculating the tensor based on a multi-layer perceptron to obtain a second feature result; The second feature result is added to the first feature result and the residual is calculated to obtain the inter-individual association information; The step of combining the group representation vector in the filled feature sequence with the inter-individual association information to obtain the group association representation specifically includes: The inter-individual correlation information is passed through three multi-layer perceptrons to obtain the corresponding feature sequences Q2, K2 and V2; Calculate the similarity between the group representation vector and other individual association information; The similarity is transformed into softmax and multiplied with the feature sequence V2 to obtain the features with high similarity to the association information between individuals in the group representation vector; The steps of perceptron learning and similarity calculation are repeated until a preset number of times is reached to obtain similar information of a second group of different subspaces; The similar information of the second group of different subspaces is concatenated to obtain the group association representation.

2. The pedestrian group characterization method based on member progressive relationship according to claim 1, characterized in that: The step of obtaining a pedestrian image and extracting pedestrian features specifically includes: Get the pedestrian images in the group and resize them to get pedestrian images of uniform size; Input pedestrian images of uniform size into the Resnet-50 network and output feature maps; The feature map is processed by average pooling to obtain pedestrian features.

3. The pedestrian group characterization method based on member progressive relationship according to claim 2, characterized in that: The step of introducing the group representation vector into the implicit encoding of the pedestrian features and filling it to obtain the filled feature sequence specifically includes: Add the group representation vector before the hidden code of pedestrian features to obtain a preliminary feature sequence; Based on the filling feature, the preliminary feature sequence is padded to a preset length to obtain a padded feature sequence; Set the pedestrian mask according to the feature interval.

4. The pedestrian group characterization method based on member progressive relationship according to claim 3, characterized in that: The step of fusing the group individual representation and the group association representation to obtain a multi-level group representation specifically includes: The group individual representation and the individual association representation are concatenated and then fused through a multi-layer perceptron to obtain a fusion result. The fusion result is regularized and then residually connected with the previous group individual representation to obtain a multi-level group representation.

5. The pedestrian group characterization method based on member progressive relationship according to claim 4, characterized in that: Also includes: Get the image to be tested and find similar images in pedestrian images based on the group association representation.

Images