Pedestrian search method and system based on full rank constraint of cross-category matrix

Abstract

The present invention provides a pedestrian search method and system based on full rank constraint of cross-category matrix, which belongs to the field of computer vision technology. The pre-trained pedestrian search model is used to process the acquired image data to be processed to identify pedestrians in the image; wherein, the training of the pre-trained pedestrian search model uses the ResNet50 network as the basic network, and the ResNet50 network is split according to the subtasks to construct a separation-reference network model, and deformable convolution is introduced to locate the entire pedestrian area. Combined with the rank-aware optimization loss, the cross-category probability matrix is ​​full-rank constrained to enhance the discriminability and diversity of inter-class features. The present invention considers the problem of feature sharing network weights of joint multi-task collaborative training, proposes a feature diversity pedestrian search framework based on the rank constraint of the cross-category probability matrix, and then introduces the deformable convolution to locate the entire pedestrian area. Finally, the rank-aware optimization loss is proposed to enhance the discriminability and diversity of inter-class features.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a pedestrian search method and system based on cross-category matrix full rank constraint. Background Art

[0002] Pedestrian search aims to locate and identify target pedestrians from a series of uncropped images, which is of great significance for applications such as criminal investigation, pedestrian behavior understanding, and pedestrian tracking. For example, it can assist security agencies in supervision and criminal investigation, quickly locate target pedestrians under a large number of video surveillance cameras, and feedback their location and portrait information, thereby greatly reducing the waste of time and resources for manual investigation. Usually, for pedestrian search tasks, you first need to know the location of pedestrians in the image (pedestrian detection), and then identify who is the correct person (pedestrian re-identification). Traditional methods usually use a two-step strategy for pedestrian search, that is, cascading pedestrian detection and pedestrian re-identification feature encoders, and training the two subtasks independently. First, the pedestrian detection model is used to predict the person's positioning box from the image, and the rectangular area of ​​the detected pedestrian is cropped according to the predicted box coordinates. Then, the feature representation of the detected pedestrian area is calculated for re-identification of pedestrians, such as Figure 1 As shown in the figure. Although the current work has made good progress, there are still two defects: On the one hand, in real video surveillance tasks, most detection models inevitably have problems such as false alarms and pedestrian misalignment (such as falsely detecting background as pedestrians, and the obtained pedestrian box cannot contain the complete pedestrian), which leads to a significant decrease in the performance of pedestrian re-identification. On the other hand, in practical applications, independent training of the two subtasks will result in two models, which has certain limitations for front-end deployment. Summary of the invention

[0003] The purpose of the present invention is to provide a pedestrian search method and system based on cross-category matrix full rank constraint to solve at least one technical problem existing in the above-mentioned background technology.

[0004] In order to achieve the above object, the present invention adopts the following technical solutions:

[0005] On the one hand, the present invention provides a pedestrian search method based on a full rank constraint of a cross-category matrix, comprising:

[0006] Acquire image data to be processed;

[0007] The acquired image data to be processed is processed using a pre-trained pedestrian search model to identify pedestrians in the image; wherein, the training of the pre-trained pedestrian search model uses the ResNet50 network as the basic network, splits the ResNet50 network according to subtasks, constructs a separation-baseline network model, introduces deformable convolution to locate the entire pedestrian area, combines rank-aware optimization loss, and performs full rank constraints on the cross-category probability matrix to enhance the discriminability and diversity of inter-class features.

[0008] Preferably, training the pedestrian search model includes:

[0009] ResNet50 is used as the backbone network; ResNet50 includes a convolution Conv1 and four convolution blocks Layer1, Layer2, Layer3, and Layer4; Conv1, Layer1, and Layer2 focus on the edge of the target, and Layer3 focuses on the detail features, namely the semantic features; the last layer of the network block, Layer4, focuses on the entire image;

[0010] Replace the convolutions in Layer1, Layer2, Layer3, and Layer4 with deformable convolutions;

[0011] Use the first half of the convolution of Conv1, Layer1, Layer2, and Layer3 of ResNet50 as the detection network;

[0012] The output of the detection network is input into the standard region proposal network layer to obtain the pedestrian candidate box;

[0013] ROI Align is used to collect pedestrian candidate boxes and calculate the feature map of the candidate boxes. The feature map is max-pooled and the merged features are input into the detection head network, which includes regression box loss and pedestrian / background classification loss.

[0014] The predicted bounding box and the features of the first half of the convolution of Layer3 are used as the input of the second half of the convolution of Layer3; finally, for N input data, P detection boxes are obtained, and the feature dimension obtained by Layer4 is N p ×2048×7×4 features, after maximum pooling and stretching, the feature dimension can be N p ×2048×1×1; finally, it is input into the person re-identification head network, which includes OIM loss and rank-aware optimization loss.

[0015] Preferably, the rank-aware optimization loss means that for each batch of images input, a cross-category probability matrix is ​​constructed, and the matrix is ​​subject to a full rank constraint, thereby obtaining a rank-aware optimization expression.

[0016] Preferably, the cross-category probability matrix is ​​obtained by the following steps:

[0017] Assume that during training, a batch of images passes through the detector to obtain P detection frames, including G pedestrian identities, and extract the D-dimensional feature vector of ReID from the P detection frames

[0018] Look up the identities of P detection boxes in the LUT lookup table. There are E identities predicted to be wrong. The corresponding features of these E identities in the LUT table are

[0019] Combine the G pedestrian identity features and the E identity features in the LUT table, that is, M identity features, M = E + G, into a cross-category probability matrix Α M×D ; Since the pedestrians in the matrix are from the same scene and wear similar clothes, in order to avoid different pedestrians in the same scene learning similar features, constraint Α M×D The rank of the matrix M×D ) is the number of M×D The maximum number of linearly independent row vectors of a matrix, i.e., rank (Α M×D )=M.

[0020] Preferably, the A M×D The rank of the matrix is ​​determined by first obtaining the singular values ​​using singular value decomposition:

[0021] Cross-class probability matrix Α M×D , rank r∈[0,min(M,D)], then the SVD decomposition of Α is:

[0022] Α=U∑V T ;

[0023] Among them, the orthogonal matrix The corresponding column vector u i ,i=1,...,m; orthogonal matrix The corresponding column vector v j ,j=1,...,d; the singular value matrix ∑ is an M×D matrix, corresponding to the element ∑ ii =σ i ≥0,∑ ij =0,i≠j;

[0024] The singular values ​​can constitute approximate functions of different ranks, where the γ norm of the matrix ΑM×D is defined as:

[0025] Among them, δ i It means to find the i-th singular value of matrix A.

[0026] Preferably, the rank of the matrix is ​​obtained by the gamma norm, and the value of the rank function is constrained to be close to M, thereby ensuring that the features of different pedestrians are independent of each other, so the inter-class loss is obtained as follows:

[0027]

[0028] In a second aspect, the present invention provides a pedestrian search system based on a cross-category matrix full rank constraint, comprising:

[0029] An acquisition module, used for acquiring image data to be processed;

[0030] A search module is used to process the acquired image data to be processed using a pre-trained pedestrian search model to identify pedestrians in the image; wherein, the training of the pre-trained pedestrian search model uses the ResNet50 network as the basic network, splits the ResNet50 network according to subtasks, constructs a separation-baseline network model, introduces deformable convolution to locate the entire pedestrian area, combines rank-aware optimization loss, and performs full rank constraints on the cross-category probability matrix to enhance the discriminability and diversity of inter-class features.

[0031] In a third aspect, the present invention provides a non-transitory computer-readable storage medium, which is used to store computer instructions. When the computer instructions are executed by a processor, the pedestrian search method based on the full rank constraint of the cross-category matrix as described above is implemented.

[0032] In a fourth aspect, the present invention provides a computer program product, comprising a computer program, which, when running on one or more processors, is used to implement the pedestrian search method based on the full rank constraint of the cross-category matrix as described above.

[0033] In a fifth aspect, the present invention provides an electronic device, comprising: a processor, a memory and a computer program; wherein the processor is connected to the memory, the computer program is stored in the memory, and when the electronic device is running, the processor executes the computer program stored in the memory so that the electronic device executes instructions for implementing the pedestrian search method based on the full rank constraint of the cross-category matrix as described above.

[0034] Terminology explanation:

[0035] Person Search (PS): aims to locate and identify target pedestrians from a series of uncropped images. It is of great significance for applications such as criminal investigation, pedestrian behavior understanding, and pedestrian tracking. It includes two subtasks: pedestrian detection and pedestrian re-identification.

[0036] Pedestrian detection: Get the location coordinate frame of pedestrians in a given image.

[0037] Pedestrian re-identification (Re-ID): Match the target pedestrian with the detected pedestrian frames under different non-overlapping cameras.

[0038] Deformable Convolution Net (DCN): A learnable offset is introduced in the receptive field to ensure that the receptive field is not a common rectangle, and the convolution is close to the actual shape of the object, so that the convolution area can better cover the shape of the object.

[0039] The beneficial effects of the present invention are as follows: considering the problem of feature sharing network weights in joint multi-task collaborative training, a Feature Diversity Person Search (FDPS) framework based on the rank constraint of a cross-category probability matrix is ​​proposed, and then deformable convolution is introduced to locate the entire pedestrian area. Finally, a Rank Perception Optimization (RPO) loss is proposed to enhance the discriminability and diversity of inter-class features.

[0040] Additional advantages of the present invention will be more clearly given in the following description or learned through the practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0042] Figure 1 A flowchart of the traditional two-step strategy for pedestrian search.

[0043] Figure 2 This is the flow chart of the existing OIM algorithm framework.

[0044] Figure 3 This is a one-step step-by-step training framework diagram for pedestrian search according to an embodiment of the present invention.

[0045] Figure 4 This is a diagram of the ResNet50 network layer learning structure described in an embodiment of the present invention.

[0046] Figure 5 This is a diagram of a pedestrian search framework based on cross-category full rank constraints according to an embodiment of the present invention.

[0047] Figure 6A schematic diagram of the full rank constraint of the cross-category probability matrix described in an embodiment of the present invention.

[0048] Figure 7 A schematic diagram of rank approximation of different paradigms involved in an embodiment of the present invention.

[0049] Figure 8 This is a schematic diagram of the Top-10 results of pedestrian search in TV scene data in the CUHK-SYSU dataset according to an embodiment of the present invention. DETAILED DESCRIPTION

[0050] The embodiments of the present invention are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements with the same or similar functions. The embodiments described below by the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be interpreted as limiting the present invention.

[0051] It should be understood by those skilled in the art that unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by those skilled in the art to which this invention belongs.

[0052] It should also be understood that terms, such as those defined in commonly used dictionaries, should be understood to have a meaning consistent with that in the context of the prior art and will not be interpreted in an idealized or overly formal sense unless as defined herein.

[0053] Those skilled in the art will appreciate that, unless otherwise stated, the singular forms "a", "an", "said" and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of the present invention refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements and / or groups thereof.

[0054] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine different embodiments or examples described in this specification and the features of different embodiments or examples, unless they are contradictory.

[0055] To facilitate understanding of the present invention, the present invention is further explained below with reference to specific embodiments in conjunction with the accompanying drawings, and the specific embodiments do not constitute a limitation on the embodiments of the present invention.

[0056] Those skilled in the art should understand that the drawings are merely schematic diagrams of embodiments, and the components in the drawings are not necessarily necessary for implementing the present invention.

[0057] Example 1

[0058] This embodiment 1 provides a pedestrian search system based on a cross-category matrix full rank constraint, the system comprising:

[0059] An acquisition module, used for acquiring image data to be processed;

[0060] A search module is used to process the acquired image data to be processed using a pre-trained pedestrian search model to identify pedestrians in the image; wherein, the training of the pre-trained pedestrian search model uses the ResNet50 network as the basic network, splits the ResNet50 network according to subtasks, constructs a separation-baseline network model, introduces deformable convolution to locate the entire pedestrian area, combines rank-aware optimization loss, and performs full rank constraints on the cross-category probability matrix to enhance the discriminability and diversity of inter-class features.

[0061] In this embodiment 1, the above system is used to implement a pedestrian search method based on a cross-category matrix full rank constraint, which includes:

[0062] Using an acquisition module to acquire image data to be processed;

[0063] The search module is used to process the acquired image data to be processed using a pre-trained pedestrian search model to identify pedestrians in the image; wherein, the training of the pre-trained pedestrian search model uses the ResNet50 network as the basic network, splits the ResNet50 network according to subtasks, constructs a separation-baseline network model, introduces deformable convolution to locate the entire pedestrian area, combines rank-aware optimization loss, and performs full rank constraints on the cross-category probability matrix to enhance the discriminability and diversity of inter-class features.

[0064] Wherein, training the pedestrian search model includes:

[0065] ResNet50 is used as the backbone network; ResNet50 includes a convolution Conv1 and four convolution blocks Layer1, Layer2, Layer3, and Layer4; Conv1, Layer1, and Layer2 focus on the edge of the target, and Layer3 focuses on the detail features, namely the semantic features; the last layer of the network block, Layer4, focuses on the entire image;

[0066] Replace the convolutions in Layer1, Layer2, Layer3, and Layer4 with deformable convolutions;

[0067] Use the first half of the convolution of Conv1, Layer1, Layer2, and Layer3 of ResNet50 as the detection network;

[0068] The output of the detection network is input into the standard region proposal network layer to obtain the pedestrian candidate box;

[0069] ROI Align is used to collect pedestrian candidate boxes and calculate the feature map of the candidate boxes. The feature map is max-pooled and the merged features are input into the detection head network, which includes regression box loss and pedestrian / background classification loss.

[0070] The predicted bounding box and the features of the first half of the convolution of Layer3 are used as the input of the second half of the convolution of Layer3; finally, for N input data, P detection boxes are obtained, and the feature dimension obtained by Layer4 is N p ×2048×7×4 features, after maximum pooling and stretching, the feature dimension can be N p ×2048×1×1; finally, it is input to the pedestrian re-identification head network, which includes OIM loss and rank-aware optimization loss.

[0071] Among them, the rank-aware optimization loss means that for each batch of images input, a cross-category probability matrix is ​​constructed, and the matrix is ​​subject to full rank constraints to obtain the rank-aware optimization expression.

[0072] The cross-category probability matrix is ​​obtained by the following steps:

[0073] Assume that during training, a batch of images passes through the detector to obtain P detection frames, including G pedestrian identities, and extract the D-dimensional feature vector of ReID from the P detection frames

[0074] Look up the identities of P detection boxes in the LUT lookup table. There are E identities predicted to be wrong. The corresponding features of these E identities in the LUT table are

[0075] Combine the G pedestrian identity features and the E identity features in the LUT table, that is, M identity features, M = E + G, into a cross-category probability matrix Α M×D ; Since the pedestrians in the matrix are from the same scene and wear similar clothes, in order to avoid different pedestrians in the same scene learning similar features, constraint Α M×D The rank of the matrix M×D) is the number of M×D The maximum number of linearly independent row vectors of a matrix, i.e., rank (Α M×D )=M.

[0076] The A M×D The rank of the matrix is ​​determined by first obtaining the singular values ​​using singular value decomposition:

[0077] Cross-class probability matrix Α M×D , rank r∈[0,min(M,D)], then the SVD decomposition of Α is:

[0078] Α=U∑V T ;

[0079] Among them, the orthogonal matrix The corresponding column vector u i ,i=1,...,m; orthogonal matrix The corresponding column vector v j ,j=1,...,d; the singular value matrix ∑ is an M×D matrix, corresponding to the element ∑ ii =σ i ≥0,∑ ij =0,i≠j;σ i represents the i-th singular value.

[0080] The singular values ​​constitute different rank functions, matrix Α M×D The γ norm of is defined as:

[0081] Among them, δ i It means to find the i-th singular value of matrix A.

[0082] The rank of the matrix is ​​obtained by the gamma norm. It is necessary to constrain the value of the rank function to be close to M. Since each row of the matrix corresponds to the characteristics of the pedestrian, the inter-class loss is:

[0083]

[0084] In this embodiment 1, the two subtasks are integrated into a network structure framework for mutual adaptation training. Pedestrian detection focuses on finding the commonalities of pedestrians of all identities, while pedestrian re-identification focuses on the differences between multiple identities. The features extracted by the model with shared network weights will reduce the accuracy of pedestrian re-identification. On the one hand, this is because the network training is not easy to converge, and on the other hand, it is because the learned features lack diversity.

[0085] Example 2

[0086] In order to narrow the gap between traditional pedestrian re-identification and real-world applications, a framework for jointly optimizing pedestrian detection and re-identification in a one-step strategy has been proposed, such as the OIM (Online Instance Matching) algorithm. Figure 2 As shown in the figure, ResNet-50 is used as the basic CNN model. ResNet50 includes a convolution (i.e. Conv1) and four convolution blocks (Layer1, Layer2, Layer3, Layer4). In the pedestrian search algorithm framework, the input image passes through Conv1, Layer1, Layer2, and the first half of Layer3, and the output is a feature map of 1024 channels, whose resolution is 1 / 16 of the original image, followed by the RPN layer and the ROI Pooling layer. The pedestrian proposal box is input into the remaining network layers of ResNet-50, and the output is a 2048-dimensional feature. After obtaining the 2048-dimensional feature, one branch is connected to the classification and regression loss of the detection. The other branch maps the 2048-dimensional feature into a 256-dimensional feature, followed by the pedestrian identity classification loss. For pedestrian identity prediction, OIM loss is used to form a lookup table (LUT) for all labeled pedestrian features. Unlabeled pedestrian features can be regarded as negative samples and stored in a circular queue. By inputting each sample of a batch and comparing the distance with the samples with known labels in the lookup table, the category of the pedestrian feature can be obtained. Compared with SoftMax loss, the parameter-free OIM loss function converges faster.

[0087] Based on the above OIM algorithm, in this embodiment 2, if Figures 3 to 8 As shown in the figure, a feature diversity person search (FDPS) framework based on cross-class matrix rank constraint is proposed to alleviate the impact of the conflicting purposes of pedestrian detection and pedestrian re-identification on the pedestrian search model. First, we split the ResNet50 network according to the subtasks, constructed a split-baseline (S-bsl) model, and then introduced deformable convolution to locate the entire pedestrian area. Finally, a rank-aware optimization loss is proposed to constrain the cross-class probability matrix to full rank to enhance the discriminability and diversity of inter-class features.

[0088] The pedestrian search model proposed in this embodiment can be trained and tested on a single GTX1080Ti GPU.

[0089] In this embodiment 2, a feature diversity person search (FDPS) framework based on cross-class matrix rank constraints is designed to alleviate the impact of the conflicting objectives of the two subtasks of pedestrian detection and pedestrian re-identification on the pedestrian search model. The processing flow chart of this method is shown in Figure 5 As shown, it specifically includes the following steps:

[0090] Step S1: ResNet50 is used as the backbone network. ResNet50 includes one convolution (i.e. Conv1) and four convolution blocks Layer1, Layer2, Layer3, Layer4 (L1, L2, L3, L4). The shallow network (Conv1, Layer1, Layer2) mainly focuses on the edge of the object. As the network goes deeper, the third layer focuses on the detail features (i.e. semantic features). The last layer of the network block focuses on the entire image, including some background information, such as Figure 4 shown.

[0091] Step S2: Replace the convolutions in the L1, L2, L3, and L4 layers with deformable convolutions, such as Figure 5 shown.

[0092] Step S3: Use the first half of the convolution of Conv1, L1, L2, and L3 of ResNet50 as the detection network.

[0093] Step S4: The output of step S3 is input to the standard Region Proposal Network (RPN) layer to obtain the pedestrian candidate box.

[0094] Step S5: Use ROI Align to collect pedestrian candidate boxes and calculate the feature map of the candidate boxes. Finally, we perform a maximum pooling operation on the feature map. The merged features are input to the Detection Head (DetH) network, which includes regression box loss and pedestrian or background classification loss.

[0095] Step S6: The predicted bounding box obtained in step S1-3 and the features of L3-1 (i.e., the first half of the convolution of L3) are used as the input of L3-2 (i.e., the second half of the convolution of L3). Finally, for N input data, P detection boxes are obtained, and the feature dimension obtained by L4 is N p ×2048×7×4 features, and then through maximum pooling and stretching, the feature dimension can be N p ×2048×1×1. Finally, it is input to the person re-identification head network (ReID Head, ReIDH), which includes OIM loss and rank-aware optimization loss.

[0096] Step S6-1: The rank-aware optimization loss mentioned in step S6 means that for each batch of images input, we will construct a cross-category probability matrix and impose a full rank constraint on this matrix to obtain a rank-aware optimization expression.

[0097] Step S6-2: The cross-category probability matrix mentioned in step S6-1 is obtained by the following steps:

[0098] Step S6-2-1: Assume that a batch of images in training pass through the detector to obtain P detection frames, including G pedestrian identities, and extract the D-dimensional feature vector of ReID from the P detection frames

[0099] Step S6-2-2: Look up the identities of P detection boxes in the LUT lookup table. There are E identities predicted to be wrong. The corresponding features of these E identities in the LUT table are

[0100] Step S6-2-3: Then we can combine the G pedestrian identity features and the E identity features in the LUT table (ie, M identity features, M = E + G) into a cross-category probability matrix Α M×D .

[0101] Step S6-3: Since different pedestrians in the matrix come from the same scene and wear similar clothes, in order to avoid different pedestrians in the same scene learning similar features, we constrain Α M×D The rank of the matrix M×D ) is the number of M×D The maximum number of linearly independent row vectors of a matrix, i.e., rank (Α M×D )=M.

[0102] Step S6-4: The rank of the matrix mentioned in step S6-3 can be determined by first obtaining singular values ​​using singular value decomposition (SVD):

[0103] Step S6-4-1: Cross-class probability matrix A M×D , rank r∈[0,min(M,D)], then the SVD decomposition of Α is:

[0104] Α=U∑V T (1)

[0105] Among them, the orthogonal matrix The corresponding column vector u i ,i=1,...,m; orthogonal matrix The corresponding column vector v j ,j=1,...,d; the singular value matrix ∑ is an M×D matrix, corresponding to the element ∑ii =σ i ≥0,∑ ij =0,i≠j.

[0106] Step S6-4-2: The singular values ​​mentioned in step S6-4-1 can form different rank functions. 7 shows a schematic diagram of different rank functions approximating the rank. It can be seen that the γ norm can better approximate the rank of the matrix. Matrix Α M×D The γ norm of is defined as follows:

[0107]

[0108] Step S6-5: The rank of the matrix is ​​obtained by the γ norm mentioned in step S6-4-2. According to step S6-3, the value of the rank function that needs to be constrained is close to M. Since each row of the matrix corresponds to the characteristics of the pedestrian, the inter-class loss is as follows:

[0109]

[0110] Step S6-6: The parameter γ mentioned in step S6-5 can be set to 1.

[0111] Step S6-7: Based on the features of different classes, we hope that the features obtained are discriminative and can classify different pedestrians. M×D Each category corresponds to a row of features, and the discriminative power of the features can be determined by the rank of the constraint matrix, that is, the matrix Α M×D The rows are linearly independent.

[0112] In summary, the feature diversity person search (FDPS) framework based on cross-class matrix rank constraints described in this embodiment 2 considers the problem of feature sharing network structure using joint multi-task collaborative training, and proposes a feature diversity pedestrian search framework based on cross-class probability matrix rank constraints. The Re-ID task is the main task, ResNet50 is used as the backbone network, and the shallow network of ResNet50 is used as the main network for pedestrian detection to determine the pedestrian candidate frame. The deep network only focuses on the pedestrian re-identification task to extract discriminative pedestrian features. In addition, for the problem of incomplete pedestrian detection frame, deformable convolution is used instead of ordinary convolution to better locate the entire pedestrian. Deformable convolution can achieve dense geometric deformation receptive field, thereby ensuring that the prediction frame can surround the entire pedestrian and provide richer feature information for ReID. Finally, in order to increase the diversity of pedestrian features, a rank-aware optimization (RPO) loss function is proposed. For each batch of images, the pedestrian category probability of each prediction frame is obtained through the pedestrian search network. The probabilities of different pedestrian categories are combined into a cross-category probability matrix, and a full rank constraint is imposed on this matrix to make the learned features more distinctive and diverse.

[0113] Example 3

[0114] Embodiment 3 of the present invention provides a non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium is used to store computer instructions. When the computer instructions are executed by a processor, a pedestrian search method based on a cross-category matrix full rank constraint is implemented. The method includes:

[0115] Acquire image data to be processed;

[0116] The acquired image data to be processed is processed using a pre-trained pedestrian search model to identify pedestrians in the image; wherein, the training of the pre-trained pedestrian search model uses the ResNet50 network as the basic network, splits the ResNet50 network according to subtasks, constructs a separation-baseline network model, introduces deformable convolution to locate the entire pedestrian area, combines rank-aware optimization loss, and performs full rank constraints on the cross-category probability matrix to enhance the discriminability and diversity of inter-class features.

[0117] Example 4

[0118] Embodiment 4 of the present invention provides a computer program (product), including a computer program, wherein when the computer program is executed on one or more processors, the method is used to implement a pedestrian search method based on a cross-category matrix full rank constraint, the method comprising:

[0119] Acquire image data to be processed;

[0120] The acquired image data to be processed is processed using a pre-trained pedestrian search model to identify pedestrians in the image; wherein, the training of the pre-trained pedestrian search model uses the ResNet50 network as the basic network, splits the ResNet50 network according to subtasks, constructs a separation-baseline network model, introduces deformable convolution to locate the entire pedestrian area, combines rank-aware optimization loss, and performs full rank constraints on the cross-category probability matrix to enhance the discriminability and diversity of inter-class features.

[0121] Example 5

[0122] Embodiment 5 of the present invention provides an electronic device, including: a processor, a memory, and a computer program; wherein the processor is connected to the memory, the computer program is stored in the memory, and when the electronic device is running, the processor executes the computer program stored in the memory, so that the electronic device executes instructions for implementing a pedestrian search method based on a full rank constraint of a cross-category matrix, the method comprising:

[0123] Acquire image data to be processed;

[0124] The acquired image data to be processed is processed using a pre-trained pedestrian search model to identify pedestrians in the image; wherein, the training of the pre-trained pedestrian search model uses the ResNet50 network as the basic network, splits the ResNet50 network according to subtasks, constructs a separation-baseline network model, introduces deformable convolution to locate the entire pedestrian area, combines rank-aware optimization loss, and performs full rank constraints on the cross-category probability matrix to enhance the discriminability and diversity of inter-class features.

[0125] In summary, the pedestrian search method based on full rank constraint of cross-category matrix described in the embodiment of the present invention considers the problem of feature sharing network weights of joint multi-task collaborative training, proposes a feature diversity pedestrian search (FDPS) framework based on cross-category probability matrix rank constraint, and then introduces deformable convolution to locate the entire pedestrian area. Finally, rank perception optimization (RPO) loss is proposed to enhance the discriminability and diversity of inter-class features. It can be applied to criminal investigation, pedestrian behavior understanding, pedestrian tracking, etc., for example, to assist security agencies in supervision and criminal investigation, quickly locate target pedestrians under a large number of video surveillance cameras, and feedback their location and portrait information, thereby greatly reducing the waste of time and resources for manual investigation.

[0126] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0127] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0128] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0129] These computer program instructions can also be loaded onto a computer or other programmable data processing device, and a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0130] Although the above describes the specific implementation mode of the present invention in conjunction with the accompanying drawings, it is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative work on the basis of the technical solution disclosed in the present invention should be included in the scope of protection of the present invention.

Claims

1. A pedestrian search method based on full rank constraint of cross-category matrix, characterized in that: include: Acquire image data to be processed; The obtained image data to be processed is processed using a pre-trained pedestrian search model to identify pedestrians in the image; wherein the training of the pre-trained pedestrian search model uses a ResNet50 network as a basic network, splits the ResNet50 network according to subtasks, constructs a separation-baseline network model, introduces deformable convolution to locate the entire pedestrian area, combines rank-aware optimization loss, and imposes full rank constraints on the cross-category probability matrix to enhance the discriminability and diversity of inter-class features; Training the pedestrian search model includes: ResNet50 is used as the backbone network; ResNet50 includes a convolution Conv1 and four convolution blocks Layer1, Layer2, Layer3, and Layer4; Conv1, Layer1, and Layer2 focus on the edge of the target, and Layer3 focuses on the detail features, namely the semantic features; the last layer of the network block, Layer4, focuses on the entire image; Replace the convolutions in Layer1, Layer2, Layer3, and Layer4 with deformable convolutions; Use the first half of the convolution of Conv1, Layer1, Layer2, and Layer3 of ResNet50 as the detection network; The output of the detection network is input into the standard region proposal network layer to obtain the pedestrian candidate box; ROI Align is used to collect pedestrian candidate boxes and calculate the feature map of the candidate boxes. The feature map is max-pooled and the merged features are input into the detection head network, which includes regression box loss and pedestrian / background classification loss. The predicted bounding box and the features of the first half of the convolution of Layer3 are used as the input of the second half of the convolution of Layer3; finally, for N input data, P detection boxes are obtained, and the feature dimension obtained by Layer4 is N p ×2048×7×4 features, after maximum pooling and stretching, the feature dimension can be N p ×2048×1×1; finally input to the pedestrian re-identification head network, which includes OIM loss and rank-aware optimization loss; The rank-aware optimization loss means that for each batch of images input, a cross-category probability matrix is ​​constructed and the matrix is ​​constrained to be full rank, thereby obtaining a rank-aware optimization expression; The cross-category probability matrix is ​​obtained by the following steps: Assume that during training, a batch of images passes through the detector to obtain P detection frames, including G pedestrian identities, and extract the D-dimensional feature vector of ReID from the P detection frames Look up the identities of P detection boxes in the LUT lookup table. There are E identities predicted to be wrong. The corresponding features of these E identities in the LUT table are Combine the G pedestrian identity features and the E identity features in the LUT table, that is, M identity features, M = E + G, into a cross-category probability matrix Α M×D ; Since the pedestrians in the matrix are from the same scene and wear similar clothes, in order to avoid different pedestrians in the same scene learning similar features, constraint Α M×D The rank of the matrix M×D ) is the number of M×D The maximum number of linearly independent row vectors of a matrix, i.e., rank (Α M×D ) = M; The A M×D The rank of the matrix is ​​determined by first obtaining the singular values ​​using singular value decomposition: Cross-class probability matrix Α M×D , rank(A M×D )∈[0,min(M,D)], the SVD decomposition form of Α is: A=U∑V T ; Among them, the orthogonal matrix The corresponding column vector u i ,i=1,...,m; orthogonal matrix The corresponding column vector v j ,j=1,…,d; the singular value matrix ∑ is an M×D matrix, corresponding to the element ∑ ii =σ i ≥0,∑ ij =0,i≠j;σ i represents the i-th singular value; The singular values ​​constitute different rank functions, matrix Α M×D The γ norm of is defined as: Among them, δ i Indicates that the i-th singular value of matrix A is obtained; The rank of the matrix is ​​obtained by the gamma norm. It is necessary to constrain the value of the rank function to be close to M. Since each row of the matrix corresponds to the characteristics of the pedestrian, the inter-class loss is:

2. A pedestrian search system based on full rank constraints of cross-category matrices, characterized in that: include: An acquisition module, used for acquiring image data to be processed; A search module is used to process the acquired image data to be processed using a pre-trained pedestrian search model to identify pedestrians in the image; wherein the training of the pre-trained pedestrian search model uses a ResNet50 network as a basic network, splits the ResNet50 network according to subtasks, constructs a separation-baseline network model, introduces deformable convolution to locate the entire pedestrian area, combines rank-aware optimization loss, and imposes full rank constraints on the cross-category probability matrix to enhance the discriminability and diversity of inter-class features; Wherein, training the pedestrian search model includes: ResNet50 is used as the backbone network; ResNet50 includes a convolution Conv1 and four convolution blocks Layer1, Layer2, Layer3, and Layer4; Conv1, Layer1, and Layer2 focus on the edge of the target, and Layer3 focuses on the detail features, namely the semantic features; the last layer of the network block, Layer4, focuses on the entire image; Replace the convolutions in Layer1, Layer2, Layer3, and Layer4 with deformable convolutions; Use the first half of the convolution of Conv1, Layer1, Layer2, and Layer3 of ResNet50 as the detection network; The output of the detection network is input into the standard region proposal network layer to obtain the pedestrian candidate box; ROI Align is used to collect pedestrian candidate boxes and calculate the feature map of the candidate boxes. The feature map is max-pooled and the merged features are input into the detection head network, which includes regression box loss and pedestrian / background classification loss. The predicted bounding box and the features of the first half of the convolution of Layer3 are used as the input of the second half of the convolution of Layer3; finally, for N input data, P detection boxes are obtained, and the feature dimension obtained by Layer4 is N p ×2048×7×4 features, after maximum pooling and stretching, the feature dimension can be N p ×2048×1×1; finally input to the pedestrian re-identification head network, which includes OIM loss and rank-aware optimization loss; The rank-aware optimization loss means that for each batch of images input, a cross-category probability matrix is ​​constructed and the matrix is ​​constrained to be full rank, thereby obtaining a rank-aware optimization expression; The cross-category probability matrix is ​​obtained by the following steps: Assume that during training, a batch of images passes through the detector to obtain P detection frames, including G pedestrian identities, and extract the D-dimensional feature vector of ReID from the P detection frames Look up the identities of P detection boxes in the LUT lookup table. There are E identities predicted to be wrong. The corresponding features of these E identities in the LUT table are Combine the G pedestrian identity features and the E identity features in the LUT table, that is, M identity features, M = E + G, into a cross-category probability matrix Α M×D ; Since the pedestrians in the matrix are from the same scene and wear similar clothes, in order to avoid different pedestrians in the same scene learning similar features, constraint Α M×D The rank of the matrix M×D ) is the number of M×D The maximum number of linearly independent row vectors of a matrix, i.e., rank (Α M×D ) = M; The A M×D The rank of the matrix is ​​determined by first obtaining the singular values ​​using singular value decomposition: Cross-class probability matrix Α M×D , rank(Α M×D )∈[0,min(M,D)], the SVD decomposition form of Α is: A=U∑V T ; Among them, the orthogonal matrix The corresponding column vector u i ,i=1,...,m; orthogonal matrix The corresponding column vector v j ,j=1,…,d; the singular value matrix ∑ is an M×D matrix, corresponding to the element ∑ ii =σ i ≥0,∑ ij =0,i≠j;σ i represents the i-th singular value; The singular values ​​constitute different rank functions, matrix Α M×D The γ norm of is defined as: Among them, δ i Indicates that the i-th singular value of matrix A is obtained; The rank of the matrix is ​​obtained by the gamma norm. It is necessary to constrain the value of the rank function to be close to M. Since each row of the matrix corresponds to the characteristics of the pedestrian, the inter-class loss is:

3. A non-transitory computer-readable storage medium, characterized in that: The non-transitory computer-readable storage medium is used to store computer instructions, and when the computer instructions are executed by the processor, the pedestrian search method based on the cross-category matrix full rank constraint as described in claim 1 is implemented.

4. A computer program product, characterized in that The invention comprises a computer program, which is used to implement the pedestrian search method based on cross-category matrix full rank constraint as claimed in claim 1 when running on one or more processors.

5. An electronic device, characterized in that: include: A processor, a memory and a computer program; wherein the processor is connected to the memory, the computer program is stored in the memory, and when the electronic device is running, the processor executes the computer program stored in the memory to enable the electronic device to execute instructions for implementing the pedestrian search method based on the full rank constraint of the cross-category matrix as described in claim 1.

Images