A pedestrian search method and system based on combining human semantics and three-dimensional reconstruction

Abstract

The present invention belongs to the field of computer vision technology, and provides a pedestrian search method and system based on combining human semantics and three-dimensional reconstruction. The steps are as follows: first, obtain the pedestrian image to be retrieved, input the image into the trained clothing-changing pedestrian feature extraction network, extract the pedestrian features in the clothing-changing situation, and expand the data set samples through the clothing-changing module of the clothing-changing pedestrian re-feature extraction network, so that the model focuses on learning the pedestrian feature representation outside the clothing area, reducing the interference caused by clothing changes, and performing three-dimensional modeling on the pedestrian image, retaining only the pedestrian head, learning additional pedestrian body shape and head features, and finally performing feature fusion of the original image features and the three-dimensional modeling image features to obtain global pedestrian features, using the obtained pedestrian features of the image to be retrieved and the pedestrian features in the retrieval library for similarity matching, and sorting according to the similarity score to obtain pedestrian retrieval results. The present invention can greatly improve the accuracy and robustness of pedestrian search.

Description

Technical Field

[0001] The invention relates to a pedestrian search method and system based on combining human semantics with three-dimensional reconstruction, belonging to the technical field of computer vision. Background Art

[0002] With the acceleration of urbanization and population growth, pedestrian search has become increasingly important in urban management and public safety. The complexity of the urban environment and the surge in pedestrian traffic have brought many challenges to traditional monitoring and search methods. Through pedestrian recognition and search technology, pedestrian images or video sequences from different monitoring devices can be effectively retrieved to achieve cross-camera pedestrian tracking, which significantly improves the efficiency and accuracy of the intelligent monitoring system. Therefore, this technology has great application prospects in the fields of security monitoring, smart retail and smart cities, and has important research value and practical significance.

[0003] In addition, the rapid development of artificial intelligence and deep learning technologies has brought new opportunities to the field of pedestrian search. The advanced algorithms and powerful computing power of deep learning provide a solid theoretical foundation and technical support for pedestrian search. With the help of deep learning models such as convolutional neural networks (CNN), pedestrian search algorithms can more effectively extract features from pedestrian images and significantly improve the accuracy and robustness of recognition.

[0004] However, it is worth noting that traditional pedestrian search methods usually assume that pedestrians appear repeatedly in a closed scene within a short period of time and that their visual appearance is roughly the same, which greatly reduces the difficulty of the search. This does not match the long time span in real open scenes and is therefore difficult to use effectively in practical applications.

[0005] In the real world, pedestrians may change their clothes over a long period of time and appear again in the surveillance footage. More importantly, suspects may disguise themselves by changing their appearance or covering their faces, which makes clothing features no longer an effective basis for pedestrian identification. Therefore, clothing-changing pedestrian search has emerged. The goal of clothing-changing search is to establish the identity connection between the same pedestrian wearing different clothes under the surveillance camera over a long period of time. Due to the change of clothes, the appearance of pedestrians is obviously different, which greatly increases the difficulty of search.

[0006] Therefore, the present invention proposes a pedestrian search method and system based on combining human semantics and three-dimensional reconstruction to solve the above problems. Summary of the invention

[0007] In view of the shortcomings of the existing technology, the present invention develops a pedestrian search method and system based on combining human semantics and three-dimensional reconstruction. By simulating the process of changing clothes to expand the training data samples, comparatively learning the features of non-clothing areas, and introducing additional body shape and head features, the model can more effectively cope with the challenges brought by the problem of changing clothes, and greatly improve the accuracy and robustness of pedestrian search for changing clothes.

[0008] The technical solution to the technical problem of the present invention is a pedestrian search method based on combining human semantics and three-dimensional reconstruction, comprising the following steps:

[0009] S1. Load pedestrian data and obtain human semantics:

[0010] A data set is constructed from the original pedestrian data obtained, and the original pedestrian data in the data set is randomly stratified and sampled. The original pedestrian images to be processed are obtained from the data set. The original pedestrian images are annotated with the semantic information of various parts of the human body through the pre-trained human semantic parsing model to obtain the corresponding human semantic mask.

[0011] S2. Human semantic clothing change, data sample expansion:

[0012] Use the human body semantic mask to change the clothes of the same pedestrian in two images with different clothes, and get the pedestrian image after changing clothes, so as to expand the data set samples;

[0013] S3. Comparative learning features, focusing on human body areas:

[0014] The backbone network is used to extract the features of the original pedestrian image and the pedestrian image after changing clothes. The mean square error loss is used to compare the changes in the features before and after changing clothes, making the two features closer.

[0015] S4. 3D human body reconstruction, head information preserved:

[0016] Firstly, the gender information of pedestrians is extracted by using the human attribute analysis model. The human mesh vertices are located in the 3D space by using the pre-trained 3D human mesh estimation model. The SMPL-X model of the corresponding gender is used to build a 3D model of pedestrians. Then, the 3D model is projected onto the background of the original pedestrian image. The head of the original pedestrian image is pasted to the corresponding head position using the human semantic head mask. Finally, the backbone network is used to extract the 3D modeling image features that retain the head.

[0017] S5. Pedestrian feature fusion and global feature matching:

[0018] The features of the extracted original pedestrian image are fused with the features of the 3D modeling image through convolutional attention to obtain the final global features. The global features are used for feature similarity matching. The similarity between features is evaluated by calculating the Euclidean distance, and the pedestrian retrieval gallery is sorted to obtain the final retrieval results.

[0019] The specific steps of S1 are as follows:

[0020] Small batch samples A batch of data after random stratified sampling of the data set, , small batch samples Included Groups of images of pedestrians with different identities, each group of pedestrians with the same identity contains Original pedestrian images with different clothing, , represents the total number of original pedestrian images in a mini-batch sample, Represents a small batch of samples Middle Original pedestrian image, The shape is , , and Respectively represent the number of channels, height and width of the original pedestrian image; and annotate the true identity of each original pedestrian image in the small batch sample. The real identity label is , ;

[0021] Use the pre-trained human semantic parsing model to analyze any original pedestrian image in any mini-batch sample in the dataset. Perform human semantic segmentation to obtain the local semantic confidence of human pixels , Indicates that it belongs to the human body semantics Some pixel coordinates are The semantic confidence of , The six semantic parts of the human body are annotated as background, head, upper clothes, lower clothes, upper limbs, and lower limbs. The activation function processes the local semantic confidence of human pixels to obtain 6 local semantic mask pixel coordinates , Indicates The pixel coordinates of the semantic mask of the semantic part of the individual body are then The function that obtains the human body semantic mask by calculating the function parameter ;

[0022] in, The specific implementation of the function is as follows:

[0023] , ,

[0024] The specific implementation of the function is as follows:

[0025] .

[0026] The specific steps of S2 are as follows:

[0027] Select the original pedestrian image Images of the same person wearing different clothes Comes from the original pedestrian image Perform the clothing change operation, specifically through the original pedestrian image Human semantic mask and images Human semantic mask Find the overlapping area of ​​the clothing masks of the two images and replace the pixels. The unreplaced clothing area is filled with the clothes of the other image. The process of obtaining the clothes of the other image is in the image A pixel block with a size of 1 / 5 of the clothing area is taken from the 1 / 4 position of the clothing area and converted into shape;

[0028] The replacement process is as follows:

[0029] ,

[0030] ,

[0031] in, Indicates the overlapping area of ​​clothing mask, Indicates the area of ​​clothing that has not been replaced, Indicates the clothing mask value, , Indicates the mask value of the top area. Indicates the mask value of the top area;

[0032] After replacement, we get a pedestrian image with changed clothes. Then select the original pedestrian image The new images are obtained by replacing other images of the same pedestrian in different clothing. Similarly, all the original pedestrian images in the data set are replaced by images of the same pedestrian in different clothing, thereby expanding the number of samples in the data set.

[0033] The specific steps of S3 are as follows:

[0034] All original pedestrian images and the pedestrian images after changing clothes are concatenated on the small batch samples to obtain a shape of Data samples, get small batch samples after data expansion , Represents a small batch of samples Middle Pedestrian images, , Represents a small batch of samples The number of pedestrian images in the image is calculated, and then the ResNet-50 backbone network is used for feature extraction. Each image gets a corresponding image feature. The image features are , and then through the classification loss and triplet loss The ResNet-50 backbone network is trained with constraints. The calculation process is as follows:

[0035] ,

[0036] ,

[0037] in, Indicates images The real identity tag, Representing images The pedestrian identity prediction probability, represents the margin in the triplet loss, setting , represents the maximum distance of a positive sample pair, where a positive sample pair represents a feature pair with the same pedestrian identity. represents the minimum distance of negative sample pairs, where negative sample pairs represent feature pairs of different pedestrian identities. Represents the operation of taking the maximum value, Indicates the operation of taking the minimum value;

[0038] Small batch samples The first half of the pedestrian image in the figure is the original pedestrian image, and the second half is the pedestrian image after changing clothes. The corresponding image features are the features of the original pedestrian image in the first half and the features of the pedestrian image after changing clothes in the second half. Select the original image features and the image features after changing clothes to perform mean square error loss Calculate, forcing the model to learn clues in areas unrelated to clothing, mean squared error loss The calculation is as follows:

[0039] ,

[0040] in, express norm, The features of the pedestrian image after changing clothes, Represents the features of the original pedestrian image, where .

[0041] The specific steps of S4 are as follows:

[0042] The pre-trained human attribute analysis model is used to obtain the gender information of the pedestrian in the original pedestrian image. The pre-trained 3D human mesh estimation model is used to predict the 3D body parameters of the pedestrian in 3D space, and then input into the SMPL-X3D human model of the corresponding gender to obtain the 3D human mesh.

[0043] Rendering 3D modeling through 3D tools in Python will project the 3D model onto the background of the original pedestrian image. The specific process is as follows:

[0044] (1) Create a blank 3D scene;

[0045] (2) Using the vertex position information in the 3D human body mesh and the vertex connection information of the SMPL-X model, a 3D model is added to the 3D scene. The model is rendered using a non-metallic, opaque, gray material.

[0046] (3) Illuminate the model with white parallel light;

[0047] (4) Calculate the camera parameters in the corresponding three-dimensional space based on the size of the original pedestrian image and the initial camera parameters. The camera parameters include the principal point and focal length. Add a camera with these parameters to the three-dimensional scene and use the camera to project the three-dimensional model onto the two-dimensional plane of the corresponding image.

[0048] (5) Merge the projected image with the original pedestrian image to obtain a 3D modeling image that retains the background ;

[0049] Then, the human semantic mask of the original pedestrian image is The pedestrian head in the original image is replaced with the corresponding head area in the 3D modeling image to obtain a 3D modeling image that retains the original pedestrian image head. The pedestrian head pixel value replacement formula is as follows:

[0050] , ,

[0051] in, express, express, and Respectively, Represents the number of pixel vectors of the head of the original pedestrian image;

[0052] Finally, the ResNet-50 backbone network is used to extract features from the 3D modeling image that retains the original pedestrian image head to obtain the 3D modeling image features. , and then constrained training is performed through classification loss and triplet loss.

[0053] The specific steps of S5 are as follows:

[0054] A lightweight convolutional block attention module (CBAM) is used to fuse the extracted original pedestrian image features and the 3D modeling image features. The CBAM module includes a channel attention module and a spatial attention module. The specific process is to fuse the 3D modeling image features. After being input into the channel attention module of the CBAM module, Perform dot product to obtain 3D modeling attention features, then concatenate the 3D modeling attention features with the original pedestrian image features in the channel dimension, perform convolution fusion and dimensionality reduction on the concatenated features, and then fuse the features into the dimension of a single feature, and then input the fused and reduced-dimensional features into the complete CBAM module, respectively passing through the channel attention module and the spatial attention module to obtain the global features of the channel and spatial attention, and finally train and optimize the obtained global features through classification loss and triplet loss;

[0055] The formula of the entire feature fusion process can be expressed as:

[0056] ,

[0057] ,

[0058] ,

[0059] in, represents the 3D modeling attention feature, represents the operation of the channel attention module, represents the dot product operation of corresponding element multiplication, Represents the features of the original input pedestrian image, represents a 1×1 convolution operation, represents the features after fusion dimensionality reduction, Represents a splicing operation, Represents global features, represents the operation of the spatial attention module, express The fused attention features obtained after the channel attention module;

[0060] Finally, feature similarity matching is performed based on global features, and the Euclidean distance between the original pedestrian image features to be retrieved and the pedestrian image features in the retrieval library is calculated. The retrieval library is sorted by the Euclidean distance to obtain the pedestrian retrieval results.

[0061] The present invention also provides a pedestrian search system based on combining human semantics with three-dimensional reconstruction, and implements a pedestrian search method based on combining human semantics with three-dimensional reconstruction.

[0062] The effects provided in the content of the invention are only the effects of the embodiments, rather than all the effects of the invention. The above technical solution has the following advantages or beneficial effects:

[0063] (1) The present invention makes full use of human semantic information and designs a clothing-changing module based on clothing sampling to expand the training data set samples. The clothing-changing module changes the appearance of pedestrians' clothing by replacing the pixel values ​​of the clothing area. Compared with the clothing-changing technology using the generative model, it is simpler and more effective, eliminating the complex model training and generation process, and can change the appearance of clothing more quickly, thereby improving the flexibility and efficiency of the application.

[0064] (2) By minimizing the mean square error of pedestrian features before and after changing clothes, the present invention allows the model to better capture features in pedestrian images that are not affected by clothing changes, thereby focusing more on extracting features related to identity. This is an effective method for solving the problem of changing clothes, with good practicality and convenience. During the model training process, not only can the adjustment of pedestrian features be quickly achieved, but the complexity of calculation can also be significantly reduced, making model training more efficient.

[0065] (3) The present invention introduces three-dimensional human reconstruction technology, which not only does not increase the additional clothing noise, but also enhances the body shape characteristics of the human body. In addition, considering that the head is the most discriminative area of ​​the human body, the present invention uses the human semantic head mask to retain the pedestrian head in the original image and extract more prominent pedestrian head features. Finally, it is fused with the original image features through the attention mechanism, which enhances the global expression ability of the features and improves the accuracy and robustness of the search for pedestrians changing clothes. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] The accompanying drawings are used to provide further understanding of the present invention and constitute a part of the specification. They are used to explain the present invention together with the embodiments of the present invention and do not constitute a limitation of the present invention.

[0067] Figure 1 It is a model framework diagram of the present invention;

[0068] Figure 2 This is a schematic diagram of a clothing changing module in the present invention;

[0069] Figure 3 It is a schematic diagram of a three-dimensional human body reconstruction module in the present invention;

[0070] Figure 4 An example display of the model retrieval result of the present invention is shown. DETAILED DESCRIPTION

[0071] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Without departing from the principles of the present invention, changes, modifications, replacements and deformations made to the embodiments of the present invention are all within the scope of protection of the present invention.

[0072] like Figures 1 to 3 As shown, the present invention provides a pedestrian search method based on combining human body semantics and three-dimensional reconstruction, comprising the following steps:

[0073] S1. Load pedestrian data and obtain human semantics:

[0074] A data set is constructed from the original pedestrian data obtained, and the original pedestrian data in the data set is randomly stratified and sampled. The original pedestrian images to be processed are obtained from the data set. The original pedestrian images are annotated with the semantic information of various parts of the human body through the pre-trained human semantic parsing model to obtain the corresponding human semantic mask.

[0075] S2. Human semantic clothing change, data sample expansion:

[0076] Use the human body semantic mask to change the clothes of the same pedestrian in two images with different clothes, and get the pedestrian image after changing clothes, so as to expand the data set samples;

[0077] S3. Comparative learning features, focusing on human body areas:

[0078] The backbone network is used to extract the features of the original pedestrian image and the pedestrian image after changing clothes. The mean square error loss is used to compare the changes in the features before and after changing clothes, making the two features closer.

[0079] S4. 3D human body reconstruction, head information preserved:

[0080] Firstly, the gender information of pedestrians is extracted by using the human attribute analysis model. The human mesh vertices are located in the 3D space by using the pre-trained 3D human mesh estimation model. The SMPL-X model of the corresponding gender is used to build a 3D model of pedestrians. Then, the 3D model is projected onto the background of the original pedestrian image. The head of the original pedestrian image is pasted to the corresponding head position using the human semantic head mask. Finally, the backbone network is used to extract the 3D modeling image features that retain the head.

[0081] S5. Pedestrian feature fusion and global feature matching:

[0082] The features of the extracted original pedestrian image are fused with the features of the 3D modeling image through convolutional attention to obtain the final global features. The global features are used for feature similarity matching. The similarity between features is evaluated by calculating the Euclidean distance, and the pedestrian retrieval gallery is sorted to obtain the final retrieval results.

[0083] The specific steps of S1 are as follows:

[0084] Small batch samples A batch of data after random stratified sampling of the data set, , small batch samples Included Groups of images of pedestrians with different identities, each group of pedestrians with the same identity contains Original pedestrian images with different clothing, , represents the total number of original pedestrian images in a mini-batch sample, Represents a small batch of samples Middle Original pedestrian image, The shape is , , and Respectively represent the number of channels, height and width of the original pedestrian image; and annotate the true identity of each original pedestrian image in the small batch sample. The real identity label is , ;

[0085] Use the pre-trained human semantic parsing model to analyze any original pedestrian image in any mini-batch sample in the dataset. Perform human semantic segmentation to obtain the local semantic confidence of human pixels , Indicates that it belongs to the human body semantics Some pixel coordinates are The semantic confidence of , The six semantic parts of the human body are annotated as background, head, upper clothes, lower clothes, upper limbs, and lower limbs. The activation function processes the local semantic confidence of human pixels to obtain 6 local semantic mask pixel coordinates , Indicates The pixel coordinates of the semantic mask of the semantic part of the individual body are then The function that obtains the human body semantic mask by calculating the function parameter ;

[0086] in, The specific implementation of the function is as follows:

[0087] , ,

[0088] The specific implementation of the function is as follows:

[0089] .

[0090] The specific steps of S2 are as follows:

[0091] Select the original pedestrian image Images of the same person wearing different clothes Comes from the original pedestrian image Perform the clothing change operation, specifically through the original pedestrian image Human semantic mask and images Human semantic mask Find the overlapping area of ​​the clothing masks of the two images and replace the pixels. The unreplaced clothing area is filled with the clothes of the other image. The process of obtaining the clothes of the other image is in the image A pixel block with a size of 1 / 5 of the clothing area is taken from the 1 / 4 position of the clothing area and converted into shape;

[0092] The replacement process is as follows:

[0093] ,

[0094] ,

[0095] in, Indicates the overlapping area of ​​clothing mask, Indicates the area of ​​clothing that has not been replaced, Indicates the clothing mask value, , Indicates the mask value of the top area. Indicates the mask value of the top area;

[0096] After replacement, we get a pedestrian image with changed clothes. Then select the original pedestrian image The new images are obtained by replacing other images of the same pedestrian in different clothing. Similarly, all the original pedestrian images in the data set are replaced by images of the same pedestrian in different clothing, thereby expanding the number of samples in the data set.

[0097] The specific steps of S3 are as follows:

[0098] All original pedestrian images and the pedestrian images after changing clothes are concatenated on the small batch samples to obtain a shape of Data samples, get small batch samples after data expansion , Represents a small batch of samples Middle Pedestrian images, , Represents a small batch of samples The number of pedestrian images in the image is calculated, and then the ResNet-50 backbone network is used for feature extraction. Each image gets a corresponding image feature. The image features are , and then through the classification loss and triplet loss The ResNet-50 backbone network is trained with constraints. The calculation process is as follows:

[0099] ,

[0100] ,

[0101] in, Indicates images The real identity tag, Representing images The pedestrian identity prediction probability, represents the margin in the triplet loss, setting , represents the maximum distance of a positive sample pair, where a positive sample pair represents a feature pair with the same pedestrian identity. represents the minimum distance of negative sample pairs, where negative sample pairs represent feature pairs of different pedestrian identities. Represents the operation of taking the maximum value, Indicates the operation of taking the minimum value;

[0102] Small batch samples The first half of the pedestrian image in the figure is the original pedestrian image, and the second half is the pedestrian image after changing clothes. The corresponding image features are the features of the original pedestrian image in the first half and the features of the pedestrian image after changing clothes in the second half. Select the original image features and the image features after changing clothes to perform mean square error loss Calculate, forcing the model to learn clues in areas unrelated to clothing, mean squared error loss The calculation is as follows:

[0103] ,

[0104] in, express norm, The features of the pedestrian image after changing clothes, Represents the features of the original pedestrian image, where .

[0105] The specific steps of S4 are as follows:

[0106] The pre-trained human attribute analysis model is used to obtain the gender information of the pedestrian in the original pedestrian image. The pre-trained 3D human mesh estimation model is used to predict the 3D body parameters of the pedestrian in 3D space, and then input into the SMPL-X3D human model of the corresponding gender to obtain the 3D human mesh.

[0107] Rendering 3D modeling through 3D tools in Python will project the 3D model onto the background of the original pedestrian image. The specific process is as follows:

[0108] (1) Create a blank 3D scene;

[0109] (2) Using the vertex position information in the 3D human body mesh and the vertex connection information of the SMPL-X model, a 3D model is added to the 3D scene. The model is rendered using a non-metallic, opaque, gray material.

[0110] (3) Illuminate the model with white parallel light;

[0111] (4) Calculate the camera parameters in the corresponding three-dimensional space based on the size of the original pedestrian image and the initial camera parameters. The camera parameters include the principal point and focal length. Add a camera with these parameters to the three-dimensional scene and use the camera to project the three-dimensional model onto the two-dimensional plane of the corresponding image.

[0112] (5) Merge the projected image with the original pedestrian image to obtain a 3D modeling image that retains the background ;

[0113] Then, the human semantic mask of the original pedestrian image is The pedestrian head in the original image is replaced with the corresponding head area in the 3D modeling image to obtain a 3D modeling image that retains the original pedestrian image head. The pedestrian head pixel value replacement formula is as follows:

[0114] , ,

[0115] in, express, express, and Respectively, Represents the number of pixel vectors of the head of the original pedestrian image;

[0116] Finally, the ResNet-50 backbone network is used to extract features from the 3D modeling image that retains the original pedestrian image head to obtain the 3D modeling image features. , and then constrained training is performed through classification loss and triplet loss.

[0117] The specific steps of S5 are as follows:

[0118] A lightweight convolutional block attention module (CBAM) is used to fuse the extracted original pedestrian image features and the 3D modeling image features. The CBAM module includes a channel attention module and a spatial attention module. The specific process is to fuse the 3D modeling image features. After being input into the channel attention module of the CBAM module, Perform dot product to obtain 3D modeling attention features, then concatenate the 3D modeling attention features with the original pedestrian image features in the channel dimension, perform convolution fusion and dimensionality reduction on the concatenated features, and then fuse the features into the dimension of a single feature, and then input the fused and reduced-dimensional features into the complete CBAM module, respectively passing through the channel attention module and the spatial attention module to obtain the global features of the channel and spatial attention, and finally train and optimize the obtained global features through classification loss and triplet loss;

[0119] The formula of the entire feature fusion process can be expressed as:

[0120] ,

[0121] ,

[0122] ,

[0123] in, represents the 3D modeling attention feature, represents the operation of the channel attention module, represents the dot product operation of corresponding element multiplication, Represents the features of the original input pedestrian image, represents a 1×1 convolution operation, represents the features after fusion dimensionality reduction, Represents a splicing operation, Represents global features, represents the operation of the spatial attention module, express The fused attention features obtained after the channel attention module;

[0124] Finally, feature similarity matching is performed based on global features, and the Euclidean distance between the original pedestrian image features to be retrieved and the pedestrian image features in the retrieval library is calculated. The retrieval library is sorted by the Euclidean distance to obtain the pedestrian retrieval results.

[0125] Example 2

[0126] A pedestrian search system based on combining human semantics with three-dimensional reconstruction implements a pedestrian search method based on combining human semantics with three-dimensional reconstruction.

[0127] Example 3

[0128] like Figure 4 As shown, it is an example display of the result of searching by the method of the present invention, Figure 4 It can be seen that three groups of results are given, that is, three pedestrians are selected, and the original pedestrian images to be queried of these three pedestrians are processed by the method of the present invention. Finally, the top 10 images are selected in the search library, and the 10 retrieved images are sorted according to the feature similarity and the Euclidean distance between the original pedestrian image features and the pedestrian image features in the search library. The search results are marked under the image according to the real identity label of the original pedestrian image to be queried, T means correct, F means wrong, and Figure 4 It can be seen from the marked T / F that the invented method has a high accuracy in searching for pedestrians who are changing clothes.

[0129] Although the above describes the specific implementation mode of the invention in conjunction with the drawings, it is not intended to limit the scope of protection of the invention. Based on the technical solution of the present invention, various modifications or variations that can be made by those skilled in the art without creative work are still within the scope of protection of the present invention.

Claims

1. A pedestrian search method based on combining human semantics and three-dimensional reconstruction, characterized in that: The following steps are involved: S1. Load pedestrian data and obtain human semantics: A data set is constructed from the original pedestrian data obtained, and the original pedestrian data in the data set is randomly stratified and sampled. The original pedestrian images to be processed are obtained from the data set. The original pedestrian images are annotated with the semantic information of various parts of the human body through the pre-trained human semantic parsing model to obtain the corresponding human semantic mask. S2. Human semantic clothing change, data sample expansion: Use the human body semantic mask to change the clothes of the same pedestrian in two images with different clothes, and get the pedestrian image after changing clothes, so as to expand the data set samples; S3. Comparative learning features, focusing on human body areas: The backbone network is used to extract the features of the original pedestrian image and the pedestrian image after changing clothes. The mean square error loss is used to compare the changes in the features before and after changing clothes, making the two features closer. S4. 3D human body reconstruction, head information preserved: Firstly, the gender information of pedestrians is extracted by using the human attribute analysis model. The human mesh vertices are located in the 3D space by using the pre-trained 3D human mesh estimation model. The SMPL-X model of the corresponding gender is used to build a 3D model of pedestrians. Then, the 3D model is projected onto the background of the original pedestrian image. The head of the original pedestrian image is pasted to the corresponding head position using the human semantic head mask. Finally, the backbone network is used to extract the 3D modeling image features that retain the head. S5. Pedestrian feature fusion and global feature matching: The features of the extracted original pedestrian image are fused with the features of the 3D modeling image through convolutional attention to obtain the final global features. The global features are used for feature similarity matching. The similarity between features is evaluated by calculating the Euclidean distance, and the pedestrian retrieval gallery is sorted to obtain the final retrieval results.

2. The pedestrian search method based on combining human semantics and three-dimensional reconstruction according to claim 1, characterized in that: The specific steps of S1 are as follows: Small batch samples A batch of data after random stratified sampling of the data set, , small batch samples Included Groups of images of pedestrians with different identities, each group of pedestrians with the same identity contains Original pedestrian images with different clothing, , represents the total number of original pedestrian images in a mini-batch sample, Represents a small batch of samples Middle Original pedestrian image, The shape is , , and Respectively represent the number of channels, height and width of the original pedestrian image; and annotate the true identity of each original pedestrian image in the small batch sample. The real identity label is , ; Use the pre-trained human semantic parsing model to analyze any original pedestrian image in any mini-batch sample in the dataset. Perform human semantic segmentation to obtain the local semantic confidence of human pixels , Indicates that it belongs to the human body semantics Some pixel coordinates are The semantic confidence of , The six semantic parts of the human body are annotated as background, head, upper clothes, lower clothes, upper limbs, and lower limbs. The activation function processes the local semantic confidence of human pixels to obtain 6 local semantic mask pixel coordinates , Indicates The pixel coordinates of the semantic mask of the semantic part of the individual body are then The function that obtains the human body semantic mask by calculating the function parameter ; in, The specific implementation of the function is as follows: ， ， The specific implementation of the function is as follows: 。 3. The pedestrian search method based on combining human semantics and three-dimensional reconstruction according to claim 1, characterized in that: The specific steps of S2 are as follows: Select the original pedestrian image Images of the same person wearing different clothes Comes from the original pedestrian image Perform the clothing change operation, specifically through the original pedestrian image Human semantic mask and images Human semantic mask Find the overlapping area of ​​the clothing masks of the two images and replace the pixels. The unreplaced clothing area is filled with the clothes of the other image. The process of obtaining the clothes of the other image is in the image A pixel block with a size of 1 / 5 of the clothing area is taken from the 1 / 4 position of the clothing area and converted into shape; The replacement process is as follows: , , in, Indicates the overlapping area of ​​clothing mask, Indicates the area of ​​clothing that has not been replaced, Indicates the clothing mask value, , Indicates the mask value of the top area. Indicates the mask value of the top area; After replacement, we get a pedestrian image with changed clothes. Then select the original pedestrian image The new images are obtained by replacing other images of the same pedestrian in different clothing. Similarly, all the original pedestrian images in the data set are replaced by images of the same pedestrian in different clothing, thereby expanding the number of samples in the data set.

4. The pedestrian search method based on combining human semantics and three-dimensional reconstruction according to claim 1, characterized in that: The specific steps of S3 are as follows: All original pedestrian images and the pedestrian images after changing clothes are concatenated on the small batch samples to obtain a shape of Data samples, get small batch samples after data expansion , Represents a small batch of samples Middle Pedestrian images, , Represents a small batch of samples The number of pedestrian images in the image is calculated, and then the ResNet-50 backbone network is used for feature extraction. Each image gets a corresponding image feature. The image features are , and then through the classification loss and triplet loss The ResNet-50 backbone network is trained with constraints. The calculation process is as follows: ， ， in, Indicates images The real identity tag, Representing images The pedestrian identity prediction probability, represents the margin in the triplet loss, setting , represents the maximum distance of a positive sample pair, where a positive sample pair represents a feature pair with the same pedestrian identity. represents the minimum distance of negative sample pairs, where negative sample pairs represent feature pairs of different pedestrian identities. Represents the operation of taking the maximum value, Indicates the operation of taking the minimum value; Small batch samples The first half of the pedestrian image in the figure is the original pedestrian image, and the second half is the pedestrian image after changing clothes. The corresponding image features are the features of the original pedestrian image in the first half and the features of the pedestrian image after changing clothes in the second half. Select the original image features and the image features after changing clothes to perform mean square error loss Calculate, forcing the model to learn clues in areas unrelated to clothing, mean squared error loss The calculation is as follows: ， in, express norm, The features of the pedestrian image after changing clothes, Represents the features of the original pedestrian image, where .

5. The pedestrian search method based on combining human semantics and three-dimensional reconstruction according to claim 1, characterized in that: The specific steps of S4 are as follows: The pre-trained human attribute analysis model is used to obtain the gender information of the pedestrian in the original pedestrian image. The pre-trained 3D human mesh estimation model is used to predict the 3D body parameters of the pedestrian in 3D space, and then input into the SMPL-X3D human model of the corresponding gender to obtain the 3D human mesh. Rendering 3D modeling through 3D tools in Python will project the 3D model onto the background of the original pedestrian image. The specific process is as follows: (1) Create a blank 3D scene; (2) Using the vertex position information in the 3D human body mesh and the vertex connection information of the SMPL-X model, a 3D model is added to the 3D scene. The model is rendered using a non-metallic, opaque, gray material. (3) Illuminate the model with white parallel light; (4) Calculate the camera parameters in the corresponding three-dimensional space based on the size of the original pedestrian image and the initial camera parameters. The camera parameters include the principal point and focal length. Add a camera with these parameters to the three-dimensional scene and use the camera to project the three-dimensional model onto the two-dimensional plane of the corresponding image. (5) Merge the projected image with the original pedestrian image to obtain a 3D modeling image that retains the background ; Then, the human semantic mask of the original pedestrian image is The pedestrian head in the original image is replaced with the corresponding head area in the 3D modeling image to obtain a 3D modeling image that retains the original pedestrian image head. The pedestrian head pixel value replacement formula is as follows: ， ， in, express, express, and Respectively, Represents the number of pixel vectors of the head of the original pedestrian image; Finally, the ResNet-50 backbone network is used to extract features from the 3D modeling image that retains the original pedestrian image head to obtain the 3D modeling image features. , and then constrained training is performed through classification loss and triplet loss.

6. The pedestrian search method based on combining human semantics and three-dimensional reconstruction according to claim 1, characterized in that: The specific steps of S5 are as follows: A lightweight convolutional block attention module (CBAM) is used to fuse the extracted original pedestrian image features and the 3D modeling image features. The CBAM module includes a channel attention module and a spatial attention module. The specific process is to fuse the 3D modeling image features. After being input into the channel attention module of the CBAM module, Perform dot product to obtain 3D modeling attention features, then concatenate the 3D modeling attention features with the original pedestrian image features in the channel dimension, perform convolution fusion and dimensionality reduction on the concatenated features, and then fuse the features into the dimension of a single feature, and then input the fused and reduced-dimensional features into the complete CBAM module, respectively passing through the channel attention module and the spatial attention module to obtain the global features of the channel and spatial attention, and finally train and optimize the obtained global features through classification loss and triplet loss; The formula of the entire feature fusion process can be expressed as: ， ， ， in, represents the 3D modeling attention feature, represents the operation of the channel attention module, represents the dot product operation of corresponding element multiplication, Represents the features of the original input pedestrian image, represents a 1×1 convolution operation, represents the features after fusion dimensionality reduction, Represents a splicing operation, Represents global features, represents the operation of the spatial attention module, express The fused attention features obtained after the channel attention module; Finally, feature similarity matching is performed based on global features, and the Euclidean distance between the original pedestrian image features to be retrieved and the pedestrian image features in the retrieval library is calculated. The retrieval library is sorted by the Euclidean distance to obtain the pedestrian retrieval results.

7. A pedestrian search system based on combining human semantics and three-dimensional reconstruction, characterized by: Execute a pedestrian search method based on combining human semantics and three-dimensional reconstruction as described in any one of claims 1-6.

Images