A pedestrian search method based on feature fusion fine-grained image generation

Abstract

This invention provides a pedestrian search method based on feature fusion for fine-grained image generation, mainly comprising: performing feature processing on the pedestrian image using a residual network; filtering the feature candidate boxes output by the region proposal network based on a confidence threshold; inputting the filtered feature candidate boxes into a detection head and using bounding box regression to obtain fine-grained pedestrian boxes; labeling the identities in the detection results as marked or unmarked based on identity similarity scores; inputting the cropped image with labeled identities into the generator of a generative adversarial network, randomly combining the appearance and structural information of the input image; feeding the newly generated image into the teacher network of a discriminator to add soft labels; feeding the soft-labeled image into the discriminator for discrimination, whereby the discriminator uses a discriminative feature extraction layer to re-identify pedestrians, and then outputting the results. This invention solves the problem that existing pedestrian search methods cannot obtain high-precision features in real-world scenarios.

Description

Technical Field

[0001] This invention relates to the field of pedestrian search technology, and more particularly to a pedestrian search method based on fine-grained image generation using feature fusion. Background Technology

[0002] In the field of computer vision, pedestrian search has always been a challenging task, aiming to find specific pedestrians from real-world cameras. Generally, existing pedestrian search methods use manually cropped videos to achieve clean and less noisy pedestrian bounding boxes, but this requires processing entire images of numerous pedestrians from actual surveillance videos, a complex and resource-intensive process that hinders widespread use in real-world scenarios. Furthermore, the detection phase needs to identify all people in the image, requiring the extraction of commonalities to correctly detect everyone. The re-identification phase, which determines the specific identity of each detected person—solving the "Who is this person?" question—requires identifying the differences between individuals. In other words, the detection phase requires extracting common features, while the re-identification phase requires extracting differences between features, further impacting pedestrian search performance. In large-scale intelligent surveillance systems, these two problems pose challenges to existing pedestrian search methods for real-time target search tasks. Existing surveillance videos used for training only contain a few photos of each pedestrian, and most of these photos share similar backgrounds and clothing, making it difficult for trained models to guarantee the accuracy of the discriminative features learned in real-world scenarios. Summary of the Invention

[0003] This invention provides a pedestrian search method based on feature fusion for fine-grained image generation, which solves the problem that existing pedestrian search methods cannot obtain high-precision features in real-world scenarios.

[0004] The technical means employed in this invention are as follows:

[0005] A pedestrian search method based on fine-grained image generation using feature fusion includes the following steps:

[0006] Pedestrian images are acquired from street view surveillance video, and residual networks are used to perform feature processing on the pedestrian images to obtain pedestrian image features;

[0007] The pedestrian image features are input into the region proposal network, and the feature candidate boxes output by the region proposal network are filtered based on the confidence threshold.

[0008] Align the contents within the feature candidate boxes with regions of interest to obtain identity feature maps of consistent length;

[0009] The identity feature map is input into the detection head, which is used to determine whether it is a pedestrian. When it is confirmed to be a pedestrian, bounding box regression is used to fine-tune the coarse pedestrian box to obtain a fine pedestrian box.

[0010] The process involves obtaining the target query, calculating the identity similarity between the target query and the detection results, and labeling the identities in the detection results as marked or unmarked based on the identity similarity score. The detection results are surveillance images with detailed pedestrian bounding boxes. The formula for calculating the identity similarity is as follows:

[0011]

[0012] in It is the identity feature vector of the query target. It is the identity feature vector of the detection result;

[0013] The cropped image with tagged identity is input into the generator of the generative adversarial network. The appearance information extractor of the generator extracts the appearance information of the person in the cropped image, and the structure information extractor of the generator extracts the structure information of the person in the cropped image. The information fusion unit of the generator randomly fuses the appearance information and structure information pairwise to generate a new image.

[0014] The newly generated image is fed into the discriminator's teacher network, which then attaches soft labels to the generated image.

[0015] The image with the soft label attached is fed into the discriminator for identification. The discriminator uses the discriminative feature extraction layer to re-identify the pedestrian and then outputs the result.

[0016] Furthermore, the residual network comprises five convolutional layers, which process the input image layer by layer, and finally extract the input image into a 2048-dimensional feature vector.

[0017] Furthermore, the coarse pedestrian boxes are fine-tuned using bounding box regression, including adjusting the coarse pedestrian boxes using a trained bounding box regression model trained with a bounding box regression loss supervised model, the bounding box regression loss being:

[0018]

[0019] in, For the input sample, The probability that each input category belongs to the true category. T This represents the number of samples.

[0020] Further, an identity similarity score is calculated between the query target and the detection results, and the identities in the detection results are labeled as marked and unlabeled based on the identity similarity score, including:

[0021] The detection results and the query target are input into the trained identity feature guidance model. The identity feature guidance model calculates the identity similarity score between the query target and the detection results to label the pedestrians in the detection results. The labeling results include labeled and unlabeled. The labeled identity is the target image in this case, and the unlabeled identity is not the target image in this case.

[0022] The identity feature-guided model is trained based on an identity feature-guided loss, which is:

[0023]

[0024] in For the number of samples, It is a threshold for controlling the ratio of hard negative samples. Indicates the first One negative sample, The number of hard negative samples is calculated as follows:

[0025] .

[0026] Furthermore, the identified, cropped image is input into the generator of the generative adversarial network. The generator randomly combines the appearance and structural information of the input image to generate a new image, including: supervising the reconstructed image using pixel-level L1 loss.

[0027]

[0028]

[0029] in, Describes the L1 loss function. Image Appearance information Image Structural information, L app L represents the loss of the appearance information extractor. str E represents the loss of the structural information extractor. app E represents the appearance information extractor. str D represents a structural information extractor, and D represents a generative adversarial network.

[0030] The identification loss is:

[0031]

[0032] in yes belong truth value The predicted probability, It involves synthesizing new images;

[0033] The adversarial loss of a generative adversarial network (GAN) is used to match the distribution of the synthesized image with the distribution of the real data. The adversarial loss of the GAN is:

[0034]

[0035] in This represents the distribution function.

[0036] Furthermore, the newly generated image is fed into the discriminator's teacher network, which then attaches soft labels to the generated image, including:

[0037] The teacher network divides the feature space of the synthesized image into 6 strip regions in the horizontal direction, then convolves the regions to obtain tensors with 6 channels, and finally performs Softmax classification for each channel to assign soft labels to the synthesized image.

[0038] Furthermore, the image with the added soft label is fed into a discriminator for identification. The discriminator uses a discriminative feature extraction layer to re-identify pedestrians and then outputs the results, including:

[0039] KL hashing is used to minimize the discriminator's predicted probability distribution. and probability assignments predicted by the teacher network The formula for KL hash is:

[0040]

[0041] in It is the number of identities. It minimizes the discriminator prediction. identity as The probability, Teacher network prediction identity as The probability of.

[0042] Furthermore, the image with the added soft label is fed into a discriminator for identification. The discriminator uses a discriminative feature extraction layer to re-identify pedestrians, and then outputs the results, which also include:

[0043] Using identity loss, the discriminator learns discriminative features from the soft-labeled images output by the teacher network. The identity loss is defined as:

[0044]

[0045] in, Teacher network prediction identity as The probability of.

[0046] Compared with the prior art, the present invention has the following advantages:

[0047] This invention employs a generative adversarial network (GAN) to synthesize images for training a more robust person search network. The detection part utilizes a feature extraction network constructed from residual networks, which then filters data using a region proposal network. Simultaneously, an identity feature guidance module proposed in this paper assists in feature extraction. The extracted features are fed into the proposed GAN image synthesis module for generative adversarial training. The generator reconstructs and fuses features from the original image, generating high-quality cross-identity person images. These new images are then fed into a discriminator for feature identification, thereby reducing the discrepancy between the detection and re-identification stages and improving the accuracy of person search. Attached Figure Description

[0048] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0049] Figure 1 This is a flowchart of a pedestrian search method based on feature fusion for fine-grained image generation according to the present invention.

[0050] Figure 2 The detailed execution flow of the pedestrian search method in this embodiment is as follows.

[0051] Figure 3 This is a structural diagram of the pedestrian detection network model in the embodiment.

[0052] Figure 4 This is a diagram of the pedestrian re-identification network model structure in the embodiment. Detailed Implementation

[0053] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0054] like Figure 1As shown, this invention provides a pedestrian search method based on feature fusion and fine-grained image generation, comprising the following steps:

[0055] S1. Obtain pedestrian images from street view surveillance video, and use a residual network to perform feature processing on the pedestrian images to obtain pedestrian image features;

[0056] S2. Input the pedestrian image features into the region proposal network, and filter the feature candidate boxes output by the region proposal network based on the confidence threshold;

[0057] S3. Input the filtered feature candidate boxes into the detection head to obtain coarse pedestrian boxes. Use bounding box regression to fine-tune the coarse pedestrian boxes to obtain fine pedestrian boxes.

[0058] S4. Calculate the identity similarity score between the query target and the detection results, and label the identities in the detection results as labeled and unlabeled identities based on the identity similarity score;

[0059] S5. Input the cropped image with tagged identity into the generator of the generative adversarial network. The generator is used to randomly combine the appearance information and structural information of the input image to generate a new image.

[0060] S6. Feed the newly generated image into the discriminator’s teacher network and use the teacher network to attach soft labels to the newly generated image.

[0061] S7. The image with the soft label attached is sent to the discriminator for identification. The discriminator uses the discriminative feature extraction layer to complete the re-identification of pedestrians and then outputs the results.

[0062] The following specific application examples will further illustrate the solution and effects of the present invention.

[0063] like Figure 2-4 As shown, this embodiment provides a pedestrian search method based on feature fusion for fine-grained image generation. The method includes the following steps:

[0064] Step 1: The surveillance camera captures street scenes of crowds, and through video processing, multiple images of people are obtained. In this invention, video processing includes decoding each frame of the surveillance video into an image.

[0065] Step 2: Use a feature extraction network to process the input image containing multiple people to obtain the input image features. The feature extraction network uses a residual convolutional network, which contains five convolutional layers. The input image is processed layer by layer, and finally the input image is extracted into 2048-dimensional features.

[0066] Step 3: Feed the extracted feature vectors into the Region Proposal Network (RPN). The RPN is as follows: Figure 3 As shown, multiple candidate boxes including the human body are proposed, and then non-maximum suppression is used to remove candidate boxes with confidence scores below a given threshold in order to obtain high-quality candidate boxes.

[0067] Step 4: Perform region of interest alignment on the contents of the candidate boxes to obtain feature vectors of consistent length.

[0068] Step 5: The feature vector is fed into the detection head to detect whether it is a pedestrian feature vector. Then, bounding box regression is used to fine-tune it, thereby obtaining a more refined pedestrian bounding box. In this step, the detection head is used to determine whether each candidate box actually contains a person.

[0069] Step 6: Separate the identities in the detection head output into labeled and unlabeled categories using identity feature guidance. Identity feature guidance is mainly trained using identity feature guidance loss to calculate the identity similarity score between the query target and the detection result.

[0070] Step 7: The identity feature-guided output cropped image that is closer to the labeled identity than other bounding boxes is fed into the generative adversarial network (GAN) image synthesis module. The GAN architecture is as follows: Figure 4 As shown.

[0071] Step 8: The Generative Adversarial Network (GAN) image synthesis module uses the generator module to divide the input image into two parts: appearance information and structural information. In this invention, appearance information includes the color, texture, and style of clothing / shoes, and other clues related to the ID card. Structural information includes body shape, hair, carrying details, posture, background, location, and viewpoint.

[0072] Step 9: The generator module combines two pieces of information from two different groups of people to generate a new image;

[0073] Step 10: The generated image is fed into the teacher network of the discriminator, and the teacher network attaches a soft label to it;

[0074] Step 11: The image with the soft label is sent to the discriminator for discrimination. The discriminator uses the discriminative feature extraction layer to complete the re-identification of the human and then outputs the result.

[0075] In a specific embodiment, region of interest alignment utilizes bilinear interpolation to segment feature maps of inconsistent sizes, thereby aligning features. After obtaining the predicted bounding boxes, the detection head is supervised by several loss functions: bounding box regression loss, classification loss, and the identity feature guidance loss proposed in this paper. These three losses optimize bounding box detection. The identity feature guidance module brings frontal examples from different images closer together, so images from the same person can obtain higher similarity scores. Bounding box regression is measured by the intersection-union ratio (IoU), i.e., the degree of overlap between the predicted box and the ground truth box. The recognition loss is defined as follows:

[0076]

[0077] in For the input sample, The probability that each input category belongs to the true category, where T is the number of samples.

[0078] In a specific embodiment, the input image features are processed using the identity feature guidance in step 5. By calculating the identity similarity score between the query target and the detection result, the identities in the detection result are separated into labeled and unlabeled. The labeled identity is the target image in this instance, while the unlabeled identity is not the target image in this instance. This generates high-quality positive samples for the re-identification stage. The identity feature-guided loss function is specifically defined as follows:

[0079]

[0080] in For the number of samples, Regarding the number of hard negative samples, we fully considered the positive role of hard negative samples in improving the network's discrimination ability, and set the number of hard negative samples in the following way:

[0081]

[0082] in It is the threshold for controlling the ratio of hard negative samples.

[0083] In a specific embodiment, the generative adversarial network (GAN) image synthesis module in step 7 comprises a generator and a discriminator, wherein the generator adopts an encoder-decoder paradigm. The GAN image synthesis module receives input features, decomposes the feature information into appearance information and structural information, and then the encoder processes the given two images... and Extract information separately, and Appearance information extracted as ,Will Structural information extraction as These two parts are then fed into the decoder module of the generator. In this invention, an appearance information extractor is used to extract appearance information, and a structural information extractor is used to extract structural information.

[0084] In a specific example, the decoder in the generator in step 8 will... and By combining them, a new image is generated, which is then represented as , This represents an image that provides structural code. This indicates the image for which the appearance code is provided. For the reconstructed image, pixel-level values ​​are used. Monitoring losses:

[0085]

[0086] in Let L1 loss function be used. Furthermore, the recognition loss is employed to force the synthesized image to maintain its identity consistency. The recognition loss is:

[0087]

[0088] in It belongs to Basic truth classes of The predicted probability, in synthesis The appearance code was provided at that time. Then, a GAN adversarial loss was used to match the distribution of the synthesized image with the distribution of the real data; the loss was:

[0089]

[0090] in The distribution function represents the process by which a GAN network fits a new image to a prior distribution during image generation, thus producing a synthetic image. This concept is a concept proposed in GAN networks and belongs to the existing technology in this field.

[0091] In a specific embodiment, in step 9, within the discriminator, the constructed teacher network divides the feature space of the synthesized image into six strip regions along the horizontal direction. These regions are then convolved to obtain six-channel tensors. Finally, Softmax classification is performed on each channel to assign soft labels to the synthesized image, thereby better guiding the discriminator to more effectively complete the re-identification task. The teacher network model is trained using the original dataset. After Softmax classification, the name of the category to which the soft label belongs becomes the soft label of the synthesized image.

[0092] In a specific embodiment, to better synthesize the image, KL hashing is further used to minimize the predicted probability distribution of the discrimination module. and probability assignments predicted by teachers The KL hash formula is:

[0093]

[0094] in It refers to the number of identities.

[0095] In a specific embodiment, identity loss is utilized to enable the discriminator to learn discriminative features from the soft-labeled images output by the teacher network. Through the pre-exchange of appearance and structural information in the synthesized human images, the discriminator is able to uncover more distinctive and refined features and learn to focus on identity attributes. The identity loss is defined as:

[0096]

[0097] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A pedestrian search method based on feature fusion-based fine-grained image generation, characterized in that, The method comprises the following steps: Pedestrian images are acquired from street view monitoring videos, and residual network is used to process features of the pedestrian images to obtain pedestrian image features; The pedestrian image features are input into a region proposal network, and feature candidate boxes output by the region proposal network are filtered based on a confidence threshold; The content in the feature candidate box is aligned to obtain an identity feature map with consistent length; The identity feature map is input into a detection head, which is used to determine whether it is a pedestrian, and when it is confirmed as a pedestrian, a bounding box regression is used to fine-tune a rough pedestrian box to obtain a fine pedestrian box; A query target is acquired, identity similarity between the query target and a detection result is calculated, and identities in the detection result are labeled as marked and unmarked identities according to the identity similarity score, wherein the detection result is a monitoring image with a fine pedestrian box; and the identity similarity calculation formula is: wherein is an identity feature vector of the query target, is an identity feature vector of the detection result; The cropped image with the labeled identity is input into a generative adversarial network synthesis image module, which includes a generator and a discriminator. The generator adopts an encoder-decoder paradigm. The generative adversarial network synthesis image module receives input features, decomposes the feature information into appearance information and structure information, and then an encoder extracts information from the two given images and respectively, and extracts the appearance information of as and extracts the structure information of as , and sends the two parts to the decoder in the generator; the decoder in the generator combines and to generate a new image , wherein represents an image providing structure code, represents an image providing appearance code. The newly generated image is sent to the discriminator's teacher network to add a soft label to the generated new image using the teacher network; the teacher network divides the feature space of the generated new image into 6 strip regions in the horizontal direction, then performs convolution on the regions to obtain a 6-channel tensor, and finally performs Softmax classification on each channel to assign a soft label to the synthesized image; The image with the added soft label is sent to the discriminator for identification, and the discriminator uses the identification feature extraction layer to complete pedestrian re-identification and then outputs the result. 2.The pedestrian search method based on feature fusion and fine-grained image generation according to claim 1, wherein, The residual network includes five convolutional layers, which process the input image layer by layer, and finally extract the input image into a 2048-dimensional feature vector. 3.The pedestrian search method based on feature fusion and fine-grained image generation according to claim 1, wherein, The bounding box regression is used to fine-tune the rough pedestrian box, including adjusting the rough pedestrian box using a trained bounding box regression model, wherein the bounding box regression model is trained using a bounding box regression loss supervision model, and the bounding box regression loss is: wherein, is the input sample, is the probability that each input class belongs to the true class, T is the number of samples. 4.The pedestrian search method based on feature fusion and fine-grained image generation of claim 1, wherein, The identity similarity score between the query target and the detection result is calculated, and the identities in the detection result are labeled as marked and unmarked identities according to the identity similarity score, including: The detection result and the query target are input into the trained identity feature guidance model, which labels the pedestrians in the detection result according to the identity similarity score between the query target and the detection result, and the labeling result includes marked and unmarked identities, wherein the marked identity is the target image this time, and the unmarked identity is not the target image this time; The identity feature guidance model is trained according to an identity feature guidance loss, and the identity feature guidance loss is: wherein is the number of samples, is a threshold value that controls the ratio of hard negative samples, denotes the hard negative sample, is the number of hard negative samples, obtained according to: 。 5. The method of claim 1, wherein the method is a feature fusion based fine-grained image generation pedestrian search method. The cropped image with the labeled identity is input into the generator of the generative adversarial network, which is used to randomly combine the appearance information and the structure information of the input image to generate a new image, including using a pixel-level L1 loss to supervise the reconstructed image: wherein, represents the LI loss function, represents the appearance information of the picture represents the structure information of the picture represents the structure information of the picture represents the structure information of the picture app represents the loss of the appearance information extractor, str represents the loss of the structure information extractor, app represents the appearance information extractor, str represents the structure information extractor, and D represents a generative adversarial network. The identification loss is: wherein is belongs to the truth value the predicted probability, is to synthesize a new image; The distribution of the synthesized image is matched with the distribution of the real data using the adversarial loss of the generative adversarial network, and the adversarial loss of the generative adversarial network is: wherein denotes the distribution function.

6. The method of claim 1, wherein the method is a feature fusion based fine-grained image generation pedestrian search method. The image with the added soft label is sent to the discriminator for identification, and the discriminator uses the identification feature extraction layer to complete pedestrian re-identification and then outputs the result, including: Using a KL hash to minimize the predicted probability distribution of the discriminator and the probability distribution predicted by the teacher network The formula for the KL hash is: wherein is the number of identities, is the discriminator prediction that the identity is with probability is the teacher network prediction that the identity is with probability.

7. The method of claim 1, wherein the method is a feature fusion based fine-grained image generation pedestrian search method. The image with the additional soft label is sent into the discriminator for identification, the discriminator uses the identification feature extraction layer to complete the re-identification of the pedestrian, and then outputs the result, which also includes: Identity loss is used to make the discriminator learn the identification features of the soft label image output by the teacher network, and the identity loss is defined as: wherein, is the teacher network prediction of the identity as the probability.

Images