Sketch pedestrian re-identification method combining step-by-step feature enhancement and cross-modal information compensation

Abstract

The invention relates to a sketch pedestrian re-identification method combining step-by-step feature enhancement and cross-modal information compensation, and belongs to the technical field of image retrieval. The method comprises the steps of extracting pedestrian photo features and pedestrian sketch features; constructing a cross-modal information compensation combination space, shortening the modal distance between the pedestrian photo feature and the pedestrian sketch feature in the space, learning common information in the modal, and performing information compensation; step-by-step feature enhancement is carried out on the features after information compensation and the features under the next scale, fine-grained feature representation of the features is learned, and final feature representation is obtained; and training a retrieval model by using the final feature representation, and searching a corresponding pedestrian photo image by using the pedestrian sketch image through the optimized retrieval model. According to the method, the identification problem caused by modal difference is effectively solved, and the characterization capability of the multi-scale features on the pedestrian identity information is also improved.

Description

Technical Field

[0001] The present invention provides a sketch pedestrian re-identification method combining step-by-step feature enhancement with cross-modal information compensation, and relates to the technical field of image retrieval. Background Art

[0002] With the widespread application and deep penetration of modern surveillance camera systems, using surveillance camera networks everywhere to lock and track suspects has evolved into an indispensable scientific and technological means for solving cases. Among them, pedestrian re-identification technology plays a core role. After obtaining the image record of the target pedestrian under a surveillance camera, the technology can accurately identify and locate the traces of the same pedestrian in the complex environment captured by other cameras with cutting-edge computer vision algorithms. However, in real scenes, due to factors such as angle, distance, and light, the faces of pedestrians may not always be clearly visible or may be blocked. Faced with this dilemma, it is necessary to focus not only on facial features but also on matching based on the entire pedestrian (including body posture, clothing, hairstyle, etc.). Investigators will carefully question witnesses and victims to obtain detailed descriptions of suspects, and rely on professional forensic profiling experts to convert these descriptions into sketch images of suspects. These sketch images are then used to search in a huge database of surveillance videos to find target suspects that match the sketches. This challenge has given rise to the demand for sketch pedestrian re-identification technology. The technology aims to solve the problem of accurately identifying and locking specific pedestrians in surveillance videos using advanced computer vision algorithms based on hand-drawn sketches rather than actual photos. This not only puts forward higher technical requirements for case detection, but also demonstrates the great potential of modern technology in maintaining social security.

[0003] Some existing methods use a cross-domain adversarial framework to filter domain-sensitive information to learn domain-invariant features. This adversarial idea reduces modal differences to improve model retrieval performance. However, it cannot effectively separate identity information from modal information, and the training is difficult. In addition, some methods explore the use of attention and other methods to pull features closer. Although this method is simple, it is not effective for modal pull-in. To address this problem, the present invention proposes a sketch pedestrian re-identification method that combines step-by-step feature enhancement with cross-modal information compensation. Summary of the invention

[0004] In view of the shortcomings of the prior art, the present invention proposes a sketch pedestrian re-identification method that combines step-by-step feature enhancement with cross-modal information compensation. During the training stage, the model uses a cross-modal information compensation mechanism to solve the differences between different modalities such as sketches and photos. The step-by-step feature enhancement technology can deeply learn the subtle differences and internal connections between the step-by-step features, thereby further enhancing the expression ability of intra-modal information, solving the problem of pedestrians being indistinguishable due to changes in viewing angles between different cameras, and enhancing the expression of fine-grained features.

[0005] The technical solution of the present invention is implemented as follows: a sketch pedestrian re-identification method combining step-by-step feature enhancement and cross-modal information compensation, the method comprising:

[0006] Step S1: Obtain pedestrian photo images and pedestrian sketch images, construct training sets and test sets, and perform image preprocessing;

[0007] Step S2: inputting the preprocessed pedestrian photo image and pedestrian sketch image into the pedestrian photo feature extraction network and the pedestrian sketch feature extraction network to extract pedestrian photo features and pedestrian sketch features respectively;

[0008] Step S3: construct a cross-modal information compensation combination space, in which the modal distance between pedestrian photo features and pedestrian sketch features is shortened, and the common information within the modality is learned and information compensation is performed;

[0009] Step S4: The features after information compensation and the features at the next scale are enhanced step by step, and the fine-grained feature representation of the learning features is used to gradually transfer the shallow features to the deep features to obtain the final feature representation;

[0010] Step S5: using the final feature representation to train the retrieval model, designing a joint optimization algorithm to optimize the retrieval model to achieve the best network model parameters, and using the pedestrian sketch image to find the corresponding pedestrian photo image through the optimized retrieval model;

[0011] Step S6: Evaluate the performance of the retrieval model and perform iterative optimization accordingly.

[0012] As a further solution of the present invention, in step S1, preprocessing the collected data sample images includes: normalizing, grayscale transforming, randomly horizontally flipping, filling, cropping and erasing the pedestrian photo images and pedestrian sketch images in the training set and the test set.

[0013] As a further solution of the present invention, step S2 includes the following:

[0014] Step S201, backbone network construction: the backbone networks of pedestrian photo feature extraction network and pedestrian sketch feature extraction network are both constructed on the basis of the pyramid vision transformer backbone PVTv2;

[0015] Step S202, dual-stream feature extraction: the pre-processed pedestrian photo image and pedestrian sketch image are respectively input into the pedestrian photo feature extraction network and the pedestrian sketch feature extraction network, and features are extracted from pedestrian samples of the two modalities to obtain pedestrian photo features and pedestrian sketch features.

[0016] As a further solution of the present invention, in step S202, the pedestrian photo feature extraction network and the pedestrian sketch feature extraction network adopt a parallel structure, and the pedestrian photo feature extraction network and the pedestrian sketch feature extraction network are respectively composed of four stages, each stage generates features of different sizes, and the pedestrian sketch features and pedestrian photo features obtained in each stage are expressed as and

[0017] As a further solution of the present invention, the specific steps of step S3 are as follows:

[0018] Step S301: represent the pedestrian sketch features and pedestrian photo features obtained at each stage and Input into the multi-head self-attention network with residual connection for information integration. The specific process calculation formula is as follows:

[0019]

[0020] Among them, MHSA is the implementation of the multi-head self-attention mechanism, x T represents the matrix transpose operation, LN is the LayerNormalization operation, i.e., the layer normalization operation; x q , x k , x v They are respectively the query vector, key vector and value vector obtained after linear mapping for any input feature x;

[0021] Step S302: The pedestrian sketch features obtained by integrating the pedestrian sketch features in step S301 are and pedestrian photo features Input the residual connection into the multi-head cross-modal cross-attention network, learn from the other modality and integrate it into its own modality information; the specific calculation formula of step S302 is as follows:

[0022]

[0023] Among them, MHCA is the implementation of the multi-head cross-modal attention network mechanism, x s and x p is a set of feature vectors for any pair of sketches and photos, The pedestrian sketch features integrated with their own information in step S301 are linearly mapped to obtain the query vector, key vector and value vector. The pedestrian photo features integrated with their own information in step S301 are linearly mapped to obtain the query vector, key vector and value vector. and are the sketch pedestrian features and photo pedestrian features after cross-modal information compensation after multi-head cross-modal cross-attention interaction; d represents the feature dimension;

[0024] Step S303: Compensate the sketch pedestrian features and the photo pedestrian features obtained in step S302 with cross-modal information and The information is integrated in the self-attention network of the input residual connection, and the final feature is residually connected. The specific calculation formula of step S303 is as follows:

[0025]

[0026] Among them, FFN is the implementation of the multi-layer perceptron network mechanism.

[0027] As a further solution of the present invention, in step S4, the pedestrian sketch features at each scale after the information obtained in step S3 is compensated are and pedestrian photo features As a low-level feature, it is the output feature after compensation in the next stage and Supplement details to realize the gradual transfer of shallow features to deep features.

[0028] As a further solution of the present invention, the specific steps of step S4 include:

[0029] Step S401: First, output pedestrian sketch features after compensation in the previous stage and pedestrian photo features Perform downsampling to adjust to the output characteristics after compensation in the next stage and Same size;

[0030] Step S402: downsample the and pedestrian photo features The output features of the next stage and Perform global average pooling GAP, 3*3 convolution operation and sigmoid activation function to generate channel weights, and combine the weights with the corresponding features to strengthen the channel information; learn the importance of each channel feature, and dynamically adjust the output weights of different channels according to these importances, and merge the weights with the corresponding features; the specific calculation method is as follows:

[0031]

[0032]

[0033] Where D(·) is the downsampling operation, σ(·) represents the sigmoid normalization operation, GAP(·) represents the global average pooling operation, Represents matrix dot product;

[0034] Step S403: Enhance the channel information and adjust the features Use 1*1 convolution operation to adjust to a unified channel space. and between, and The feature enhancement is performed step by step, and the specific calculation method is as follows:

[0035]

[0036] in, are different mapping matrices, is the attention shrinkage factor, and They are and The query vector obtained by linear mapping is and They are and The key vector obtained by linear mapping is and V i p yes and The value vector obtained after linear mapping, and It is the feature of the i+1th layer after step-by-step feature enhancement.

[0037] As a further solution of the present invention, in step S5, the results obtained by the pedestrian photo feature extraction network and the pedestrian sketch feature extraction network are constrained by supervision, and the constraint loss is specifically expressed as:

[0038]

[0039] in, and is the characteristic of the final output of S403, W p and W s They are the pedestrian identity classifiers for photo images and sketch images respectively. and yes and The corresponding identity tag, and They are The corresponding cross-modal difficult positive samples and cross-modal difficult negative samples in the same batch, and They are The corresponding cross-modal hard positive samples and cross-modal hard negative samples in the same batch, m is a constant, n is the number of samples in a batch, max(·) is the maximum value function, d(·) is the Euclidean distance function, and the total global classification supervision loss L total for:

[0040]

[0041] As a further solution of the present invention, in step S6, first, the test data is preprocessed and its size is adjusted to match the size of the training data; then, the preprocessed data is input into the trained retrieval model to obtain the final retrieval features; in this process, the query pedestrian sketch image given in the test set and a set of images containing M candidate pedestrian photos are used, and the features of the query pedestrian sketch image and any image in the candidate pedestrian photo image set are extracted through the retrieval model, and the feature similarity between them is calculated, and the pedestrian photos in the candidate pedestrian photo image set are sorted according to this similarity, and the sorting results are evaluated by two objective evaluation indicators, namely the cumulative matching curve CMC and the mean average accuracy mAP.

[0042] Beneficial effects of the present invention:

[0043] 1. The present invention enhances the features between modalities at each stage of the feature extraction process by constructing cross-modal information compensation. Through the unique cross-modal information compensation mechanism, the present invention significantly reduces the inherent difference between the sketch and photo modalities, so that the sketch features and photo features can learn from each other and enrich their own modal information. This cross-modal information fusion effectively improves the accuracy of pedestrian re-identification;

[0044] 2. By constructing a pyramid vision transformer, the present invention can simultaneously obtain multi-level features in pedestrian sketches and photos, including detailed local features and global features of the overall contour. This multi-scale feature extraction method helps to more comprehensively describe the identity information of pedestrians;

[0045] 3. The step-by-step feature enhancement technology of the present invention can deeply learn the subtle differences and internal connections between step-by-step features, thereby further enhancing the expressiveness of information within the modality; this technology enables better use of the hierarchical relationship and correlation between features in subsequent processing;

[0046] 4. The cross-modal information compensation of the present invention is combined with the step-by-step feature enhancement within the modality, which not only effectively solves the recognition problem caused by modal differences, but also improves the ability of multi-scale features to represent pedestrian identity information. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 A flow chart of a sketch pedestrian re-identification method combining step-by-step feature enhancement and cross-modal information compensation provided by the present invention.

[0048] Figure 2 A model framework diagram of a sketch pedestrian re-identification method combining step-by-step feature enhancement and cross-modal information compensation provided by the present invention.

[0049] Figure 3 This is a framework diagram of the pyramid vision transformer (PVTv2) model provided by the present invention.

[0050] Figure 4 This is a framework diagram of the cross-modal information compensation space model provided by the present invention.

[0051] Figure 5 A framework diagram of the step-by-step feature enhancement model provided by the present invention;

[0052] Figure 6 A schematic diagram comparing the top 10 search results of the method of the present invention (right) and the baseline method (left); red boxes represent incorrect search results, and green boxes represent correct search results.

[0053] Figure 7 Schematic diagram of the comparison of attention heat maps between the method of the present invention and the baseline method; (a) and (d) represent the original photo image and the corresponding sketch image, (b) and (e) represent the areas focused on by the baseline method, and (c) and (f) represent the areas focused on by our method. DETAILED DESCRIPTION

[0054] When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementations described in the following examples do not represent all implementations consistent with the present invention. They are only examples of methods consistent with some aspects of the present invention as detailed in the claims.

[0055] Example 1: Figure 1-Figure 7As shown, a sketch pedestrian re-identification method combining step-by-step feature enhancement and cross-modal information compensation is provided. The specific steps of the method are as follows to illustrate the embodiment in detail;

[0056] Step S1: Obtain pedestrian photo images and pedestrian sketch images, and construct training sets and test sets to perform image preprocessing; preprocessing the collected data sample images includes: normalizing the pedestrian photo images and pedestrian sketch images in the training set and the test set, grayscale transformation, random horizontal flipping, padding, cropping and erasing, etc.

[0057] Step S2: inputting the preprocessed pedestrian photo image and pedestrian sketch image into the pedestrian photo feature extraction network and the pedestrian sketch feature extraction network to extract pedestrian photo features and pedestrian sketch features respectively;

[0058] As a further solution of the present invention, step S2 includes the following:

[0059] Step S201, backbone network construction: The backbone networks of the pedestrian photo feature extraction network and the pedestrian sketch feature extraction network are both constructed on the basis of the pyramid vision transformer backbone PVTv2; Figure 3 As shown in the figure, the final classification features are guaranteed to have long-range dependencies, and multi-scale features are learned to reduce the scale effect caused by the change in the distance between the target and the surveillance camera;

[0060] Step S202, dual-stream feature extraction: the pre-processed pedestrian photo image and pedestrian sketch image are respectively input into the pedestrian photo feature extraction network and the pedestrian sketch feature extraction network, and features are extracted from pedestrian samples of the two modalities to obtain pedestrian photo features and pedestrian sketch features.

[0061] As a further solution of the present invention, in step S202, the pedestrian photo feature extraction network and the pedestrian sketch feature extraction network adopt a parallel structure, and the pedestrian photo feature extraction network and the pedestrian sketch feature extraction network are respectively composed of four stages, each stage generates features of different sizes, and the pedestrian sketch features and pedestrian photo features obtained in each stage are expressed as and

[0062] Step S3: Construct cross-modal information compensation combination space, such as Figure 4 As shown, in this space, the modal distance between the pedestrian photo feature and the pedestrian sketch feature is shortened, the common information within the modality is learned and information compensation is performed, so as to achieve the purpose of reducing the modal difference;

[0063] As a further solution of the present invention, the specific steps of step S3 are as follows:

[0064] Step S301: represent the pedestrian sketch features and pedestrian photo features obtained at each stage and The input is integrated into a multi-head self-attention network with residual connections to ensure the transmission of self-modal information. In feature integration, residual connections can ensure that the original feature information is not lost during the transmission process. In addition, the self-attention mechanism allows the model to learn the dependencies between elements within the input sequence and assign different weights to each element. In feature integration, this can be used to emphasize features that are important to the final task and suppress irrelevant features. The specific process calculation formula is as follows:

[0065]

[0066] Among them, MHSA is the implementation of the multi-head self-attention mechanism, x T represents the matrix transpose operation, LN is the LayerNormalization operation, i.e., the layer normalization operation; x q , x k , x v They are respectively the query vector, key vector and value vector obtained after linear mapping for any input feature x;

[0067] Step S302: The pedestrian sketch features obtained by integrating the pedestrian sketch features in step S301 are and pedestrian photo features In the multi-head cross-modal attention network with input residual connection, learn from the other modality and integrate it into the own modality information to achieve the purpose of compensating the own modality information; the specific calculation formula of step S302 is as follows:

[0068]

[0069] Among them, MHCA is the implementation of the multi-head cross-modal attention network mechanism, x s and x p is a set of feature vectors for any pair of sketches and photos, The pedestrian sketch features integrated with their own information in step S301 are linearly mapped to obtain the query vector, key vector and value vector. The pedestrian photo features integrated with their own information in step S301 are linearly mapped to obtain the query vector, key vector and value vector. and are the sketch pedestrian features and photo pedestrian features after cross-modal information compensation after multi-head cross-modal cross-attention interaction; d represents the feature dimension;

[0070] Step S303: Compensate the sketch pedestrian features and the photo pedestrian features obtained in step S302 with cross-modal information and The information is integrated in the self-attention network of the input residual connection, and the final feature is residually connected. The specific calculation formula of step S303 is as follows:

[0071]

[0072] FFN is the implementation of the multi-layer perceptron network mechanism. The compensated sketch pedestrian features and photo pedestrian features are used as the input of the next stage to continue forward propagation;

[0073] Step S4: Enhance the features after information compensation and the features at the next scale step by step, and learn the fine-grained feature representation of the features to gradually transfer the shallow features to the deep features, such as Figure 5 As shown, the final feature representation is obtained;

[0074] As a further solution of the present invention, in step S4, the pedestrian sketch features at each scale after the information obtained in step S3 is compensated are and pedestrian photo features As a low-level feature, it is the output feature after compensation in the next stage and Supplement details to realize the gradual transfer of shallow features to deep features.

[0075] As a further solution of the present invention, the specific steps of step S4 include:

[0076] Step S401: First, output pedestrian sketch features after compensation in the previous stage and pedestrian photo features Perform downsampling to adjust to the output characteristics after compensation in the next stage and Same size;

[0077] Step S402: downsample the and pedestrian photo features The output features of the next stage and Perform global average pooling GAP, 3*3 convolution operation and sigmoid activation function to generate channel weights, and combine the weights with the corresponding features to strengthen the channel information; learn the importance of each channel feature, and dynamically adjust the output weights of different channels according to these importances, and merge the weights with the corresponding features; the specific calculation method is as follows:

[0078]

[0079]

[0080] Where D(·) is the downsampling operation, σ(·) represents the sigmoid normalization operation, GAP(·) represents the global average pooling operation, Represents matrix dot product;

[0081] Step S403: Enhance the channel information and adjust the features Use 1*1 convolution operation to adjust to a unified channel space. and between, and The feature enhancement is performed step by step, and the specific calculation method is as follows:

[0082]

[0083] in, are different mapping matrices, is the attention shrinkage factor, and They are and The query vector obtained by linear mapping is and They are and The key vector obtained by linear mapping is and yes and The value vector obtained after linear mapping, and It is the feature of the i+1th layer after step-by-step feature enhancement.

[0084] Step S5: using the final feature representation to train the retrieval model, designing a joint optimization algorithm to optimize the retrieval model to achieve the best network model parameters, and using the pedestrian sketch image to find the corresponding pedestrian photo image through the optimized retrieval model;

[0085] The step S5 includes: inputting the final feature representation into the identity classifier to supervise and constrain the extracted identity features. and The pedestrian classifiers under the corresponding modes are sent to obtain the output results of the pedestrian classifiers under the two modes, and the results are sorted to obtain the maximum classification value;

[0086] As a further solution of the present invention, in step S5, the results obtained by the pedestrian photo feature extraction network and the pedestrian sketch feature extraction network are constrained by supervision, and the constraint loss is specifically expressed as:

[0087]

[0088] in, and is the characteristic of the final output of S403, W p and W s They are the pedestrian identity classifiers for photo images and sketch images respectively. and yes and The corresponding identity tag, and They are The corresponding cross-modal difficult positive samples and cross-modal difficult negative samples in the same batch, and They are The corresponding cross-modal hard positive samples and cross-modal hard negative samples in the same batch, m is a constant, n is the number of samples in a batch, max(·) is the maximum value function, d(·) is the Euclidean distance function, and the total global classification supervision loss L total for:

[0089]

[0090] Step S6: Evaluate the performance of the retrieval model and perform iterative optimization accordingly.

[0091] As a further solution of the present invention, in step S6, first, the test data is preprocessed and its size is adjusted to match the size of the training data; then, the preprocessed data is input into the trained retrieval model to obtain the final retrieval features; in this process, the query pedestrian sketch image given in the test set and a set of images containing M candidate pedestrian photos are used, and the features of the query pedestrian sketch image and any image in the candidate pedestrian photo image set are extracted through the retrieval model, and the feature similarity between them is calculated, and the pedestrian photos in the candidate pedestrian photo image set are sorted according to this similarity, and the sorting results are evaluated by two objective evaluation indicators, namely the cumulative matching curve CMC and the mean average accuracy mAP. The ultimate goal is to ensure that the photo image obtained by the query can be as close as possible to the real matching item of its corresponding sketch image in the sorting result.

[0092] In order to illustrate the effect of the present invention, the present invention has done the following experiments for verification:

[0093] The PKU Sketch Re-ID dataset was used for training and testing in the experiment. The dataset contains photos of 200 pedestrians, each of which has a corresponding sketch image and two photos. These photos were taken by two cameras with cross views during the day, ensuring the diversity and authenticity of the data. In order to ensure that each photo focuses on a specific pedestrian, the original image (or video frame) was manually cropped. In terms of the production of sketch images, this dataset adopts a unique approach. The sketches were drawn by five professional painters based on the descriptions of eyewitnesses. Each painter has his own unique painting style. This diversity of painting styles not only increases the richness of the dataset.

[0094] To evaluate the accuracy of retrieval, Rank-K (K value is 1, 5, 10, etc.) and mean average precision (mAP) are used as evaluation indicators.

[0095] In order to mine modality-invariant feature representations, Pang et al. used a cross-domain adversarial framework to filter domain-sensitive information to learn domain-invariant features and reduce modality differences to improve model retrieval performance. Gui et al. conducted an in-depth study of the multi-level feature representations of sketches and photo images. Unlike these methods, the method of the present invention explores an information compensation mechanism, including step-by-step enhancement and cross-modal information compensation mechanisms, to significantly improve performance. Compared with the mainstream methods in recent years, the method of the present invention has shown significant advantages. The method evaluation results on the PKU Sketch dataset are shown in Table 1.

[0096] Table 1 Comparison of research methods based on PKU Sketch Re-ID dataset

[0097]

[0098]

[0099] Note: Bold indicates the optimal value, — indicates that the data is not provided in the original text;

[0100] Visual search results Figure 6 The figure shows some results of searching on the PKU-Sketch Re-ID dataset. Four query sample images are randomly selected and the top ten photo images are retrieved from the gallery. The left side shows the results of the baseline method, while the right side shows the results of the method proposed by the present invention. By comparison, it can be clearly seen that compared with the baseline method, the method of the present invention can more accurately capture the core features of the query sample, thereby accurately retrieving similar images in the gallery.

[0101] On the PKU-Sketch Re-ID dataset, the baseline model and the model proposed in this paper were trained respectively, and Score-CAM was applied to generate the corresponding attention maps, such as Figure 7 As shown in the figure. By comparing the attention maps of the two, significant differences can be observed. Although the attention map of the baseline model can capture some basic features of the sketch image, its attention area is often scattered and lacks clear discriminability. This makes it difficult for the model to identify the similarities and differences between different sketch images, affecting the accuracy of retrieval.

[0102] The above is a detailed description of the specific implementation method of the present invention in conjunction with the accompanying drawings, but the above is only a specific implementation example of the present invention, rather than limiting the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention are included in the protection scope of the present invention.

Claims

1. A sketch person re-identification method combining step-by-step feature enhancement and cross-modal information compensation, characterized in that: The method comprises: Step S1: Obtain pedestrian photo images and pedestrian sketch images, construct training sets and test sets, and perform image preprocessing; Step S2: inputting the preprocessed pedestrian photo image and pedestrian sketch image into the pedestrian photo feature extraction network and the pedestrian sketch feature extraction network to extract pedestrian photo features and pedestrian sketch features respectively; Step S3: construct a cross-modal information compensation combination space, in which the modal distance between pedestrian photo features and pedestrian sketch features is shortened, and the common information within the modality is learned and information compensation is performed; Step S4: The features after information compensation and the features at the next scale are enhanced step by step, and the fine-grained feature representation of the learning features is used to gradually transfer the shallow features to the deep features to obtain the final feature representation; Step S5: using the final feature representation to train the retrieval model, designing a joint optimization algorithm to optimize the retrieval model to achieve the best network model parameters, and using the pedestrian sketch image to find the corresponding pedestrian photo image through the optimized retrieval model; Step S6: Evaluate the performance of the retrieval model and perform iterative optimization accordingly.

2. The sketch person re-identification method combining step-by-step feature enhancement and cross-modal information compensation according to claim 1 is characterized in that: In the step S1, preprocessing the collected data sample images includes: normalizing, grayscale transforming, randomly horizontally flipping, filling, cropping and erasing the pedestrian photo images and pedestrian sketch images in the training set and the test set.

3. The sketch person re-identification method combining step-by-step feature enhancement and cross-modal information compensation according to claim 1 is characterized in that: The step S2 comprises the following: Step S201, backbone network construction: the backbone networks of pedestrian photo feature extraction network and pedestrian sketch feature extraction network are both constructed on the basis of the pyramid vision transformer backbone PVTv2; Step S202, dual-stream feature extraction: the pre-processed pedestrian photo image and pedestrian sketch image are respectively input into the pedestrian photo feature extraction network and the pedestrian sketch feature extraction network, and features are extracted from pedestrian samples of the two modalities to obtain pedestrian photo features and pedestrian sketch features.

4. The method for person re-identification from sketches combining step-by-step feature enhancement and cross-modal information compensation according to claim 3 is characterized in that: In step S202, the pedestrian photo feature extraction network and the pedestrian sketch feature extraction network adopt a parallel structure. The pedestrian photo feature extraction network and the pedestrian sketch feature extraction network are respectively composed of four stages, each of which generates features of different sizes. The pedestrian sketch features and pedestrian photo features obtained in each stage are expressed as and 5. The sketch person re-identification method combining step-by-step feature enhancement and cross-modal information compensation according to claim 1 is characterized in that: The specific steps of step S3 are as follows: Step S301: represent the pedestrian sketch features and pedestrian photo features obtained at each stage and Input into the multi-head self-attention network with residual connection for information integration. The specific process calculation formula is as follows: Among them, MHSA is the implementation of the multi-head self-attention mechanism, x T represents the matrix transpose operation, LN is the LayerNormalization operation, i.e., the layer normalization operation; x q , x k , x v They are respectively the query vector, key vector and value vector obtained after linear mapping for any input feature x; Step S302: The pedestrian sketch features obtained by integrating the pedestrian sketch features in step S301 are and pedestrian photo features Input the residual connection into the multi-head cross-modal cross-attention network, learn from the other modality and integrate it into its own modality information; the specific calculation formula of step S302 is as follows: Among them, MHCA is the implementation of the multi-head cross-modal attention network mechanism, x s and x p is a set of feature vectors for any pair of sketches and photos, The pedestrian sketch features integrated with their own information in step S301 are linearly mapped to obtain the query vector, key vector and value vector. The pedestrian photo features integrated with their own information in step S301 are linearly mapped to obtain the query vector, key vector and value vector. and are the sketch pedestrian features and photo pedestrian features after cross-modal information compensation after multi-head cross-modal cross-attention interaction; d represents the feature dimension; Step S303: Compensate the sketch pedestrian features and the photo pedestrian features obtained in step S302 with cross-modal information and The information is integrated in the self-attention network of the input residual connection, and the final feature is residually connected. The specific calculation formula of step S303 is as follows: Among them, FFN is the implementation of the multi-layer perceptron network mechanism.

6. The method for person re-identification from sketches combining step-by-step feature enhancement and cross-modal information compensation according to claim 1, characterized in that: In step S4, the pedestrian sketch features at each scale after the information obtained in step S3 is compensated are and pedestrian photo features As a low-level feature, it is the output feature after compensation in the next stage and Supplement details to realize the gradual transfer of shallow features to deep features.

7. The method for person re-identification from sketches combining step-by-step feature enhancement and cross-modal information compensation according to claim 6, characterized in that: The specific steps of step S4 include: Step S401: First, output pedestrian sketch features after compensation in the previous stage and pedestrian photo features Perform downsampling to adjust to the output characteristics after compensation in the next stage and Same size; Step S402: downsample the and pedestrian photo features The output features of the next stage and Perform global average pooling GAP, 3*3 convolution operation and sigmoid activation function to generate channel weights, and combine the weights with the corresponding features to strengthen the channel information; learn the importance of each channel feature, and dynamically adjust the output weights of different channels according to these importances, and merge the weights with the corresponding features; the specific calculation method is as follows: Where D(·) is the downsampling operation, σ(·) represents the sigmoid normalization operation, GAP(·) represents the global average pooling operation, Represents matrix dot product; Step S403: Enhance the channel information and adjust the features Use 1*1 convolution operation to adjust to a unified channel space. and between, and The feature enhancement is performed step by step, and the specific calculation method is as follows: in, are different mapping matrices, is the attention shrinkage factor, and They are and The query vector obtained by linear mapping is and They are and The key vector obtained by linear mapping is and V i p yes and The value vector obtained after linear mapping, and It is the feature of the i+1th layer after step-by-step feature enhancement.

8. The method for person re-identification from sketches combining step-by-step feature enhancement and cross-modal information compensation according to claim 7, characterized in that: In step S5, the results obtained by the pedestrian photo feature extraction network and the pedestrian sketch feature extraction network are constrained by supervision, and the constraint loss is specifically expressed as: in, and is the characteristic of the final output of S403, W p and W s They are the pedestrian identity classifiers for photo images and sketch images respectively. and yes and The corresponding identity tag, and They are The corresponding cross-modal difficult positive samples and cross-modal difficult negative samples in the same batch, and They are The corresponding cross-modal hard positive samples and cross-modal hard negative samples in the same batch, m is a constant, n is the number of samples in a batch, max(·) is the maximum value function, d(·) is the Euclidean distance function, and the total global classification supervision loss L total for:

9. The method for person re-identification from sketches combining step-by-step feature enhancement and cross-modal information compensation according to claim 1, characterized in that: In step S6, first, the test data is preprocessed to adjust its size to match the size of the training data; then, the preprocessed data is input into the trained retrieval model to obtain the final retrieval features; In this process, the query pedestrian sketch image given in the test set and a set of M candidate pedestrian photos are used. The retrieval model is used to extract features of the query pedestrian sketch image and any image in the candidate pedestrian photo image set, and the feature similarity between them is calculated. The pedestrian photos in the candidate pedestrian photo image set are sorted based on this similarity, and the two objective evaluation indicators, cumulative matching curve CMC and mean average accuracy mAP, are used to evaluate the sorting results.

Images