Sketch pedestrian re-identification method and device based on cross-modal reliable subjective features

Abstract

The present invention discloses a method and device for re-identifying a person in sketch based on reliable cross-modal subjective features, and relates to the technical field of character sketch image processing. The method and device for re-identifying a person in sketch based on reliable cross-modal subjective features mainly include: obtaining a training set, a validation set, and a test set according to the character RGB and sketch images; constructing a dual-path feature enhancement network using a residual network and training using the training set to obtain a trained dual-path feature enhancement network; and obtaining a character sketch retrieval result using the trained dual-path feature enhancement network according to the test set. Implementing the method and device for re-identifying a person in sketch based on reliable cross-modal subjective features provided by the present invention can improve the accuracy and robustness of cross-modal retrieval tasks.

Description

Technical Field

[0001] The present invention relates to the technical field of character sketch image processing, and more specifically, to a method and device for re-identifying a person in a sketch based on cross-modal reliable subjective features. Background Art

[0002] Person re-identification (ReID) is one of the most important technologies in intelligent surveillance systems. It aims to identify the same pedestrian by comparing images taken by different cameras. Traditional ReID methods mainly rely on RGB images captured from visible light cameras to identify pedestrians based on their appearance features. However, in some crime scenes, criminals often do not appear within the coverage of the camera, making it impossible to obtain their photo information. In this case, the police usually rely on the description of eyewitnesses and draw a sketch of the criminal based on it. Sketch images usually contain the key appearance features of the criminal, such as facial contours and body features, which makes sketch-based person re-identification a potential solution.

[0003] Sketch person re-identification is a cross-modal re-identification task, which aims to identify a specific target by matching a hand-drawn sketch image with a photo in a gallery. However, this task faces great challenges due to the significant differences between sketches and photos in terms of imaging principles, detail preservation, and information expression. Sketch images are often abstract and simple, lacking rich visual information such as color and texture, and only retain the basic outline or structural information of the object, while photos contain more detailed appearance features. Eyewitness descriptions of the suspect's appearance are often incomplete and may be affected by subjective biases. Differences in viewpoints and perceptual nuances between eyewitnesses may lead to conflicting descriptions, making it more complicated to accurately portray the suspect. In addition, sketches may be biased due to the artist's personal aesthetic preferences and artistic styles. Therefore, understanding and dealing with these subjective factors is crucial for accurately identifying and matching features in sketches.

[0004] Some existing research works are dedicated to alleviating this cross-modal difference, mainly focusing on modal alignment through feature extraction and metric learning. Some methods propose cross-modal embedding learning, trying to map sketches and photos to a shared feature space for easy matching. However, these methods show certain limitations when faced with sketch images with diverse styles. In particular, due to the subjectivity and artistic style differences of different painters when drawing sketches, existing methods are difficult to fully capture the key identification features in sketch images. In addition, although some studies use generative adversarial networks (GANs) for image transformation, such as converting sketches into photos to reduce modal differences, these generative methods still have problems with generated noise and data domain bias.

[0005] With the development of technology, many sketch pedestrian re-identification methods based on deep learning have emerged in recent years. These methods mainly improve the alignment effect of cross-modal features by designing a dual-stream network structure or introducing a new loss function. The patent "CN117953539A Sketch Pedestrian Re-identification Method with Cross-modal Prompt Adjustment" discloses a sketch pedestrian re-identification method with cross-modal prompt adjustment. The invention guides the shallow frozen pre-trained model to perform sketch retrieval tasks from the perspective of sample prompts and feature enhancement, while taking into account the problem of large gaps between modalities.

[0006] In previous studies, researchers proposed a method that uses a two-stream network architecture that processes sketch and photo modalities simultaneously to capture complementary information. Although existing sketch person re-identification methods have made significant progress in the field of deep learning, have shown promising results, and have begun to focus on the impact of subjectivity on recognition tasks, these methods still face multiple problems, such as ignoring the objectivity hidden in subjectivity, large modality differences, and insufficient model generalization capabilities. They are still insufficient in capturing subjectivity and style variability in sketches. They usually ignore subjective factors in sketches, such as artistic style and stroke changes, which are crucial for accurate matching.

[0007] Sketch images usually have diverse artistic styles, and different artistic perceptions lead to varied artistic tendencies. However, even under these different artistic expressions, there are still some reliable common features in sketch images, such as the target person's hairstyle, clothing outline, etc. These features are universally cognizable and can be used to enhance the accuracy of cross-modal matching. Therefore, there is an increasing need for more complex models that can explicitly model the subjective characteristics of sketches and learn robust representations that can bridge the gap between different modes. To this end, more intelligent and flexible algorithms are urgently needed to better balance the subjective information differences between sketches and photos, and improve the accuracy and robustness of sketch pedestrian re-identification. Summary of the invention

[0008] The purpose of the present invention is to provide a method and device for sketch pedestrian re-identification based on cross-modal reliable subjective features, which can improve the accuracy and robustness of cross-modal retrieval tasks.

[0009] The present invention provides a sketch pedestrian re-identification method based on cross-modal reliable subjective features, comprising the following steps: S01: acquiring a person RGB and a sketch image, preprocessing the person RGB and the sketch image to obtain a person RGB and sketch image data set, and dividing the person RGB and sketch image data set into a training set, a verification set and a test set; S02: constructing a dual-path feature enhancement network by using a residual network; S03: training the dual-path feature enhancement network by using the training set to obtain a trained dual-path feature enhancement network; S04: obtaining a person sketch retrieval result by using the trained dual-path feature enhancement network according to the test set.

[0010] Furthermore, the preprocessing of the sketch pedestrian re-identification method based on cross-modal reliable subjective features includes: resizing, graying and normalization.

[0011] Furthermore, the dual-path feature enhancement network of the above-mentioned sketch pedestrian re-identification method based on cross-modal reliable subjective features takes ResNet50 as the backbone network, and connects a cross-modal feature enhancement module in series after each residual module Block of ResNet50; the cross-modal feature enhancement module is used to extract features and further integrate local and global information; the cross-modal feature enhancement module includes an average pooling layer, a first fully connected layer, a linear rectification function ReLU, a second fully connected layer, a Sigmoid function and an element-by-element multiplication operation connected in series in sequence; wherein, the input end of the average pooling layer is connected in parallel with the input end of the element-by-element multiplication operation.

[0012] Furthermore, step S03 of the above-mentioned sketch pedestrian re-identification method based on cross-modal reliable subjective features specifically includes: S031: setting the training parameters of the dual-path feature enhancement network and adjusting the weights, and inputting the training set and the verification set into the dual-path feature enhancement network; S032: using the cross-modal feature enhancement module to extract the input photo features and the input key features into RGB key features and sketch key features respectively; S033: constructing a loss function based on the RGB key features and the sketch key features; S034: repeating steps S032-S033 to a preset number of times, calculating the loss function, and taking the model with the smallest loss function as the trained dual-path feature enhancement network.

[0013] Furthermore, step S032 of the above-mentioned sketch pedestrian re-identification method based on cross-modal reliable subjective features specifically includes: using a cross-modal feature enhancement module to extract input photo features and input key features into RGB key features and sketch key features, respectively, as shown in formula:

[0014] ,

[0015] ,

[0016] in, represents the feature vector of the cth class or channel, is the eigenvector of the cth class obtained after transformation, represents the Sigmoid activation function, represents the weight matrix of the first fully connected layer, represents the weight matrix of the second fully connected layer, represents the compressed channel feature representation, Denotes the set of transformed feature vectors representing all categories.

[0017] Furthermore, step S033 of the above-mentioned sketch pedestrian re-identification method based on cross-modal reliable subjective features specifically includes: constructing a loss function according to the RGB key features and the sketch key features, such as the formula:

[0018] ,

[0019] ,

[0020] ,

[0021] in, is the loss function, is the average distance to other positive samples within and across modalities, is the constraint formula, is the total number of samples, represents the number of samples in a mini-batch, is the distance to other positive samples within the modality, is the distance to other positive samples under cross-modality, Indicates The feature vector of the sample sketch image, Indicates The feature vector of the RGB image, Indicates The feature vector of the samples, Indicates The feature vector of the samples, It means to perform mean operation on a set of data. Indicates that it belongs to the category The characteristic center vector of the sample is Indicates category The average distance of negative samples.

[0022] The present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-mentioned sketch pedestrian re-identification method based on cross-modal reliable subjective features.

[0023] The present invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the above-mentioned sketch pedestrian re-identification method based on cross-modal reliable subjective features are implemented.

[0024] The present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of the above-mentioned sketch pedestrian re-identification method based on cross-modal reliable subjective features.

[0025] The method and device for re-identifying a person from sketch based on reliable cross-modal subjective features provided by the present invention have the following beneficial effects:

[0026] The present invention uses a standardized preprocessed data set to ensure the input consistency of the model and reduce the impact of data differences; the dual-path feature enhancement network of the present invention combines the ResNet50 backbone network and the cross-modal feature enhancement module. Module), by extracting features from RGB images and sketch images, and combining the fusion of global and local features, it is possible to capture key features under different modalities, improve the interactivity and consistency between features of different modalities, and improve the cross-modal retrieval performance in the sketch pedestrian re-identification task; the present invention can better handle the differences between features of different modalities by optimizing the distance between intra-modal and cross-modal features, while enhancing the utilization of reliable features and reducing the impact of unreliable features; the present invention effectively improves the pedestrian re-identification performance in different scenarios and conditions by aligning and optimizing features throughout the training process, and demonstrates a strong generalization ability; the present invention combines feature enhancement technology to fully explore reliable subjective features between modalities, and through cross-modal feature alignment and enhancement mechanism, enables common features under different artistic styles to be more effectively aligned, thereby minimizing the differences between the two modalities. Specifically, the present invention utilizes soft alignment technology and a bidirectional alignment mechanism, through a recognizable target center loss (Recognizable TargetCentroidLoss, The introduction of cross-modal retrieval algorithm (RTCL) ensures the matching of RGB image and sketch image features in the same space, ensures the effective alignment of features of different modalities, and reduces the inconsistency between features, especially between sketch images of different artistic styles and diversified RGB images. This invention significantly improves the average precision of cross-modal retrieval tasks, improves the robustness of cross-modal retrieval, and improves the accuracy of retrieval. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] The present invention will be further described below with reference to the accompanying drawings and embodiments, in which:

[0028] Figure 1 It is a flow chart of a sketch pedestrian re-identification method based on cross-modal reliable subjective features provided by the present invention;

[0029] Figure 2 It is a schematic diagram of the dual-path feature enhancement network structure provided by the present invention;

[0030] Figure 3 It is a structural block diagram of the computer device provided by the present invention. DETAILED DESCRIPTION

[0031] In order to have a clearer understanding of the technical features, purposes and effects of the present invention, specific embodiments of the present invention are now described in detail with reference to the accompanying drawings.

[0032] Figure 1 A schematic diagram of a method for re-identifying a person from a sketch based on reliable cross-modal subjective features of this embodiment is shown. In this embodiment, the method for re-identifying a person from a sketch based on reliable cross-modal subjective features includes the following steps:

[0033] S01: Obtain RGB and sketch images of a person, preprocess the RGB and sketch images of the person, obtain a data set of the RGB and sketch images of the person, and divide the data set of the RGB and sketch images of the person into a training set and a test set;

[0034] Specifically, preprocessing includes: resizing, grayscale, and normalization;

[0035] It should be noted that the RGB and sketch images of people are from the MARKET-SKETCH-1K dataset and the PKUSketchRe-ID dataset in the public datasets; the sketch images drawn by six painters from the MARKET-SKETCH-1K dataset and the sketch images drawn by five painters from the PKU SketchRe-ID dataset as well as the RGB images in the two datasets are standardized and preprocessed, including resizing, grayscale, normalization and other operations to meet the input requirements of the model; the preprocessed datasets are divided into training sets and test sets, and samples are allocated in a certain proportion to ensure that the training set is used for model learning and the test set is used for model performance evaluation;

[0036] S02: Use residual network to build dual-path feature enhancement network;

[0037] Specifically, the dual-path feature enhancement network uses ResNet50 as the backbone network, and a cross-modal feature enhancement module is connected in series after each residual module Block of ResNet50; the cross-modal feature enhancement module is used to extract features and further integrate local and global information; the cross-modal feature enhancement module includes an average pooling layer, a first fully connected layer, a linear rectification function ReLU, a second fully connected layer, a Sigmoid function, and an element-by-element multiplication operation connected in series in sequence; wherein the input end of the average pooling layer is connected in parallel with the input end of the element-by-element multiplication operation;

[0038] It should be noted that the Dual-Path Feature-Enhance Network (DPFEN) is used for character sketch retrieval; Figure 2 The figure shows the schematic diagram of the dual-path feature enhancement network structure. The dual-path feature enhancement network uses ResNet50 as the backbone network and integrates a cross-modal feature enhancement module (CMFEM) after each block. The cross-modal feature enhancement module first performs a global average pooling operation to transform the input feature map Compress in space and generate channel descriptor ; Then, the module uses two fully connected layers to descriptor Perform nonlinear processing to obtain the channel weight coefficient ; Finally, the weight coefficient With the original feature map Multiply channel by channel to generate a weighted feature map ;

[0039] It should be noted that the dual-path feature enhancement network extracts features after each layer of the block, and further fuses local and global information through the cross-modal feature enhancement model; these feature-enhanced RGB and sketch features will be input into the Recognisable Target Centeroid Loss (RTCL) in the final stage of the dual-path feature enhancement network. The Recognisable Target Centeroid Loss is used to optimize the feature expression of the extracted RGB and sketch images. The Recognisable Target Centeroid Loss RTCL calculates the distance between these extracted features and the center of the target category; by minimizing the distance between the features of different modalities, namely RGB and sketch, and the category center, RTCL ensures that these features can be better aligned in cross-modal and different artistic style scenarios, and improves the recognition performance of the model; Among them, the Recognisable Target Centeroid Loss RTCL first calculates the average distance of the features within the same modality and the average distance between different modalities. Specifically, the channel descriptor of the feature map is first calculated by global average pooling, and then the average distance between the descriptor and other positive samples in the same modality is calculated. and the average distance to positive samples of different modalities ; Then calculate the feature center of each identity, and based on this, calculate the average distance between the feature center and the negative sample to form a dynamic boundary ; Finally, RTCL combines this distance information to enhance the utilization of reliable features by minimizing the distance difference between modalities and optimizing feature alignment, while reducing the impact of unreliable features, thereby improving the effect of cross-modal retrieval with subjective information;

[0040] S03: training the dual-path feature enhancement network using the training set to obtain a trained dual-path feature enhancement network;

[0041] Specifically, step S03 includes:

[0042] S031: Setting training parameters of the dual-path feature enhancement network and adjusting weights, and inputting the training set and the validation set into the dual-path feature enhancement network;

[0043] S032: using a cross-modal feature enhancement module to extract input photo features and input key features into RGB key features and sketch key features respectively;

[0044] Specifically, step S032 includes: using a cross-modal feature enhancement module to extract input photo features and input key features into RGB key features and sketch key features, respectively, as shown in the formula:

[0045] ,

[0046] ,

[0047] in, represents the feature vector of the cth class or channel, is the eigenvector of the cth class obtained after transformation, represents the Sigmoid activation function, represents the weight matrix of the first fully connected layer, represents the weight matrix of the second fully connected layer, represents the compressed channel feature representation, represents the set of transformed feature vectors representing all categories;

[0048] S033: Construct a loss function based on RGB key features and sketch key features;

[0049] Specifically, step S033 includes: constructing a loss function according to the RGB key features and the sketch key features, such as the formula:

[0050] ,

[0051] ,

[0052] ,

[0053] in, is the loss function, is the average distance to other positive samples within and across modalities, is the constraint formula, is the total number of samples, represents the number of samples in a mini-batch, is the distance to other positive samples within the modality, is the distance to other positive samples under cross-modality, Indicates The feature vector of the sample sketch image, Indicates The feature vector of the RGB image, Indicates The feature vector of the samples, Indicates The feature vector of the samples, It means to perform mean operation on a set of data. Indicates that it belongs to the category The characteristic center vector of the sample is Indicates category The average distance of negative samples;

[0054] It should be noted that the average distance between the model and other positive samples within and across modalities is calculated. In the distance optimization process, the inconsistent features of the two modes due to subjective understanding will be suppressed, while the more reliable features in the two modes will be enhanced. The constraint formula is , the loss function is calculated to enhance the utilization of reliable subjective features;

[0055] S034: repeat steps S032-S033 to a preset number of times, calculate the loss function, and use the model with the smallest loss function as the trained dual-path feature enhancement network;

[0056] S04: Based on the test set, the trained dual-path feature enhancement network is used to obtain the character sketch retrieval results;

[0057] Specifically, step S04 includes:

[0058] The image to be detected in the test set is input into the trained dual-path feature enhancement network, and then passes through the four-layer cross-modal feature enhancement module for feature extraction.

[0059] When processing multiple sketch images at the same time, the features extracted by the cross-modal feature enhancement module are fused through non-local features to make the RGB key features and sketch key features Map to the same feature space dimension to ensure that the features of the RGB image and the sketch image can be compared in a unified feature space; Based on feature alignment, the recognizable target center loss RTCL is used to optimize the consistency features in cross-modality; Bidirectional alignment is performed. In the whole process, the alignment of photo features to sketch features and the alignment of sketch features to photo features are symmetrical. By calculating the cross-modal similarity and matching probability, and then using this information for soft alignment, the final result is the alignment of the two modal features in the same space. The reconstructed features are:

[0060] ,

[0061] Among them, M represents the character segmentation map. The feature alignment from photo to sketch is similar. The reconstruction features from photo to sketch and sketch to photo are obtained for comprehensive analysis to obtain the character retrieval results.

[0062] It should be noted that in step S04, the key features are first calculated and Similarity:

[0063] ,

[0064] in, and Respectively represent the RGB image features and sketch image features at positions b and c, represents the L2 normalization function; then calculate the matching probability:

[0065] ,

[0066] Finally, the softening function is calculated:

[0067] .

[0068] In some embodiments, the above-mentioned sketch pedestrian re-identification method based on cross-modal reliable subjective features can also be implemented in the following way.

[0069] In this embodiment, the sketch pedestrian re-identification method based on cross-modal reliable subjective features includes the following steps:

[0070] S1: Obtain a public dataset of RGB and sketch images of a person, preprocess the public dataset of RGB and sketch images of a person, obtain a dataset of RGB and sketch images of a person, and divide the dataset of RGB and sketch images of a person into a training set and a test set;

[0071] S2: Construct a dual-path feature enhancement network for character sketch retrieval (Dual-Path Feature-EnhanceNetwork);

[0072] S3: Train the dual-path feature enhancement network through the training set to obtain a trained model;

[0073] S4: Input the test set into the trained model to obtain the character sketch retrieval results.

[0074] Specifically, step S1 includes: S11: obtaining the MARKET-SKETCH-1K dataset and the PKUSketchRe-ID dataset from the public datasets of character RGB and sketch images; S12: performing standardized preprocessing on the sketch images drawn by six painters from the MARKET-SKETCH-1K dataset, the sketch images drawn by five painters from the PKU SketchRe-ID dataset, and the RGB images in the two datasets, including resizing, grayscale, normalization and other operations, to meet the input requirements of the model; S13: dividing the preprocessed dataset into a training set and a test set, allocating samples in a certain proportion to ensure that the training set is used for model learning and the test set is used for model performance evaluation.

[0075] It should be noted that, for step S2, the dual-path feature enhancement network includes: a cross-modal feature enhancement module (CMFEM) and a recognizable target centroid loss (RTCL); wherein the dual-path feature enhancement network uses ResNet50 as the backbone network, integrates the cross-modal feature enhancement module CMFEM after each Block, and the recognizable target centroid loss RTCL plays a key role in the network, which is used to optimize the feature expression of the extracted RGB and sketch images;

[0076] Specifically, the network extracts features after each layer of blocks, and further fuses local and global information through the cross-modal feature enhancement module CMFEM; these feature-enhanced RGB and sketch features are input into the recognizable target center loss module RTCL in the final stage of the network. RTCL calculates the distance between these extracted features and the target category center; by minimizing the distance between the features of different modes, namely RGB and sketch, and the category center, RTCL ensures that these features can be better aligned in cross-modal and different artistic style scenarios, and improves the recognition performance of the model;

[0077] Specifically, the cross-modal feature enhancement module CMFEM first performs a global average pooling operation to transform the input feature map Compress in space and generate channel descriptor ; Then, the module uses two fully connected layers to descriptor Perform nonlinear processing to obtain the channel weight coefficient ; Finally, the weight coefficient With the original feature map Multiply channel by channel to generate a weighted feature map ;

[0078] The recognizable target center loss RTCL first calculates the average distance of features within the same modality and the average distance between different modalities; specifically, it first calculates the channel descriptor of the feature map through global average pooling, and then calculates the average distance between the descriptor and other positive samples in the same modality. and the average distance to positive samples of different modalities ; Next, the feature center of each identity is calculated, and based on this, the average distance between the feature center and the negative sample is calculated to form a dynamic boundary ; Finally, RTCL combines this distance information to enhance the utilization of reliable features by minimizing the distance difference between modalities and optimizing feature alignment, while reducing the impact of unreliable features, thereby improving the effect of cross-modal retrieval with subjective information.

[0079] Specifically, step S3 includes: S31: setting the training parameters of the dual-path feature enhancement network and adjusting the weights, and inputting the training set and the validation set into the network; S32: using the cross-modal feature enhancement module CMFEM to respectively transform the input photo features into and sketch features Extracted into key features and ; S33: Calculate the average distance between the model and other positive samples within and across modalities In the distance optimization process, the inconsistent features of the two modes due to subjective understanding will be suppressed, while the more reliable features in the two modes will be enhanced. The constraint formula is ; Calculate the RTCL function loss ,in, , enhance the use of reliable subjective features; S34: repeat steps S32-S33 until the maximum number of times, The smallest model is taken as the trained model.

[0080] Specifically, the key features and The calculation formula is:

[0081] ,

[0082] in,

[0083] , ,

[0084] for , represents the weight matrix of the first fully connected layer, represents the weight matrix of the second fully connected layer, Represents the Sigmoid activation function; variable represents the compressed channel feature representation;

[0085] Specifically, for the average distances to other positive samples within and across modalities , such as the formula:

[0086] in,

[0087] Among them, for and , K represents the number of samples in a mini-batch, represents the feature vector of the a-th sample sketch image, Represents the feature vector of the i-th RGB image;

[0088] Specifically, the constraint formula for:

[0089] in represents the feature vector of the jth sample, represents the feature vector of the kth sample, and mean represents the mean operation on a set of data. Indicates that it belongs to the category The characteristic center vector of the sample is Indicates the category The average distance of negative samples.

[0090] Specifically, step S4 includes: S41: inputting the image to be detected in the test set into the trained dual-path feature enhancement network, and sequentially passing through the four-layer cross-modal feature enhancement module for feature extraction; S42: when processing multiple sketch images input at the same time, the features extracted by the cross-modal feature enhancement module are fused through non-local features to make the key features and Map to the same feature space dimension to ensure that the features of the RGB image and the sketch image can be compared in a unified feature space; S43: Based on feature alignment, the recognizable target center loss RTCL is used to optimize the consistency features in cross-modality; S44: Perform bidirectional alignment. In the whole process, the alignment of photo features to sketch features and the alignment of sketch features to photo features are symmetrical; By calculating the cross-modal similarity and matching probability, and then using this information for soft alignment, the final result is the alignment of the two modal features in the same space, and the reconstructed features are:

[0091]

[0092] Among them, M represents the character segmentation map;

[0093] It should be noted that the feature alignment from photo to sketch is similar. The reconstructed features from photo to sketch and sketch to photo are obtained for comprehensive analysis to obtain the person retrieval results.

[0094] It should be noted that in step S4, the key features are first calculated and Similarity:

[0095]

[0096] in, and Represents the RGB image features and sketch image features at positions b and c, represents the L2 normalization function; then calculate the matching probability:

[0097]

[0098] Finally, the softening function is calculated: .

[0099] In this embodiment, by introducing a dual-path feature enhancement network (Dual-Path Feature-EnhanceNetwork) and a cross-modal feature enhancement module (Cross-Modal Feature Enhancement Module), the cross-modal retrieval performance in the sketch pedestrian re-identification task is effectively improved; specifically, the dual-path feature enhancement network can capture key features under different modalities by extracting features from RGB images and sketch images and combining the fusion of global and local features; by introducing the recognizable target centroid loss (Recognizable Target Centroid Loss, RTCL), the effective alignment of features of different modalities is ensured, the inconsistency between features is reduced, and the robustness of cross-modal retrieval is improved; first, the standardized pre-processed data set is used to ensure the input consistency of the model and reduce the impact of data differences; secondly, the dual-path feature enhancement network is combined with the ResNet50 backbone network and the cross-modal feature enhancement module to further improve the interactivity and consistency between features of different modalities; finally, through the soft alignment technology and the bidirectional alignment mechanism, the matching of RGB image and sketch image features in the same space is ensured, thereby improving the accuracy of retrieval.

[0100] Table 1 is a comparison table of the effects of different pedestrian re-identification methods, where DDAG is the method in "Dynamic dual-attentive aggregation learning for visible-infrared person re-identification", CM-NAS is the method in "Cross-modality neural architecture search for visible-infrared person re-identification", CAJ is the method in "Channel augmented joint learning for visible-infrared recognition", MMN is the method in "Towards a unified middle modality learning for visible-infrared person re-identification", DART is the method in "Learning with twin noisy labels for visible-infrared person re-identification", and DCLNet is the method in "Not all pixels are matched: Dense contrastive learning for visible-infrared person re-identification". cross-modality person re-identification, Not all pixels match: Dense contrast learning for cross-modal person re-identification", DSCNet is the method in "Dual-semantic consistency learning for visible-infrared person re-identification", DEEN is "Diverse embedding expansion network and low-light cross-modality benchmark for visible-infrared person re-identification",Diversity embedding extended network and low-light cross-modal benchmark for visible-light-infrared pedestrian re-identification", and the baseline (single query) and baseline (multi-query) are the methods in "Beyond domain gap: Exploiting subjectivity in sketch-based person retrieval". As can be seen from Table 1, the sketch pedestrian re-identification method based on cross-modal reliable subjective features of this embodiment significantly improves the mean average precision mAP of the cross-modal retrieval task, especially between sketch images of different artistic styles and diverse RGB images. By optimizing the distance between intra-modal and cross-modal features, the sketch pedestrian re-identification method based on cross-modal reliable subjective features of this embodiment can better handle the differences between features of different modalities, while enhancing the utilization of reliable features and reducing the influence of unreliable features. In addition, by aligning and optimizing features throughout the training process, the sketch pedestrian re-identification method based on cross-modal reliable subjective features of this embodiment effectively improves the pedestrian re-identification performance in different scenarios and conditions, and shows strong generalization ability.

[0101] Table 1: Comparison of the effects of different pedestrian re-identification methods

[0102]

[0103] Considering the recently launched MARKET-SKETCH-1K dataset and the scarcity of sketch Re-ID research on this dataset, the experiment of this embodiment used a variety of visible light Re-ID methods similar to sketch Re-ID, and supplemented some sketch-specific Re-ID methods. In a single query scenario, this embodiment surpassed the best performing visible light Re-ID method by 6.14% in mAP evaluation indicators and exceeded the sketch Re-ID method by 1.26%, verifying that the sketch pedestrian re-identification method based on cross-modal reliable subjective features of this embodiment is effective in solving sketch Re-I D The effectiveness and persistence of the inherent challenges of the task; In Table 1, it can be found that there are significant differences between the multi-query and single-query methods in the sketch pedestrian re-identification method based on cross-modal reliable subjective features of this embodiment, and the differences in mAP, Rank-1, Rank-5 and Rank-10 are 6.56%, 8.61%, 9.81% and 9.51% respectively. This phenomenon vividly demonstrates the importance of integrating multiple subjective queries. The sketch pedestrian re-identification method based on cross-modal reliable subjective features of this embodiment is keenly aware of this and regards it as a key factor in improving retrieval performance.

[0104] Table 2: Comparison of all sketches in the PKU Sketch Re-ID dataset with other methods

[0105]

[0106] Table 2 is a comparison table of all sketches in the PKU Sketch Re-ID dataset with other methods, where Triplet SN is the method in "Sketch me that shoe", GN Siamese is the method in "The sketchy database: learning to retrieve badly drawn bunnies", AFL Net is the method in "Cross-domain adversarial feature learning for sketch re-identification", RCD is the method in "Eliminate deviation with deviation for data augmentation and a general multi-modal data learning method", TC-Net is the method in "TC-Net for ISBIR: Triplet Classification Network for Instance-Level Sketch Based Image Retrieval", and MDFL Net is the method in "Learning multi-level domain invariant features for sketch The method in “Learning multi-level domain-invariant features for sketch re-identification” is used here. UFE is the method in “Instance-level heterogeneous domain adaptation for limited-labeled sketch-to-photo retrieval”, CDA is the method in “Cross-domain attention and center loss for sketch re-identification”, and the baseline is the method in “Beyonddomain gap: Exploiting subjectivity in sketch-based person retrieval”.As shown in Table 2, this embodiment compares and evaluates the sketch pedestrian re-identification method based on cross-modal reliable subjective features with a group of state-of-the-art algorithms on the PKU Sketch Re-ID dataset. Triplet SN is designed to identify hand-drawn sketches, and its Rank-1 accuracy is 9.00%, which highlights the challenge of extracting invariant features across domains; GN Siamese combines dual GoogLeNet branches and optimizes Siamese and classification losses, and its Rank-1 accuracy is improved to 28.90%, demonstrating the ability of this method to learn common features between sketch and RGB domains; AFL Net is a cross-domain adversarial model that achieves a Rank-1 accuracy of 34.00%, indicating that it can jointly learn identity and domain invariant features in the Sketch-RGB person re-identification task; RCD achieves a Rank-1 accuracy of 42.50% through random transformation; TC-Net uses a triplet network to achieve a Rank-1 accuracy of 48.10%; DFL Net achieves a competitive Rank-1 accuracy of 49.00% by fusing multi-level features extracted from different modalities; UFE The unbiased feature extractor solves the feature extraction problem and achieves a Rank-1 accuracy of 57.14%. CDA proposes a cross-domain attention mechanism and achieves a Rank-1 accuracy of 60.80%. In contrast, this embodiment achieves a competitive Rank-1 accuracy of 70.00%, which is consistent with the baseline method. It is worth noting that this embodiment performs well in higher ranking indicators, with a Rank-5 accuracy of 90.00%, a Rank-10 accuracy of 96.00%, and a Rank-20 accuracy of 100.00%, surpassing all other methods listed in the table. In addition, this embodiment achieves an average accuracy mAP of 69.17% on the PKU Sketch Re-ID dataset, which proves the effectiveness, robustness and excellent performance of the sketch pedestrian re-identification method based on cross-modal reliable subjective features in the sketch-based person re-identification task. ;

[0107] Table 3: Comparison of mAP evaluation by single query training and testing

[0108]

[0109] Table 4: Comparison of mAP evaluation by multi-query training and testing

[0110]

[0111] Table 5: Comparison of mAP evaluation by single query training and multi-query testing

[0112]

[0113] Table 6: Comparison of mAP evaluation by multi-query training and single-query testing

[0114]

[0115] Tables 3, 4, 5 and 6 evaluate the unrecognizable styles during the training and testing process, and demonstrate the experiments of the sketch pedestrian re-identification method based on cross-modal reliable subjective features of this embodiment under various styles. It can be seen that the sketch pedestrian re-identification method based on cross-modal reliable subjective features of this embodiment is always better than the baseline in terms of mAP indicators on all data, further verifying the superiority of the sketch pedestrian re-identification method based on cross-modal reliable subjective features of this embodiment. Under all query and test settings in Tables 3, 4, 5 and 6, it can be observed that the experimental results gradually improve with the change of the number of styles, which highlights the powerful generalization ability of the sketch pedestrian re-identification method based on cross-modal reliable subjective features of this embodiment. In addition, regardless of the configuration of the training and test parameters, the model of this embodiment exhibits better performance when trained and tested under a multi-query setting. This experimental result further verifies the importance of using multiple subjective queries.

[0116] This embodiment provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the above-mentioned sketch pedestrian re-identification method based on cross-modal reliable subjective features are implemented. Among them, the storage medium can be a disk, an optical disk, a read-only memory (ROM), a random access memory (RAM), a flash memory, a hard disk drive (HDD) or a solid-state drive (SSD), etc.; the storage medium can also include a combination of the above-mentioned types of memory.

[0117] This embodiment provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the steps of the above-mentioned method for re-identifying a person in sketch based on cross-modal reliable subjective features are implemented.

[0118] like Figure 3As shown, the computer device 120 may include: at least one processor 121, such as a CPU (Central Processing Unit), at least one communication interface 123, a memory 124, and at least one communication bus 122. Among them, the communication bus 122 is used to realize the connection and communication between these components. Among them, the communication interface 123 may include a display screen (Display), a keyboard (Keyboard), and the optional communication interface 123 may also include a standard wired interface and a wireless interface. The memory 124 may be a high-speed RAM memory (Random Access Memory, volatile random access memory), or a non-volatile memory (non-volatile memory), such as at least one disk memory. The memory 124 may also be at least one storage device located away from the aforementioned processor 121. Among them, the memory 124 stores an application program, and the processor 121 calls the program code stored in the memory 124 to perform any of the above method steps. Among them, the communication bus 122 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The communication bus 122 can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 3Only one line is used to represent it, but it does not mean that there is only one bus or one type of bus. Among them, the memory 124 may include a volatile memory (English: volatile memory), such as a random access memory (English: random-access memory, abbreviated: RAM); the memory may also include a non-volatile memory (English: non-volatile memory), such as a flash memory (English: flash memory), a hard disk drive (English: hard disk drive, abbreviated: HDD) or a solid-state drive (English: solid-state drive, abbreviated: SSD); the memory 124 may also include a combination of the above-mentioned types of memory. Among them, the processor 121 may be a central processing unit (English: central processing unit, abbreviated: CPU), a network processor (English: network processor, abbreviated: NP) or a combination of CPU and NP. Among them, the processor 121 may further include a hardware chip. The above-mentioned hardware chip may be an application-specific integrated circuit (English: application-specific integrated circuit, abbreviated: ASIC), a programmable logic device (English: programmable logic device, abbreviated: PLD) or a combination thereof. The PLD may be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof. Optionally, the memory 124 is also used to store program instructions. The processor 121 may call the program instructions to implement the sketch pedestrian re-identification method based on cross-modal reliable subjective features as in the present embodiment.

[0119] This embodiment provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of the above-mentioned sketch pedestrian re-identification method based on cross-modal reliable subjective features.

[0120] The embodiments of the present invention are described above in conjunction with the accompanying drawings, but the present invention is not limited to the above-mentioned specific implementation modes, which are merely illustrative rather than restrictive. Under the guidance of the present invention, ordinary technicians in this field can also make many forms without departing from the scope of protection of the present invention and the claims, all of which are within the protection of the present invention.

Claims

1. A sketch person re-identification method based on cross-modal reliable subjective features, characterized in that: The following steps are involved: S01: Acquire RGB and sketch images of a person, preprocess the RGB and sketch images of the person to obtain a data set of the RGB and sketch images of the person, and divide the data set of the RGB and sketch images of the person into a training set, a verification set, and a test set; S02: Use residual network to build dual-path feature enhancement network; S03: training the dual-path feature enhancement network using the training set to obtain a trained dual-path feature enhancement network; S04: according to the test set, using the trained dual-path feature enhancement network, obtaining a character sketch retrieval result; The preprocessing includes: resizing, graying and normalization; The dual-path feature enhancement network uses ResNet50 as the backbone network, and a cross-modal feature enhancement module is connected in series after each residual module Block of ResNet50; the cross-modal feature enhancement module is used to extract features and further integrate local and global information; the cross-modal feature enhancement module includes an average pooling layer, a first fully connected layer, a linear rectification function ReLU, a second fully connected layer, a Sigmoid function, and an element-by-element multiplication operation connected in series in sequence; wherein the input end of the average pooling layer is connected in parallel with the input end of the element-by-element multiplication operation; Step S03 specifically includes: S031: setting training parameters of the dual-path feature enhancement network and adjusting weights, and inputting the training set and the verification set into the dual-path feature enhancement network; S032: using a cross-modal feature enhancement module to extract input photo features and input key features into RGB key features and sketch key features respectively; S033: constructing a loss function according to the RGB key features and the sketch key features; S034: repeat steps S032-S033 to a preset number of times, calculate the loss function, and use the model with the smallest loss function as the trained dual-path feature enhancement network; Step S032 specifically includes: using a cross-modal feature enhancement module to extract input photo features and input key features into RGB key features and sketch key features, respectively, as shown in the formula: ， ， in, represents the feature vector of the cth class or channel, is the eigenvector of the cth class obtained after transformation, represents the Sigmoid activation function, represents the weight matrix of the first fully connected layer, represents the weight matrix of the second fully connected layer, represents the compressed channel feature representation, represents the set of transformed feature vectors representing all categories; Step S033 specifically includes: constructing a loss function according to the RGB key features and the sketch key features, such as the formula: ， ， ， in, is the loss function, is the average distance to other positive samples within and across modalities, is the constraint formula, is the total number of samples, represents the number of samples in a mini-batch, is the distance to other positive samples within the modality, is the distance to other positive samples under cross-modality, Indicates The feature vector of the sample sketch image, Indicates The feature vector of the RGB image, Indicates The feature vector of the samples, Indicates The feature vector of the samples, It means to perform mean operation on a set of data. Indicates that it belongs to the category The characteristic center vector of the sample is Indicates category The average distance of negative samples.

2. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method for re-identifying a person from a sketch based on cross-modal reliable subjective features as claimed in claim 1 are implemented.

3. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the sketch pedestrian re-identification method based on cross-modal reliable subjective features as described in claim 1 are implemented.

4. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method for re-identifying a person from a sketch based on reliable cross-modal subjective features as described in claim 1 are implemented.

Images