Sketch pedestrian re-identification method and system based on optimal transmission theory and text prompt modeling

Abstract

The invention discloses a sketch pedestrian re-identification method and system based on an optimal transmission theory and text prompt modeling. The method comprises the following steps: acquiring a real image and a sketch; feature extraction is carried out on the real image and the sketch, image features are obtained, and the image features comprise the global feature and the local feature of the real image and the global feature and the local feature of the sketch; acquiring global features and local features of the text based on the real image; global alignment is carried out on the global features of the real image, the global features of the sketch and the global features of the text, and the global features of the real image and the global features of the sketch after alignment are obtained; based on the local features of the text, carrying out local alignment on the local features of the real image and the local features of the sketch, and obtaining the aligned local features of the real image and the sketch; and re-identifying sketch pedestrians based on the aligned real image global features and sketch global features and the aligned real image local features and sketch local features.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer vision and information retrieval, and in particular relates to a sketch pedestrian re-identification method and system based on optimal transmission theory and text prompt modeling. Background Art

[0002] Sketch person re-identification is an important research direction in the field of computer vision and information retrieval. It aims to accurately locate target pedestrians from massive surveillance images through sketches drawn by artists.

[0003] Currently, an effective solution for person re-identification from sketches is to achieve hard alignment by leveraging the interaction of different modalities in feature space. These methods rely on loss constraints to force the features of different modalities to be mapped into the same vector space, thereby achieving alignment. However, this hard alignment approach has certain limitations. It focuses solely on the coarse-grained conceptual representations of different modalities, but fails to fully consider the interaction of fine-grained semantic information within each modality. To compensate for the inevitable loss of detail during direct bimodal interaction, another class of methods attempts to introduce an intermediate modality as a bridge to supplement information between modalities. For example, this is achieved by generating virtual sketches or manually annotating text information. However, generating virtual sketches can introduce unnecessary noise, while manually annotating text requires significant labor costs.

[0004] In order to better achieve cross-modal alignment between sketches and RGB images, most existing methods use an intermediate modality as a bridge. Previous technologies used RGB images to generate virtual sketches, or used manually annotated text descriptions. Among them, the method of generating virtual sketches is prone to introducing unnecessary noise; and although text descriptions can serve as higher-level semantic knowledge to bridge the modality gap, the manual annotation method not only consumes a lot of manpower costs but also has no scalability. In addition, when achieving more accurate retrieval results, it is extremely important to pay attention to and extract fine-grained features. However, when processing sketches and RGB images, existing methods often extract local features independently, which makes it impossible to effectively align these features in cross-modal scenarios, making it difficult to fully realize the potential of fine-grained features to improve retrieval accuracy. Summary of the Invention

[0005] To solve the above technical problems, the present invention proposes a sketch pedestrian re-identification method and system based on optimal transfer theory and text prompt modeling, which generates text attribute information based on the image by introducing a pre-trained visual question answering model (VQA). In this way, without any manual annotation, text knowledge as an intermediate modality can be obtained to guide cross-modal alignment. In order to better assist the network in extracting fine-grained information, the present invention introduces an optimal distance transfer algorithm to map local text embeddings into a more discriminative feature representation, namely text consensus information. Under the guidance of text consensus, the model selectively focuses on key details, promotes in-depth modal interaction between sketches and RGB images, and mines fine-grained modal representations.

[0006] The present invention provides a sketch person re-identification method based on optimal transmission theory and textual hint modeling, comprising:

[0007] Get real images and sketches;

[0008] Performing feature extraction on the real image and the sketch to obtain image features, wherein the image features include: global features and local features of the real image, and global features and local features of the sketch;

[0009] Based on the real image, obtaining global features and local features of the text;

[0010] Performing global alignment on the global features of the real image, the global features of the sketch, and the global features of the text to obtain aligned global features of the real image and global features of the sketch;

[0011] Based on the local features of the text, the local features of the real image and the local features of the sketch are locally aligned to obtain the aligned local features of the real image and the local features of the sketch;

[0012] Based on the aligned real image global features and sketch global features, and the aligned real image local features and sketch local features, the sketch pedestrian is re-identified.

[0013] Optionally, performing feature extraction on the real image and the sketch to obtain image features includes:

[0014] A shared visual encoder is used to perform feature extraction on the real image and the sketch respectively to obtain the image features.

[0015] Optionally, obtaining global features and local features of the text based on the real image includes:

[0016] Based on the real image, obtaining text attributes;

[0017] The text attributes and dynamic prompt information are input into a text encoder to obtain global features and local features of the text.

[0018] Optionally, performing global alignment on the global features of the real image, the global features of the sketch, and the global features of the text to obtain the aligned global features of the real image and the global features of the sketch includes:

[0019] Performing cross-modal interaction on the global features of the real image, the global features of the sketch, and the global features of the text to obtain fused features;

[0020] The fused features are refined using standard Transformer blocks to obtain aligned real image global features and sketch global features.

[0021] Optionally, based on the local features of the text, local features of the real image and local features of the sketch are locally aligned. The aligned local features of the real image and local features of the sketch are obtained by:

[0022] Performing local alignment enhancement on the local features of the text, the local features of the real image, and the local features of the sketch to obtain enhanced local features;

[0023] Initializing the text atoms in the enhanced local features and calculating the cost matrix;

[0024] Expanding the cost matrix to obtain an expanded matrix;

[0025] Removing features irrelevant to the text information from the extended matrix to obtain an optimal transmission matrix;

[0026] The enhanced local features and the optimal transfer matrix are aggregated to obtain aligned real image local features and sketch local features.

[0027] Optionally, a triplet assignment loss function is used to constrain global and local alignment;

[0028] The triplet assignment loss function is:

[0029]

[0030] Among them, (R i ,S i ) is a positive sample input pair, is a negative sample pair, It is R i The most similar negative sample of is a negative sample pair, It's S i The most similar negative sample, m is the minimum interval that controls the distance between positive and negative samples.

[0031] The present invention also provides a sketch person re-identification system based on optimal transmission theory and text prompt modeling, comprising: a feature extraction module, a text prompt reasoning module, a text-injected coarse-grained alignment module, and a consensus-guided fine-grained interaction module;

[0032] The feature extraction module is used to extract features from real images and sketches to obtain image features;

[0033] The text prompt reasoning module is used to introduce text as an intermediate modality to guide the coordination between modalities and obtain text consensus features;

[0034] The coarse-grained alignment module of the text injection is used to globally align the global features of the real image, the global features of the sketch, and the global features of the text, and obtain the aligned global features of the real image and the global features of the sketch;

[0035] The consensus-guided fine-grained interaction module is used to locally align the local features of the real image and the local features of the sketch based on the local features of the text, obtain the aligned local features of the real image and the local features of the sketch, and re-identify the sketch pedestrian based on the aligned global features of the real image and the global features of the sketch, and the aligned local features of the real image and the local features of the sketch.

[0036] Optionally, the coarse-grained alignment module for text injection includes: a cross-modal interaction unit and a feature enhancement unit;

[0037] The cross-modal interaction unit is used to interact with the image features and the local features of the text;

[0038] The feature enhancement unit is used to enhance the features after interaction to obtain a global concept representation of the sketch.

[0039] Compared with the prior art, the present invention has the following advantages and technical effects:

[0040] (1) This paper uses a pre-trained VQA model to generate multi-dimensional text attribute information based on RGB images. Without any manual annotation intervention, the introduction of text as an intermediary facilitates cross-modal alignment between RGB images and sketches, greatly improving the autonomy and efficiency of the alignment process.

[0041] (2) This paper applies the optimal distance transfer theory to filter out non-informative features in textual knowledge and map local text embeddings into more discriminative feature representations, namely text consensus. This consensus information can guide RGB images and sketches to focus on key details, not only promoting deeper modal interaction but also successfully mining fine-grained feature representations, significantly enhancing the accuracy and depth of feature extraction.

[0042] (3) The triplet allocation loss proposed in the present invention can comprehensively analyze the overall sample status. In this way, a more accurate distance measurement is achieved, thereby achieving a better optimization effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] The accompanying drawings, which constitute part of this application, are intended to provide a further understanding of this application. The exemplary embodiments and descriptions of this application are intended to explain this application and do not constitute an improper limitation on this application. In the accompanying drawings:

[0044] Figure 1 This is a flow chart of a sketch person re-identification method based on optimal transmission theory and text prompt modeling according to an embodiment of the present invention. DETAILED DESCRIPTION

[0045] It should be noted that, in the absence of conflict, the embodiments and features of the embodiments in this application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0046] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0047] This embodiment proposes a sketch pedestrian re-identification method based on optimal transmission theory and text prompt modeling. Figure 1 As shown, the specific steps include:

[0048] Get real images and sketches;

[0049] Perform feature extraction on the real image and the sketch to obtain image features, wherein the image features include: global features and local features of the real image, and global features and local features of the sketch;

[0050] Based on real images, obtain global and local features of text;

[0051] Perform global alignment on the global features of the real image, the global features of the sketch, and the global features of the text to obtain the aligned global features of the real image and the global features of the sketch;

[0052] Based on the local features of the text, the local features of the real image and the local features of the sketch are locally aligned to obtain the aligned local features of the real image and the local features of the sketch;

[0053] Sketch pedestrians are re-identified based on the aligned real image global features and sketch global features, and the aligned real image local features and sketch local features.

[0054] Specifically, the textual hint inference module first uses a pre-trained visual question-answering model to generate multi-dimensional person attribute text based on real images, thereby incorporating textual information without additional manual annotation. To dynamically transfer the pre-trained language-visual knowledge to downstream tasks, the textual hint inference module inserts these attributes as fixed components into the learnable hint information to obtain a textual embedding representation. The text-infused coarse-grained alignment module injects the generated global textual information into the cross-modal coarse-grained alignment, guiding the alignment of global features between sketches and images. Furthermore, to filter out non-informative features from the textual knowledge, the textual hint inference module maps local text embeddings into a more discriminative feature representation, namely, textual consensus. This mapping can be formulated as an optimal transport problem. Subsequently, guided by textual consensus, it helps images and sketches selectively focus on key details, promotes deep modal interaction, and mines fine-grained feature representations. Because Euclidean distance only considers the current sample in its distance metric and ignores the overall inter-data set, this can lead to inaccurate sample distance estimates and potentially suboptimal local minima. To this end, this paper proposes a new triplet assignment loss to establish a more reasonable metric to evaluate the similarity of local features.

[0055] Furthermore, feature extraction is performed on the real image and the sketch, and the image features obtained include:

[0056] The shared visual encoder is used to extract features from real images and sketches respectively to obtain image features.

[0057] Specifically, a shared visual encoder is used for feature extraction between the sketch and the real image. In this invention, a visual encoder based on CLIP is used. The input RGB image is embedded by CLIP. where p is the number of non-overlapping blocks, R cls and denote the d-dimensional global and local features respectively. Similarly, the sketch embedding can be expressed as S= Among them S cls and denote the d-dimensional global and local features respectively. It is worth noting that the image encoders for images and sketches adopt shared weights to ensure that the features are mapped to a unified semantic space.

[0058] Furthermore, based on real images, the global and local features of the text are obtained, including:

[0059] Get text attributes based on real images;

[0060] The text attributes and dynamic prompt information are input into the text encoder to obtain the global and local features of the text.

[0061] Specifically, the significant modality gap and inherent abstract nature of sketches and RGB images can lead to semantic misalignment during knowledge learning, severely impacting the model's reasoning and generalization capabilities. To address this issue, the Text Hint Reasoning module introduces text as an intermediate modality to guide inter-modal coordination without any additional cost.

[0062] Furthermore, based on the real image, obtaining text attributes includes:

[0063] Feed real images into a pre-trained visual question answering model to obtain text attributes.

[0064] Specifically, text attribute generation: Unlike traditional re-identification tasks, sketches are easily affected by the subjective emotions and painting skills of the artist, resulting in a lack of detailed information that is crucial for model learning. The objectivity and flexibility of text prompt the model to pay more attention to semantic context information in the process of knowledge acquisition. However, directly generating a comprehensive text description of a pedestrian image will inevitably introduce irrelevant noise, thereby reducing model performance. Therefore, this embodiment retains the advanced modeling capabilities of large-scale language vision models for images as much as possible. Specifically, for a given RGB image, the text reasoning module uses a pre-trained visual question-answering model to answer questions about k specific details, including but not limited to the target's gender, hair type, top and bottom categories and colors, etc., and obtains the corresponding description of the target: att = {att1, att2,…, att k Importantly, this process only introduces textual details for guidance during model training, and remains consistent with other methods during inference.

[0065] Learnable Hint Strategy: This paper combines learnable hint information with original text attributes. Compared with manually set hint information, this method does not require the introduction of additional expert knowledge. Specifically, the text reasoning module initially converts these text attributes into tokens through CLIP tokenizer, i.e. a = Tokenizer(att). Then, a learnable hint information p1, p2, ..., p l Embedded into these fixed attribute tags to form a text description: q = {p1, a1, p2, a2, ..., p l ,a kThis integration is a dynamic knowledge learning mechanism that enhances flexible cross-modal interactions and improves the transferability of text embeddings. Subsequently, the entire token q is fed into the frozen text encoder to generate text embeddings Where T sos and T eos Represents [sos] and [eos] tokens, and n is the number of d-dimensional word tokens. Based on the general token selection method, T eos is used as a global representation, and Represents a series of basic text local tokens.

[0066] Furthermore, the global features of the real image, the global features of the sketch, and the global features of the text are globally aligned, and the aligned global features of the real image and the global features of the sketch are obtained, including:

[0067] Perform cross-modal interaction on the global features of real images, sketches, and texts to obtain fused features;

[0068] The fused features are refined using standard Transformer blocks to obtain aligned real image global features and sketch global features.

[0069] Furthermore, based on the local features of the text, local features of the real image and local features of the sketch are locally aligned, and the aligned local features of the real image and local features of the sketch are obtained, including:

[0070] Perform local alignment enhancement on the local features of the text, the local features of the real image, and the local features of the sketch to obtain enhanced local features;

[0071] Initialize the text atoms in the enhanced local features and calculate the cost matrix;

[0072] Expand the cost matrix to obtain an expanded matrix;

[0073] Remove features irrelevant to text information in the expanded matrix to obtain the optimal transmission matrix;

[0074] Aggregate the enhanced local features and optimal transfer matrix to obtain the aligned real image local features and sketch local features.

[0075] Specifically, in order to enhance the ability of text features to represent detailed information, this paper explores methods to filter out non-informative features, aiming to more effectively address fine-grained semantic variations across modalities (e.g., hats, shoes). Therefore, metric learning is used to design a dynamic consensus acquisition strategy to capture discriminative text consensus representation X.

[0076] First, in order to reduce the inductive bias that may be introduced by certain priors, the local text representation T local Input into the consensus multi-layer perceptron product module to obtain enhanced features Then, initialize a set of learnable text atoms And calculate the cost matrix Where the (i,j)th element C i,j Represents the cost of assigning features to atoms. Some irrelevant features related to text information, such as features representing background, may destroy the model's ability to learn target details. Inspired by the solutions in graph matching and key point matching, this embodiment sets a "trash can" in O to capture non-information features. Specifically, this embodiment expands the cost matrix from C to in, w is a learnable parameter, and 1 n Represents an n-dimensional all-1 vector. According to the optimal transmission theory, this embodiment considers that the enhanced text representation The clustering process to the centroid O is an optimal transport problem:

[0077]

[0078] The optimal allocation method P = Diag (u) KDiag (v) can be solved by alternating the following two equations through the Sinkhorn algorithm: (z) =α / (Kv (z-1) )and in ∈ is the regularization coefficient, z is the number of iterations. In order to obtain better feature quality, this embodiment intends to discard the extra "trash cans" to obtain Finally, by aggregating local text features and the optimal transmission matrix To obtain the enhanced comprehensive representation, i.e. consensus

[0079] More specifically, the consensus-guided fine-grained interaction module: Due to the inherent complexity of sketches and RGB images and the semantic misalignment that may occur during the learning process, it is crucial to capture the detailed changes in the modality. Fortunately, the fine-grained text consensus X provided by the text hint reasoning module contains rich detailed information, which provides a key solution to this problem. The consensus-guided fine-grained interaction module adopts a Transformer structure based on multi-head cross attention, which transforms the original local features R local and S local Converting to a more discriminative representation enhances the robustness of the model:

[0080]

[0081] in and Represents the mapping layer parameters of the h-th head of the two modalities. Therefore, the textual context information about pedestrian features can be integrated into the final fine-grained features through the cross-attention operation, which can help the model solve the problems caused by the diversity and uncertainty of sketches and RGB images.

[0082] Furthermore, the triplet assignment loss function is used to constrain global and local alignment;

[0083] Specifically, triplet assignment loss: triplet loss is a commonly used matching loss in cross-modal learning. In scenarios such as image-text matching and video-text retrieval, it improves the performance of the model by adjusting the distance of the most difficult samples. However, when multiple samples from different modalities show slight differences, this strategy trains each semantic part independently with equal contribution, ignoring the influence of the overall data distribution. For example, when a sketch shows high similarity with multiple RGB samples, the present invention believes that direct similarity measurement is biased and the different contributions of samples within and between modalities should be taken into account. This oversight may lead to inaccurate sample distance estimates and may lead to suboptimal local minima. To this end, the present invention proposes a new triplet assignment loss to establish a more reasonable metric to evaluate the similarity of local features.

[0084] For a positive sample input pair (R i ,S i ), and the feature representations obtained through model inference are expressed as f(R i ) and f(S i ). If all sample feature sets of different modalities in x are considered as two discrete distributions, their alignment can be regarded as an optimal transmission problem. Cost matrix From pairwise feature similarity: The goal of this invention is to obtain the optimal transmission matrix P at the minimum cost. * ,in represents the (R i ,S j ). The triplet assignment loss can be expressed as the weighted sum of the original distance and the optimal assignment distance according to the traditional triplet loss, and dynamically updated at a certain ratio γ:

[0085]

[0086] in and is an image pair in x (R i ,S i), [x] + =max(x,0). At the same time, E(R i , S i )=||f(R i )-f(S i )||2 represents the Euclidean distance between feature representations.

[0087] This embodiment also provides a sketch person re-identification system based on optimal transmission theory and text prompt modeling, including: a feature extraction module, a text prompt reasoning module, a text injection coarse-grained alignment module, and a consensus-guided fine-grained interaction module;

[0088] Feature extraction module, used to extract features from real images and sketches to obtain image features;

[0089] The text prompt reasoning module is used to introduce text as an intermediate modality to guide the coordination between modalities and obtain text consensus features;

[0090] The coarse-grained alignment module for text injection is used to globally align the global features of the real image, the global features of the sketch, and the global features of the text to obtain the aligned global features of the real image and the global features of the sketch;

[0091] A consensus-guided fine-grained interaction module is used to locally align the local features of the real image and the local features of the sketch based on the local features of the text, obtain the aligned local features of the real image and the local features of the sketch, and re-identify the sketch pedestrian based on the aligned global features of the real image and the global features of the sketch, and the aligned local features of the real image and the local features of the sketch.

[0092] Furthermore, the coarse-grained alignment module for text injection includes: a cross-modal interaction unit and a feature enhancement unit;

[0093] Cross-modal interaction unit, used to interact with image features and local features of text;

[0094] The feature enhancement unit is used to enhance the features after interaction and obtain the global concept representation of the sketch.

[0095] The above are merely preferred embodiments of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

Claims

1. A sketch person re-identification method based on optimal transmission theory and textual hint modeling, characterized by: include: Get real images and sketches; Performing feature extraction on the real image and the sketch to obtain image features, wherein the image features include: global features and local features of the real image, and global features and local features of the sketch; Based on the real image, obtaining global features and local features of the text; Performing global alignment on the global features of the real image, the global features of the sketch, and the global features of the text to obtain aligned global features of the real image and global features of the sketch; Based on the local features of the text, the local features of the real image and the local features of the sketch are locally aligned to obtain the aligned local features of the real image and the local features of the sketch; Based on the aligned real image global features and sketch global features, and the aligned real image local features and sketch local features, the sketch pedestrian is re-identified.

2. The sketch person re-identification method based on optimal transmission theory and textual hint modeling according to claim 1 is characterized in that: Performing feature extraction on the real image and the sketch to obtain image features includes: A shared visual encoder is used to perform feature extraction on the real image and the sketch respectively to obtain the image features.

3. The sketch person re-identification method based on optimal transmission theory and textual hint modeling according to claim 1 is characterized in that: Based on the real image, obtaining global features and local features of the text includes: Based on the real image, obtaining text attributes; The text attributes and dynamic prompt information are input into a text encoder to obtain global features and local features of the text.

4. The sketch person re-identification method based on optimal transmission theory and textual hint modeling according to claim 1 is characterized in that: Performing global alignment on the global features of the real image, the global features of the sketch, and the global features of the text to obtain the aligned global features of the real image and the global features of the sketch includes: Performing cross-modal interaction on the global features of the real image, the global features of the sketch, and the global features of the text to obtain fused features; The fused features are refined using standard Transformer blocks to obtain aligned real image global features and sketch global features.

5. The sketch person re-identification method based on optimal transmission theory and textual prompt modeling according to claim 1 is characterized in that: Based on the local features of the text, the local features of the real image and the local features of the sketch are locally aligned. The aligned local features of the real image and the local features of the sketch include: Performing local alignment enhancement on the local features of the text, the local features of the real image, and the local features of the sketch to obtain enhanced local features; Initializing the text atoms in the enhanced local features and calculating the cost matrix; Expanding the cost matrix to obtain an expanded matrix; Removing features irrelevant to the text information from the extended matrix to obtain an optimal transmission matrix; Aggregating the enhanced local features and the optimal transfer matrix to obtain aligned real image local features and sketch local features.

6. The sketch person re-identification method based on optimal transmission theory and textual hint modeling according to claim 4 or 5, characterized in that: The triplet assignment loss function is used to constrain global and local alignment; The triplet assignment loss function is: Among them, (R i ,S i ) is a positive sample input pair, is a negative sample pair, It is R i The most similar negative sample of is a negative sample pair, It's S i The most similar negative sample, m is the minimum interval that controls the distance between positive and negative samples.

7. A sketch person re-identification system based on optimal transmission theory and textual prompt modeling, characterized by: include: Feature extraction module, text prompt reasoning module, coarse-grained alignment module for text injection, and consensus-guided fine-grained interaction module; The feature extraction module is used to extract features from real images and sketches to obtain image features; The text prompt reasoning module is used to introduce text as an intermediate modality to guide the coordination between modalities and obtain text consensus features; The coarse-grained alignment module of the text injection is used to globally align the global features of the real image, the global features of the sketch, and the global features of the text, and obtain the aligned global features of the real image and the global features of the sketch; The consensus-guided fine-grained interaction module is used to locally align the local features of the real image and the local features of the sketch based on the local features of the text, obtain the aligned local features of the real image and the local features of the sketch, and re-identify the sketch pedestrian based on the aligned global features of the real image and the global features of the sketch, and the aligned local features of the real image and the local features of the sketch.

8. The sketch person re-identification system based on optimal transmission theory and text prompt modeling according to claim 7 is characterized in that: The coarse-grained alignment module for text injection includes: a cross-modal interaction unit and a feature enhancement unit; The cross-modal interaction unit is used to interact with the image features and the local features of the text; The feature enhancement unit is used to enhance the features after interaction to obtain a global concept representation of the sketch.

Images