Text-image pedestrian retrieval method based on neighbor matching hybrid experts

Abstract

The application discloses a text-image pedestrian retrieval method based on a near neighbor matching mixed expert, relates to the technical field of multi-modal retrieval, and comprises the following steps: constructing and training a text-image pedestrian retrieval model, extracting global image features and global text features by using a visual branch and a text branch respectively, and establishing a pedestrian image feature database; obtaining a text to be queried and performing cross-modal preliminary matching on the text by using a near neighbor matching mixed expert reordering module to screen candidate pedestrian images; mining the near neighbor relationship between text label features and image block features for the candidate image-text pairs, obtaining reliable cross-modal local matching representations, and selecting corresponding expert networks by using an expert routing mechanism to adaptively enhance and correct the local matching relationship between the local features of the text end and the image end, so that reordered retrieval results are obtained. The application solves the problems of how to improve the reliability of text and image local semantic matching and how to improve the accuracy of text-image pedestrian retrieval.

Description

Technical Field

[0001] This invention relates to the field of multimodal retrieval, specifically to a text-image pedestrian retrieval method based on nearest neighbor matching and hybrid experts. Background Technology

[0002] Text-image pedestrian retrieval aims to retrieve target pedestrian images that match natural language descriptions from a pedestrian image database. However, since local descriptions in natural language, such as clothing color, shoe style, and carried items, typically only correspond to a portion of a pedestrian image, there is no fixed one-to-one correspondence between text tags and image blocks. Overlay is affected by background, occlusion, and similar appearance regions. When relying solely on local similarity for dense matching or nearest neighbor selection, local correspondences unrelated to actual semantics are easily generated, which is detrimental to the performance of text-image pedestrian retrieval. Furthermore, the semantic information and matching reliability contained in different local cross-modal correspondences vary. Applying a uniform feature transformation and similarity calculation method still makes it difficult to differentiate between different local matching relationships, further limiting the performance of text-image pedestrian retrieval. Summary of the Invention

[0003] The purpose of this application is to propose a text-image pedestrian retrieval method based on nearest neighbor matching and hybrid experts to address the aforementioned technical problems. This method obtains reliable cross-modal local matching representations from candidate text-image pairs and performs adaptive feature enhancement and matching correction on the text-side feature correction and image-side feature correction and their local matching relationship based on the output of the expert network, thereby improving the performance of text-image pedestrian retrieval.

[0004] This invention provides a text-image pedestrian retrieval method based on nearest neighbor matching hybrid experts, comprising the following steps:

[0005] A text-image pedestrian retrieval model is constructed and trained to obtain the trained text-image pedestrian retrieval model. The text-image pedestrian retrieval model includes a visual branch, a text branch, and a nearest neighbor matching hybrid expert re-ranking module. The visual branch includes a pre-trained visual encoder, which is used to extract global visual features and image patch feature sequences corresponding to pedestrian images. The text branch includes a pre-trained text encoder, which is used to extract global text features and text tag feature sequences corresponding to natural language text descriptions. Several pedestrian images are encoded using the visual branch of the trained text-image pedestrian retrieval model to extract global visual features and image patch feature sequences, and a pedestrian image feature database is constructed.

[0006] The system obtains the natural language text description of the person to be queried and inputs it into a trained text-image pedestrian retrieval model. After text branching, it extracts the global text features and text tag feature sequences of the person to be queried. The global text features and text tag feature sequences of the person to be queried, along with the global visual features and image patch feature sequences from the pedestrian image feature database, are input into a nearest neighbor matching hybrid expert re-ranking module. First, cross-modal similarity is calculated between the global text features and the global visual features from the pedestrian image feature database to obtain an initial cross-modal similarity. Based on the initial cross-modal similarity, the top K candidate pedestrian images are selected. Then, based on the text tag feature sequences, the corresponding image patch feature sequences of each candidate pedestrian image are filtered to obtain nearest neighbor text tag-image patch matching pairs. Finally, based on the text tag-image patch matching pairs, a new algorithm is generated. A cross-modal local matching representation is generated. Each cross-modal local matching representation is input into an expert routing unit to select the corresponding expert network, and the routing weights corresponding to each cross-modal local matching representation and the selected expert networks are calculated. Each cross-modal local matching representation is input into the corresponding selected expert network to generate text-side feature correction, image-side feature correction, and matching correction values. The corresponding text tag features and image patch features are updated based on the text-side feature correction and image-side feature correction. The matching correction values ​​output by multiple expert networks are weighted according to the routing weights to obtain candidate matching correction terms. The re-ranking score is calculated based on the updated text features, updated image features, candidate matching correction terms, and initial cross-modal similarity. The text-image pedestrian retrieval results for the person to be queried are determined based on the re-ranking score.

[0007] Preferably, text tag-image patch matching pairs that are neighbors are selected, specifically including:

[0008] The text marker feature sequence and the corresponding image patch feature sequence of the candidate pedestrian image are represented as follows:

[0009] ;

[0010] ;

[0011] in, Represents a sequence of text tag features; Represents the feature sequence of an image patch; Indicates the first Each text tag feature; Indicates the first Image patch features; This represents the total number of text tag features; This represents the total number of features in an image patch; Let T denote the feature dimension, and T denote the transpose of the matrix. Represents the set of real numbers;

[0012] The text marker feature sequence and the image patch feature sequence are normalized using L2 norm, and the local similarity matrix is ​​calculated as follows:

[0013] ;

[0014] in, This represents the L2 norm normalization operation performed along the feature dimension; Represents the local similarity matrix;

[0015] For any text tag feature and any image patch feature, determine whether they constitute a nearest neighbor text tag-image patch matching pair according to the following formula:

[0016] ;

[0017] in, Indicates the first The text tag feature and the first Nearest neighbor matching indicator values ​​between features of image patches; when At that time, the first The text tag feature and the first If the features of each image patch constitute a text tag-image patch matching pair that are neighbors, then no text tag-image patch matching pair will be formed. This indicates an indicator function; its value is 1 if the condition is met, and 0 otherwise. Represents the first element in the local similarity matrix. The vector formed by the local similarity between each text tag feature and all image patch features. Represents the first element in the local similarity matrix. The vector formed by the local similarity between each image patch feature and all text tag features; Indicates the first The vector formed by the local similarities between each text tag feature and all image patch features is the one with the highest local similarity. Image patch features; Indicates the first The vector formed by the local similarities between each image patch feature and all text tag features is the one with the highest local similarity. Each text tag feature; , min means taking the minimum value; This indicates the preset number of nearest neighbors.

[0018] Preferably, cross-modal local matching representations are generated based on text tag-image patch matching pairs, specifically including:

[0019] Regarding the first For each text tag-image patch matching pair, the corresponding text tag features, image patch features, their element-wise product, their element-wise absolute difference, and their corresponding local similarity are concatenated, and then feature mapping is performed through a matching representation projection network, as shown in the following equation:

[0020] ;

[0021] in, Indicates the index of the text tag-image patch matching pair; and They represent the first Each text tag-image patch matching pair contains the indexes of the corresponding text tag features and the indexes of the image patch features; Indicates the first Cross-modal local matching representations corresponding to each text tag-image patch matching pair; Indicates feature concatenation operation; This represents element-wise multiplication; This indicates taking the absolute value of each element; This indicates a matching representation of the projection network.

[0022] Preferably, the cross-modal local matching representations are input into the expert routing unit to select the corresponding expert network, and the routing weights selected by each expert network corresponding to each cross-modal local matching representation are calculated, specifically including:

[0023] Each cross-modal local matching representation is input into a linear router, and the corresponding expert fit score is calculated through a Sigmoid function layer. The effective capacity of a single expert network is determined based on the total number of cross-modal local matching representations, the total number of expert networks, and the expert capacity factor. Finally, the network with the highest corresponding expert fit score is selected from all cross-modal local matching representations. Each cross-modal local matching characterization is used to calculate the corresponding routing weights, as shown in the following formula:

[0024] ;

[0025] in, Indicates the first The cross-modal local matching representation for the _th Expert fit score of an expert network; This represents the learnable weight matrix of a linear router; Represents the first in the vector One element; Represents the Sigmoid function; This represents the effective capacity of a single expert network; Indicates the expert capacity factor; This represents the total number of cross-modal local matching representations of the current input; This represents the total number of expert networks; Indicates rounding down; Indicates the first Is the cross-modal local matching representation of the i-th modality represented by the i-th modality local matching representation? The indicator value selected by the expert network is 1 if it is selected, and 0 otherwise. Indicates the first The cross-modal local matching representation is the first The routing weights selected by the expert network; An index representing the cross-modal local matching representation; An index representing the expert network; This represents a preset positive number used to prevent the denominator from being zero.

[0026] Preferably, the matching characterization projection network includes a first normalization layer, a first linear projection layer, a first Gaussian error linear unit activation layer, and a second linear projection layer connected in sequence; the expert routing unit includes a linear router and a Sigmoid function layer connected in sequence.

[0027] Preferably, the cross-modal local matching representations are input into the corresponding selected expert network to generate text-side feature corrections, image-side feature corrections, and matching correction values; the corresponding text tag features and image patch features are updated based on the text-side feature corrections and image-side feature corrections, specifically including:

[0028] Will be the The expert network selected the first The first cross-modal local matching representation input An expert network is used to obtain text-side feature corrections, image-side feature corrections, and matching correction values, as shown in the following formula:

[0029] ;

[0030] in, Indicates the first The function corresponding to each expert network; Indicates the first The expert network is targeting the first The text-side feature correction amount generated by cross-modal local matching representation; Indicates the first The expert network is targeting the first The image end feature correction amount generated by cross-modal local matching characterization; Indicates the first The expert network is targeting the first Each cross-modal local matching characterizes the output matching correction value;

[0031] The text-side feature corrections and image-side feature corrections from multiple expert networks are weighted according to routing weights to obtain the first... The text-side weighted correction and image-side weighted correction corresponding to each cross-modal local matching representation are shown in the following formula:

[0032] ;

[0033] ;

[0034] in, and They represent the first The text-to-image patch matching pairs correspond to the text-side weighted correction and the image-side weighted correction.

[0035] According to the The index of the corresponding text tag features for each text tag-image patch matching pair. and index of image patch features The weighted correction values ​​for the text and the image are respectively added to the positions of the corresponding text marker features and image patch features to obtain the corresponding text marker feature update values ​​and image patch feature update values, as shown in the following formula:

[0036] ;

[0037] ;

[0038] in, and They respectively represent aggregation to the first The text tag feature update amount at the _th text tag position and the aggregation to the _th Image patch feature update amount for each image patch location; Indicates all that satisfy of ;

[0039] The text tag features and image patch features are updated based on the text tag feature update amount, the image patch feature update amount, and the preset residual scaling factor, as shown in the following formula:

[0040] ;

[0041] in, Indicates the updated number Each text tag feature; Indicates the updated number Image patch features; This indicates the preset residual scaling factor.

[0042] As a preferred option, the first Each expert network consists of a second normalization layer, a first linear layer, a second Gaussian error linear unit activation layer, and a second linear layer connected in sequence. After the second linear layer, a text correction projection layer, an image correction projection layer, and a matching correction projection layer are set to output the text-side feature correction amount, the image-side feature correction amount, and the matching correction value, respectively.

[0043] Preferably, the initial cross-modal similarity is calculated based on global text features and global visual features, as shown in the following formula:

[0044] ;

[0045] in, This represents the global text features output by a pre-trained text encoder; This represents the global visual features output by the pre-trained visual encoder; Describing the L2 norm, This represents the initial cross-modal similarity.

[0046] Preferably, the matching correction values ​​output by multiple expert networks are weighted according to the routing weights to obtain candidate matching correction terms, which specifically include:

[0047] The candidate matching correction terms are obtained by weighting the matching correction values ​​output by each expert network in the near-neighbor matching hybrid expert reordering module according to the routing weights. Specifically:

[0048] ;

[0049] in, Indicates the first The expert network is targeting the first The matching correction value output by the cross-modal local matching representation; the natural language text of the person to be queried and the candidate pedestrian images constitute candidate image-text pairs. This indicates the total number of text tag-image patch matching pairs in the current candidate text-image pair; This indicates taking the maximum value; This indicates a candidate matching correction term.

[0050] Preferably, the re-ranking score is calculated based on the updated text features, updated image features, candidate matching correction terms, and initial cross-modal similarity, specifically including:

[0051] The updated text tag features and updated image patch features are then subjected to mean pooling and normalization operations, respectively, to obtain enhanced global text features. With enhanced global visual features As shown in the following formula:

[0052] ;

[0053] ;

[0054] The reordering score for candidate image-text pairs is calculated using the following formula:

[0055] ;

[0056] in, The reordering score for candidate image-text pairs; Cosine similarity; and This represents the first adjustable weight coefficient and the second adjustable weight coefficient.

[0057] Compared with the prior art, the present invention has the following beneficial effects:

[0058] (1) The text-image pedestrian retrieval method based on nearest neighbor matching and hybrid experts proposed in this invention calculates the local similarity between text tag features and image patch features, and combines the bidirectional nearest neighbor selection results from text to image and image to text to filter text tag-image patch matching pairs that are neighbors; only matching relationships with high local similarity in both directions are retained. Compared with dense matching or unidirectional nearest neighbor matching, this invention can reduce the situation where background, occluded areas and semantically ambiguous local features form incorrect matching pairs, reduce the impact of accidental high similarity local relationships on the matching results, and improve the reliability of the matching relationship between fine-grained text description and pedestrian local visual regions.

[0059] (2) The text-image pedestrian retrieval method based on nearest-neighbor matching hybrid experts proposed in this invention calculates the fit between cross-modal local matching representations and different expert networks, and each expert network selects the corresponding cross-modal local matching representations for differentiated processing under a preset capacity constraint; the expert networks generate text-side feature correction values, image-side feature correction values, and matching correction values ​​respectively, and update text features and image features collaboratively based on routing weights, and complete the re-ranking of candidate text-image pairs by combining the initial cross-modal similarity and candidate matching correction terms. This method can enhance the model's ability to express different fine-grained matching modes, achieve adaptive correction of local cross-modal matching relationships, reduce the interference of local erroneous matching on the final ranking results, and improve the accuracy of text-image pedestrian retrieval in complex scenarios. Attached Figure Description

[0060] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0061] Figure 1 This is a flowchart illustrating the text-image pedestrian retrieval method based on nearest neighbor matching hybrid experts, as an embodiment of this application.

[0062] Figure 2 This is a schematic diagram of the overall architecture of the text-image pedestrian retrieval method based on nearest neighbor matching hybrid experts, which is an embodiment of this application.

[0063] Figure 3 This is a schematic diagram of the structure of the nearest neighbor matching hybrid expert re-ranking module in the text-image pedestrian retrieval method based on nearest neighbor matching hybrid expert, which is an embodiment of this application. Detailed Implementation

[0064] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of this invention, and not all embodiments. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0065] Figure 1 The present invention, as illustrated in the embodiments of this application, provides a text-image pedestrian retrieval method based on nearest neighbor matching hybrid experts, comprising the following steps:

[0066] S1. Construct and train a text-image pedestrian retrieval model to obtain a trained text-image pedestrian retrieval model. The text-image pedestrian retrieval model includes a visual branch, a text branch, and a nearest neighbor matching hybrid expert re-ranking module. The visual branch includes a pre-trained visual encoder, used to extract global visual features and image patch feature sequences corresponding to pedestrian images. The text branch includes a pre-trained text encoder, used to extract global text features and text tag feature sequences corresponding to natural language text descriptions. The visual branch of the trained text-image pedestrian retrieval model is used to encode several pedestrian images, extract global visual features and image patch feature sequences, and construct a pedestrian image feature database.

[0067] Specifically, refer to Figure 2The text-image pedestrian retrieval model proposed in this application includes a visual branch, a text branch, and a nearest neighbor matching hybrid expert re-ranking module. The visual branch is a pre-trained visual encoder used to extract global visual features and image patch feature sequences corresponding to pedestrian images; the text branch is a pre-trained text encoder used to extract global text features and text tag feature sequences corresponding to natural language text descriptions; the nearest neighbor matching hybrid expert re-ranking module is connected after the visual branch and the text branch, and is used to perform local matching correction and re-ranking of candidate image-text pairs.

[0068] In one example, the pre-trained visual encoder and pre-trained text encoder in the embodiments of this application may employ a Contrastive Language-Image Pre-Training (CLIP) model as the structure in the visual and text dual-branch Transformer backbone network. The visual and text dual-branch Transformer backbone network includes an image feature embedding layer, a text tag embedding layer, and multiple stacked Transformer encoder layers.

[0069] The nearest-neighbor matching hybrid expert re-ranking module includes a local similarity calculation unit, a nearest-neighbor matching filtering unit, a cross-modal local matching representation construction unit, an expert routing unit, a parallel expert network unit, a feature update unit, and a re-ranking score calculation unit. Specifically, the local similarity calculation unit performs cross-modal similarity calculation between global text features and global visual features in the pedestrian image feature database to obtain an initial cross-modal similarity, and then filters the top K candidate pedestrian images based on this initial cross-modal similarity. The nearest-neighbor matching filtering unit filters the text tag feature sequence and the corresponding image patch feature sequences of each candidate pedestrian image to obtain nearest-neighbor text tag-image patch matching pairs, and generates a cross-modal local matching representation based on these pairs. The parallel expert network unit includes multiple expert networks with independent parameters. The expert routing unit calculates the expert adaptation score based on the cross-modal local matching representation and selects the corresponding cross-modal local matching representation for each expert network under a preset expert capacity constraint. Each expert network processes the selected cross-modal local matching representation, outputting text-side feature corrections, image-side feature corrections, and matching correction values. The feature update unit updates the corresponding text tag features and image patch features based on the text-side and image-side feature corrections. The re-ranking score calculation unit weights the matching correction values ​​output by multiple expert networks according to routing weights to obtain candidate matching correction terms. The re-ranking score is calculated based on the updated text features, updated image features, candidate matching correction terms, and initial cross-modal similarity. The text-image person retrieval result is determined based on the re-ranking score. During training, the global visual features and image patch feature sequences output by the visual branch and the global text features and text tag feature sequences output by the text branch are used simultaneously, and the text-image person retrieval model is trained based on image-text matching tags. The nearest-neighbor matching hybrid expert re-ranking module can be jointly trained with the pre-trained visual encoder and pre-trained text encoder, or only the nearest-neighbor matching hybrid expert re-ranking module can be trained while freezing the pre-trained visual encoder and pre-trained text encoder. After obtaining the trained text-image pedestrian retrieval model, the visual branch can be used to extract the global visual features and image patch feature sequences of each pedestrian image in the pedestrian image database to construct a pedestrian image feature database.

[0070] Specifically, during the training process of the text-image pedestrian retrieval model, collected image samples and their corresponding text descriptions are combined to form image-text pairs, and several image-text pairs form training batches. The image samples and text descriptions in the training batches are respectively input into the pre-trained visual encoder and text encoder to extract the global visual features and image patch feature sequences corresponding to the image samples, as well as the global text features and text tag feature sequences corresponding to the text descriptions.

[0071] Furthermore, an initial cross-modal similarity is calculated based on the global text features of each text description and the global visual features of each image sample in the training batch, and candidate image-text pairs are constructed based on the initial cross-modal similarity. For each candidate image-text pair, the corresponding text tag feature sequence, image patch feature sequence, and initial cross-modal similarity are input into the nearest neighbor matching hybrid expert re-ranking module to obtain the re-ranking score for each candidate image-text pair. The re-ranking scores corresponding to multiple image-text pairs within the same training batch are arranged to form a re-ranking score matrix, where the matrix elements corresponding to the true matching image-text pairs are positive matching scores, and the remaining matrix elements are negative matching scores.

[0072] In one embodiment, the main loss function of the text-image pedestrian retrieval model is an image-text contrast loss function. This image-text contrast loss function constrains truly matching image-text pairs to have higher re-ranking scores and constrains non-matching image-text pairs to have lower re-ranking scores. Further, the image-text contrast loss function includes a text-to-image contrast loss and an image-to-text contrast loss, and the weighted average of these two losses is used as the training loss.

[0073] In one example, the image-text pair to be processed is input into the text-image pedestrian retrieval model. Global visual features and image patch feature sequences corresponding to the image samples, as well as global text features and text tag feature sequences corresponding to the text descriptions, are extracted using a pre-trained visual encoder and a pre-trained text encoder, respectively. An initial cross-modal similarity is calculated based on the global text features and global visual features, and candidate image-text pairs are constructed based on this initial cross-modal similarity. For each candidate image-text pair, the corresponding text tag feature sequence, image patch feature sequence, and initial cross-modal similarity are input into the nearest neighbor matching hybrid expert re-ranking module.

[0074] The nearest neighbor matching hybrid expert re-ranking module first calculates the local similarity matrix between text tags and image patches, and then filters text tag-image patch matching pairs that simultaneously satisfy both text-side and image-side nearest neighbor conditions. Subsequently, it constructs cross-modal local matching representations based on these matching pairs, and selects corresponding cross-modal local matching representations for multiple expert networks under a preset expert capacity constraint through an expert routing unit. Each expert network processes the selected cross-modal local matching representations, outputting text-side feature corrections, image-side feature corrections, and matching correction values. The matching correction values ​​output by multiple expert networks are weighted according to routing weights to obtain candidate matching correction terms. The text-side and image-side feature corrections are aggregated according to routing weights to update the text tag feature sequence and image patch feature sequence. Finally, the re-ranking score of the candidate text-image pair is calculated based on the updated text features, updated image features, candidate matching correction terms, and the initial cross-modal similarity.

[0075] During training, an image-text contrast loss function is calculated based on the re-ranking score matrix to constrain truly matching image-text pairs to have higher re-ranking scores and non-matching image-text pairs to have lower re-ranking scores. Optionally, one or more of identity classification loss, matching correction value auxiliary loss, or expert load balancing loss can be combined to construct a total loss function, and the model parameters are updated using gradient descent until a preset training stopping condition is met, resulting in a trained text-image pedestrian retrieval model. After training, the global visual features and image patch feature sequences extracted from pedestrian images with known identities using a visual encoder can be stored in a pedestrian image feature database as a reference for subsequent natural language description retrieval.

[0076] S2: Obtain the natural language text description of the person to be queried and input it into the trained text-image pedestrian retrieval model. After text branching, extract the global text features and text tag feature sequences of the person to be queried. Input the global text features and text tag feature sequences of the person to be queried, as well as the global visual features and image patch feature sequences from the pedestrian image feature database, into the nearest neighbor matching hybrid expert re-ranking module. First, perform cross-modal similarity calculation between the global text features and the global visual features from the pedestrian image feature database to obtain the initial cross-modal similarity. Then, select the top K candidate pedestrian images based on the initial cross-modal similarity. Based on the text tag feature sequences, filter the image patch feature sequences corresponding to each candidate pedestrian image to obtain text tag-image patch matching pairs that are neighbors. Generate cross-modal local matching representations; input each cross-modal local matching representation into an expert routing unit to select the corresponding expert network, and calculate the routing weights corresponding to each expert network selected by each cross-modal local matching representation; input each cross-modal local matching representation into the corresponding selected expert network to generate text-side feature correction, image-side feature correction, and matching correction values; update the corresponding text tag features and image patch features based on the text-side feature correction and image-side feature correction; weight the matching correction values ​​output by multiple expert networks according to the routing weights to obtain candidate matching correction terms; calculate the re-ranking score based on the updated text features, updated image features, candidate matching correction terms, and initial cross-modal similarity; and determine the text-image pedestrian retrieval results for the person to be queried based on the re-ranking score.

[0077] Specifically, the trained text-image pedestrian retrieval model is deployed. During the inference process, the natural language text description of the person to be queried is input into the trained text-image pedestrian retrieval model. The text branch is used to extract the global text features and text tag feature sequences corresponding to the natural language text description of the person to be queried. These are then combined with the global visual features and image patch feature sequences from the pedestrian image feature database and input into the nearest neighbor matching hybrid expert re-ranking module to filter out candidate pedestrian images. The natural language text of the person to be queried and the candidate pedestrian images are combined to form candidate image-text pairs. The candidate image-text pairs are then re-ranked to obtain the text-image pedestrian retrieval results.

[0078] The following details the specific process of each step in the nearest neighbor matching hybrid expert reordering module.

[0079] In a specific embodiment, filtering text tag-image patch matching pairs that are neighbors includes:

[0080] The text marker feature sequence and the corresponding image patch feature sequence of the candidate pedestrian image are represented as follows:

[0081] ;

[0082] ;

[0083] in, Represents a sequence of text tag features; Represents the feature sequence of an image patch; Indicates the first Each text tag feature; Indicates the first Image patch features; This represents the total number of text tag features; This represents the total number of features in an image patch; Let T denote the feature dimension, and T denote the transpose of the matrix. Represents the set of real numbers;

[0084] The text marker feature sequence and the image patch feature sequence are normalized using L2 norm, and the local similarity matrix is ​​calculated as follows:

[0085] ;

[0086] in, This represents the L2 norm normalization operation performed along the feature dimension; Represents the local similarity matrix;

[0087] For any text tag feature and any image patch feature, determine whether they constitute a nearest neighbor text tag-image patch matching pair according to the following formula:

[0088] ;

[0089] in, Indicates the first The text tag feature and the first Nearest neighbor matching indicator values ​​between features of image patches; when At that time, the first The text tag feature and the first If the features of each image patch constitute a text tag-image patch matching pair that are neighbors, then no text tag-image patch matching pair will be formed. This indicates an indicator function; its value is 1 if the condition is met, and 0 otherwise. Represents the first element in the local similarity matrix. The vector formed by the local similarity between each text tag feature and all image patch features. Represents the first element in the local similarity matrix. The vector formed by the local similarity between each image patch feature and all text tag features; Indicates the first The vector formed by the local similarities between each text tag feature and all image patch features is the one with the highest local similarity. Image patch features; Indicates the first The vector formed by the local similarities between each image patch feature and all text tag features is the one with the highest local similarity. Each text tag feature; , min means taking the minimum value; This indicates the preset number of nearest neighbors.

[0090] Specifically, refer to Figure 3 In the nearest neighbor matching hybrid expert reordering module mentioned in the embodiments of this application, the local similarity between any text tag feature and any image patch feature is first calculated and the nearest neighbor matching indicator value between the two is calculated. The nearest neighbor matching indicator value is used to filter out the text tag-image patch matching pairs that are neighbors.

[0091] In a specific embodiment, generating a cross-modal local matching representation based on text tag-image patch matching pairs specifically includes:

[0092] Regarding the first For each text tag-image patch matching pair, the corresponding text tag features, image patch features, their element-wise product, their element-wise absolute difference, and their corresponding local similarity are concatenated, and then feature mapping is performed through a matching representation projection network, as shown in the following equation:

[0093] ;

[0094] in, Indicates the index of the text tag-image patch matching pair; and They represent the first Each text tag-image patch matching pair contains the indexes of the corresponding text tag features and the indexes of the image patch features; Indicates the first Cross-modal local matching representations corresponding to each text tag-image patch matching pair; Indicates feature concatenation operation; This represents element-wise multiplication; This indicates taking the absolute value of each element; This indicates a matching representation of the projection network.

[0095] In a specific embodiment, each cross-modal local matching representation is input into the expert routing unit to select the corresponding expert network, and the routing weights corresponding to each expert network selected by each cross-modal local matching representation are calculated, specifically including:

[0096] Each cross-modal local matching representation is input into a linear router, and the corresponding expert fit score is calculated through a Sigmoid function layer. The effective capacity of a single expert network is determined based on the total number of cross-modal local matching representations, the total number of expert networks, and the expert capacity factor. Finally, the network with the highest corresponding expert fit score is selected from all cross-modal local matching representations. Each cross-modal local matching characterization is used to calculate the corresponding routing weights, as shown in the following formula:

[0097] ;

[0098] in, Indicates the first The cross-modal local matching representation for the _th Expert fit score of an expert network; This represents the learnable weight matrix of a linear router; Represents the first in the vector One element; Represents the Sigmoid function; This represents the effective capacity of a single expert network; Indicates the expert capacity factor; This represents the total number of cross-modal local matching representations of the current input; This represents the total number of expert networks; Indicates rounding down; Indicates the first Is the cross-modal local matching representation of the i-th modality represented by the i-th modality local matching representation? The indicator value selected by the expert network is 1 if it is selected, and 0 otherwise. Indicates the first The cross-modal local matching representation is the first The routing weights selected by the expert network; An index representing the cross-modal local matching representation; An index representing the expert network; This represents a preset positive number used to prevent the denominator from being zero.

[0099] In a specific embodiment, the matching characterization projection network includes a first normalization layer, a first linear projection layer, a first Gaussian error linear unit activation layer, and a second linear projection layer connected in sequence; the expert routing unit includes a linear router and a Sigmoid function layer connected in sequence.

[0100] Specifically, in the embodiments of this application, cross-modal local matching representations are generated based on neighboring text tag-image patch matching pairs. This is implemented using a matching representation projection network, which concatenates the text tag features, image patch features, their element-wise product, their element-wise absolute difference, and the corresponding local similarity of each text tag-image patch matching pair to obtain a dimension of [missing information]. The concatenation feature mapping is of dimension . The cross-modal local matching representation is then used. The expert routing unit calculates the corresponding expert fit score for each cross-modal local matching representation, and selects the top-ranked representations with the highest expert fit scores from all cross-modal local matching representations based on these scores. Each cross-modal local matching representation is used to calculate the corresponding routing weights. Each cross-modal local matching representation is the input feature fed into the corresponding expert network. Expert fit scores are calculated, and a fixed-capacity selection of expert networks is performed. This involves each expert network selecting its top C cross-modal local matching representations with the highest fit, rather than selecting the same number of expert networks for each cross-modal local matching representation.

[0101] In a specific embodiment, the cross-modal local matching representations are input into the corresponding selected expert network to generate text-side feature corrections, image-side feature corrections, and matching correction values; the corresponding text tag features and image patch features are updated based on the text-side feature corrections and image-side feature corrections, specifically including:

[0102] Will be the The expert network selected the first The first cross-modal local matching representation input An expert network is used to obtain text-side feature corrections, image-side feature corrections, and matching correction values, as shown in the following formula:

[0103] ;

[0104] in, Indicates the first The function corresponding to each expert network; Indicates the first The expert network is targeting the first The text-side feature correction amount generated by cross-modal local matching representation; Indicates the first The expert network is targeting the first The image end feature correction amount generated by cross-modal local matching characterization; Indicates the first The expert network is targeting the first Each cross-modal local matching characterizes the output matching correction value;

[0105] The text-side feature corrections and image-side feature corrections from multiple expert networks are weighted according to routing weights to obtain the first... The text-side weighted correction and image-side weighted correction corresponding to each cross-modal local matching representation are shown in the following formula:

[0106] ;

[0107] ;

[0108] in, and They represent the first The text-to-image patch matching pairs correspond to the text-side weighted correction and the image-side weighted correction.

[0109] According to the The index of the corresponding text tag features for each text tag-image patch matching pair. and index of image patch features The weighted correction values ​​for the text and the image are respectively added to the positions of the corresponding text marker features and image patch features to obtain the corresponding text marker feature update values ​​and image patch feature update values, as shown in the following formula:

[0110] ;

[0111] ;

[0112] in, and They respectively represent aggregation to the first The text tag feature update amount at the _th text tag position and the aggregation to the _th Image patch feature update amount for each image patch location; Indicates all that satisfy of ;

[0113] The text tag features and image patch features are updated based on the text tag feature update amount, the image patch feature update amount, and the preset residual scaling factor, as shown in the following formula:

[0114] ;

[0115] in, Indicates the updated number Each text tag feature; Indicates the updated number Image patch features; This indicates the preset residual scaling factor.

[0116] In a specific embodiment, the first Each expert network consists of a second normalization layer, a first linear layer, a second Gaussian error linear unit activation layer, and a second linear layer connected in sequence. After the second linear layer, a text correction projection layer, an image correction projection layer, and a matching correction projection layer are set to output the text-side feature correction amount, the image-side feature correction amount, and the matching correction value, respectively.

[0117] In a specific embodiment, the initial cross-modal similarity is calculated based on global text features and global visual features, as shown in the following formula:

[0118] ;

[0119] in, This represents the global text features output by a pre-trained text encoder; This represents the global visual features output by the pre-trained visual encoder; Describing the L2 norm, This represents the initial cross-modal similarity.

[0120] In a specific embodiment, the matching correction values ​​output by multiple expert networks are weighted according to the routing weights to obtain candidate matching correction terms, specifically including:

[0121] The candidate matching correction terms are obtained by weighting the matching correction values ​​output by each expert network in the near-neighbor matching hybrid expert reordering module according to the routing weights. Specifically:

[0122] ;

[0123] in, Indicates the first The expert network is targeting the first The matching correction value output by the cross-modal local matching representation; the natural language text of the person to be queried and the candidate pedestrian images constitute candidate image-text pairs. This indicates the total number of text tag-image patch matching pairs in the current candidate text-image pair; This indicates taking the maximum value; This indicates a candidate matching correction term.

[0124] In a specific embodiment, the re-ranking score is calculated based on the updated text features, updated image features, candidate matching correction terms, and initial cross-modal similarity, specifically including:

[0125] The updated text tag features and updated image patch features are then subjected to mean pooling and normalization operations, respectively, to obtain enhanced global text features. With enhanced global visual features As shown in the following formula:

[0126] ;

[0127] ;

[0128] The reordering score for candidate image-text pairs is calculated using the following formula:

[0129] ;

[0130] in, The reordering score for candidate image-text pairs; Cosine similarity; and This represents the first adjustable weight coefficient and the second adjustable weight coefficient.

[0131] Specifically, each expert network generates corresponding text-side feature corrections, image-side feature corrections, and matching correction values, and updates the corresponding text tag features and image patch features. Mean pooling and L2 normalization are then performed on the updated text tag features and image patch features to obtain enhanced global text features and enhanced global visual features. The matching correction values ​​output by each expert network are weighted using routing weights and normalized according to the number of text tag-image patch matching pairs in the same candidate text-image pair to obtain candidate matching correction terms. Finally, the re-ranking scores of candidate text-image pairs are calculated, and the candidate text-image pair with the highest re-ranking score is selected. The pedestrian image in this candidate text-image pair is used as the image result retrieved based on the natural language text description of the person being queried. If the pedestrian images in the pedestrian image feature database have corresponding known identities, the candidate pedestrian image with the highest re-ranking score can be found based on the text-image pedestrian retrieval results, and the identity of the person being queried can be determined.

[0132] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0133] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A text-image pedestrian retrieval method based on nearest neighbor matching hybrid experts, characterized in that, Includes the following steps: A text-image pedestrian retrieval model is constructed and trained to obtain a trained text-image pedestrian retrieval model. The text-image pedestrian retrieval model includes a visual branch, a text branch, and a nearest neighbor matching hybrid expert re-ranking module. The visual branch includes a pre-trained visual encoder for extracting global visual features and image patch feature sequences corresponding to pedestrian images. The text branch includes a pre-trained text encoder for extracting global text features and text tag feature sequences corresponding to natural language text descriptions. Several pedestrian images are encoded using the visual branch of the trained text-image pedestrian retrieval model to extract global visual features and image patch feature sequences, and a pedestrian image feature database is constructed. The system obtains the natural language text description of the person to be queried and inputs it into a trained text-image pedestrian retrieval model. After text branching, it extracts the global text features and text tag feature sequences of the person to be queried. The global text features and text tag feature sequences of the person to be queried, along with the global visual features and image patch feature sequences from the pedestrian image feature database, are input into the nearest neighbor matching hybrid expert re-ranking module. First, the global text features and the global visual features from the pedestrian image feature database are used to calculate cross-modal similarity, obtaining an initial cross-modal similarity. Then, the top K candidate pedestrian images are selected based on the initial cross-modal similarity. Finally, based on the text tag features... The sequence filters the image patch feature sequences corresponding to each candidate pedestrian image to obtain neighboring text tag-image patch matching pairs. Cross-modal local matching representations are generated based on these text tag-image patch matching pairs. Each cross-modal local matching representation is input into an expert routing unit to select the corresponding expert network, and the routing weights corresponding to each expert network selection are calculated. Each cross-modal local matching representation is input into the corresponding selected expert network to generate text-side feature correction, image-side feature correction, and matching correction values. The corresponding text tag features and image patch features are updated based on the text-side feature correction and image-side feature correction. The matching correction values ​​output by multiple expert networks are weighted according to the routing weights to obtain candidate matching correction items; the re-ranking score is calculated based on the updated text features, updated image features, candidate matching correction items and initial cross-modal similarity; and the text-image pedestrian retrieval result of the person to be queried is determined based on the re-ranking score.

2. The text-image pedestrian retrieval method based on nearest neighbor matching hybrid experts according to claim 1, characterized in that, Filtering text tag-image patch matching pairs that are neighbors, specifically including: The text marker feature sequence and the corresponding image patch feature sequence of the candidate pedestrian image are represented as follows: ； ； in, Represents a sequence of text tag features; Represents the feature sequence of an image patch; Indicates the first Each text tag feature; Indicates the first Image patch features; This represents the total number of text tag features; This represents the total number of features in an image patch; Let T denote the feature dimension, and T denote the transpose of the matrix. Represents the set of real numbers; The text marker feature sequence and the image patch feature sequence are respectively normalized to L2, and the local similarity matrix is ​​calculated as shown in the following formula: ； in, This represents the L2 norm normalization operation performed along the feature dimension; Represents the local similarity matrix; For any text tag feature and any image patch feature, determine whether they constitute a nearest neighbor text tag-image patch matching pair according to the following formula: ； in, Indicates the first The text tag feature and the first Nearest neighbor matching indicator values ​​between features of image patches; when At that time, the first The text tag feature and the first If the features of each image patch constitute a text tag-image patch matching pair that are neighbors, then no text tag-image patch matching pair will be formed. This indicates an indicator function; its value is 1 if the condition is met, and 0 otherwise. Represents the first element in the local similarity matrix. The vector formed by the local similarity between each text tag feature and all image patch features. Represents the first element in the local similarity matrix. The vector formed by the local similarity between each image patch feature and all text tag features; Indicates the first The vector formed by the local similarities between each text tag feature and all image patch features is the one with the highest local similarity. Image patch features; Indicates the first The vector formed by the local similarities between each image patch feature and all text tag features is the one with the highest local similarity. Each text tag feature; , min means taking the minimum value; This indicates the preset number of nearest neighbors.

3. The text-image pedestrian retrieval method based on nearest neighbor matching hybrid experts according to claim 2, characterized in that, Generate a cross-modal local matching representation based on the text tag-image patch matching pair, specifically including: Regarding the first For each text tag-image patch matching pair, the corresponding text tag features, image patch features, their element-wise product, their element-wise absolute difference, and their corresponding local similarity are concatenated, and then feature mapping is performed through a matching representation projection network, as shown in the following equation: ； in, Indicates the index of the text tag-image patch matching pair; and They represent the first Each text tag-image patch matching pair contains the indexes of the corresponding text tag features and the indexes of the image patch features; Indicates the first Cross-modal local matching representations corresponding to each text tag-image patch matching pair; Indicates feature concatenation operation; This represents element-wise multiplication; This indicates taking the absolute value of each element; This indicates a matching representation of the projection network.

4. The text-image pedestrian retrieval method based on nearest neighbor matching hybrid experts according to claim 3, characterized in that, Each cross-modal local matching representation is input into the expert routing unit to select the corresponding expert network, and the routing weights corresponding to each cross-modal local matching representation and each expert network are calculated, specifically including: Each cross-modal local matching representation is input into a linear router, and the corresponding expert fit score is calculated through a Sigmoid function layer. The effective capacity of a single expert network is determined based on the total number of cross-modal local matching representations, the total number of expert networks, and the expert capacity factor. Finally, the network with the highest corresponding expert fit score is selected from all cross-modal local matching representations. Each cross-modal local matching representation is used to calculate the corresponding routing weight, as shown in the following formula: ； in, Indicates the first The cross-modal local matching representation for the _th Expert fit score of an expert network; This represents the learnable weight matrix of a linear router; Represents the first in the vector One element; Represents the Sigmoid function; This represents the effective capacity of a single expert network; Indicates the expert capacity factor; This represents the total number of cross-modal local matching representations of the current input; This represents the total number of expert networks; Indicates rounding down; Indicates the first Is the cross-modal local matching representation of the i-th modality represented by the i-th modality local matching representation? The indicator value selected by the expert network is 1 if it is selected, and 0 otherwise. Indicates the first The cross-modal local matching representation is the first The routing weights selected by the expert network; An index representing the cross-modal local matching representation; An index representing the expert network; This represents a preset positive number used to prevent the denominator from being zero.

5. The text-image pedestrian retrieval method based on nearest neighbor matching hybrid experts according to claim 4, characterized in that, The matching representation projection network includes a first normalization layer, a first linear projection layer, a first Gaussian error linear unit activation layer, and a second linear projection layer connected in sequence; the expert routing unit includes a linear router and a Sigmoid function layer connected in sequence.

6. The text-image pedestrian retrieval method based on nearest neighbor matching hybrid experts according to claim 4, characterized in that, Each cross-modal local matching representation is input into the corresponding selected expert network to generate text-side feature correction, image-side feature correction, and matching correction value; Update the corresponding text marker features and image patch features based on the text-side feature correction amount and the image-side feature correction amount, specifically including: Will be the The expert network selected the first The cross-modal local matching representation input is the first... An expert network is used to obtain text-side feature corrections, image-side feature corrections, and matching correction values, as shown in the following formula: ； in, Indicates the first The function corresponding to each expert network; Indicates the first The expert network is targeting the first The text-side feature correction amount generated by cross-modal local matching representation; Indicates the first The expert network is targeting the first The image end feature correction amount generated by cross-modal local matching characterization; Indicates the first The expert network is targeting the first Each cross-modal local matching characterizes the output matching correction value; The text-side feature corrections and image-side feature corrections output by multiple expert networks are weighted according to the routing weights to obtain the first... The text-side weighted correction and image-side weighted correction corresponding to each cross-modal local matching representation are shown in the following formula: ； ； in, and They represent the first The text-to-image patch matching pairs correspond to the text-side weighted correction and the image-side weighted correction. According to the The index of the corresponding text tag features for each text tag-image patch matching pair. and index of image patch features The text-side weighted correction and image-side weighted correction are respectively accumulated at the positions of the corresponding text marker features and image patch features to obtain the corresponding text marker feature update and image patch feature update, as shown in the following formula: ； ； in, and They respectively represent aggregation to the first The text tag feature update amount at the _th text tag position and the aggregation to the _th Image patch feature update amount for each image patch location; Indicates all that satisfy of ; The text marker features and image patch features are updated based on the text marker feature update amount, the image patch feature update amount, and the preset residual scaling factor, as shown in the following formula: ； in, Indicates the updated number Each text tag feature; Indicates the updated number Image patch features; This indicates the preset residual scaling factor.

7. The text-image pedestrian retrieval method based on nearest neighbor matching hybrid experts according to claim 6, characterized in that, The first Each expert network includes a second normalization layer, a first linear layer, a second Gaussian error linear unit activation layer, and a second linear layer connected in sequence. After the second linear layer, a text correction projection layer, an image correction projection layer, and a matching correction projection layer are set to output the text-side feature correction amount, the image-side feature correction amount, and the matching correction value, respectively.

8. The text-image pedestrian retrieval method based on nearest neighbor matching hybrid experts according to claim 6, characterized in that, The initial cross-modal similarity is calculated based on global text features and global visual features, as shown in the following formula: ； in, This represents the global text features output by a pre-trained text encoder; This represents the global visual features output by the pre-trained visual encoder; Describing the L2 norm, This represents the initial cross-modal similarity.

9. The text-image pedestrian retrieval method based on nearest neighbor matching hybrid experts according to claim 8, characterized in that, The matching correction values ​​output by multiple expert networks are weighted according to the routing weights to obtain candidate matching correction terms, which specifically include: The candidate matching correction terms are obtained by weighting the matching correction values ​​output by each expert network in the nearest neighbor matching hybrid expert reordering module according to the routing weights. Specifically: ； in, Indicates the first The expert network is targeting the first The matching correction value output by the cross-modal local matching representation; the natural language text of the person to be queried and the candidate pedestrian image constitute a candidate image-text pair. This indicates the total number of text tag-image patch matching pairs in the current candidate text-image pair; This indicates taking the maximum value; This indicates a candidate matching correction term.

10. The text-image pedestrian retrieval method based on nearest neighbor matching hybrid experts according to claim 9, characterized in that, The re-ranking score is calculated based on the updated text features, updated image features, the candidate matching correction term, and the initial cross-modal similarity, specifically including: The updated text marker features and the updated image patch features are then subjected to mean pooling and normalization operations, respectively, to obtain enhanced global text features. With enhanced global visual features As shown in the following formula: ； ； The reordering score for the candidate image-text pairs is calculated according to the following formula: ； in, The reordering score for candidate image-text pairs; Cosine similarity; and This represents the first adjustable weight coefficient and the second adjustable weight coefficient.

Images