Text-image-based cross-modal pedestrian re-identification method and system, computer equipment and computer readable storage medium

Abstract

The invention discloses a text-image-based cross-modal pedestrian re-identification method and system, computer equipment and a computer readable storage medium, and belongs to the cross technical field of computer vision and natural language process.The method comprises the steps that a local alignment network is added, the weight of a text mark vector corresponding to each local feature is learned, and the weight of the text mark vector corresponding to each local feature is obtained; weighting to obtain corresponding text local features, enabling the image local features to correspond to the text local features, and carrying out cross-modal local fine-grained alignment; an attribute alignment network is added, the attention position of each attribute is calculated according to the image block mark vector, and cross-modal attribute coarse-grained alignment is carried out by taking an attribute label obtained according to the attributes in the text as supervision; and performing cosine similarity sorting on the obtained global image features and global text features to obtain a pedestrian re-identification retrieval result. The method has the advantages of being high in semantic consistency, high in robustness and good in interpretability.

Description

A text-image based cross-modal person re-identification method and system, computer equipment, and computer-readable storage medium. Technical Field

[0001] This application relates to the interdisciplinary fields of computer vision and natural language processing, and in particular to a cross-modal pedestrian re-identification method and system based on text-image, computer equipment, and computer-readable storage medium. Background Technology

[0002] Person re-identification (Re-ID) aims to identify the same pedestrian across cameras and scenes. Traditional methods are mainly based on image modalities, relying on pedestrian appearance features (such as color, texture, and pose). However, in complex scenes such as changes in lighting, occlusion, and differences in viewing angle, image quality degrades, leading to a significant decrease in recognition performance.

[0003] In recent years, Text-Based Person Search (TBPS) has attracted attention, allowing users to retrieve target pedestrians using natural language descriptions, such as "wearing a red T-shirt, carrying a black backpack, and wearing a white hat." However, a significant semantic gap exists between text and images, and text descriptions are highly abstract and ambiguous, leading to difficulties in cross-modal alignment and low retrieval accuracy.

[0004] Existing methods are mostly based on global feature alignment or simple attention mechanisms, which fail to fully explore fine-grained semantic correspondences and have poor robustness to noise description. Summary of the Invention

[0005] This application aims to provide a text-image-based cross-modal person re-identification method and system, computer device, and computer-readable storage medium. Addressing the problem that discriminative semantics are overwhelmed by irrelevant information during aggregation due to factors such as occlusion, viewpoint changes, and pose shifts, a cross-modal local alignment method is proposed. This method deconstructs the image into several components and constructs a mapping between these components and text words, reducing the heterogeneous variance of similar cross-modal samples and thus compensating for the information loss in fine-grained discrimination caused by global alignment. To address the modal differences between text and images, a cross-modal attribute alignment method is proposed. This method uses text descriptions to label pedestrians with the same identity using attribute tags, and compares the attributes recognized in the image to construct a mapping between attributes and visual features, improving cross-modal semantic consistency. The method exhibits strong semantic consistency, high robustness, and good interpretability.

[0006] To achieve the above objectives, the technical solution of this application is: a text-image-based cross-modal person re-identification method, comprising: preparing a text-image person re-identification dataset; constructing attribute labels using text descriptions; adding a local alignment network based on a language-image pre-trained model to learn the weights of the text label vectors corresponding to each local feature, obtaining the corresponding text local features through weighted summation, and mapping the image local features and text local features to each other for cross-modal fine-grained local alignment; adding an attribute alignment network to calculate the attention position of each attribute based on the image patch label vectors, and assigning the attribute labels obtained from the attributes in the text. As supervision, coarse-grained alignment of cross-modal attributes is performed to obtain a text-image-based cross-modal person re-identification model. A loss function suitable for the text-image-based cross-modal person re-identification model is constructed. Based on minimizing the loss function, the parameters of the text-image-based cross-modal person re-identification model are iteratively updated to obtain the trained text-image-based cross-modal person re-identification model. The trained text-image-based cross-modal person re-identification model is used for person re-identification, and the obtained global image features and global text features are sorted by cosine similarity to obtain the person re-identification retrieval results.

[0007] Optionally, a text-image-based cross-modal person re-identification model is constructed by incorporating a local alignment network and attribute alignment features into the language-image pre-trained model. This includes: constructing an image branch to output image features, where the image branch includes an image segmentation layer, an image patch linear mapping layer, a location embedding layer, and a self-attention-based deep learning module encoding layer, used for image segmentation, image patch linear mapping, location embedding, and self-attention-based encoding to output image features, respectively; constructing a text branch to output text features, where the text branch includes a text mapping layer, a word embedding layer, a location embedding layer, and a self-attention-based deep learning module encoding layer, used for text mapping, word embedding, location embedding, and self-attention-based encoding to output text features, respectively; constructing a local alignment network for cross-modal fine-grained local alignment, where the local alignment network includes an image local feature extraction layer and a text local feature selection layer, used for image local feature extraction and text local feature extraction, respectively; and constructing an attribute alignment network for cross-modal coarse-grained attribute alignment, where the attribute alignment network includes an image attribute feature extraction layer and a text attribute label extraction layer, used for image attribute feature extraction and attribute matching label extraction, respectively.

[0008] Optionally, a local alignment network is constructed to perform cross-modal local fine-grained alignment, including: constructing an image local feature extraction layer, splitting the output features obtained from the image branch, including the learnable classification label vector and all image patch label vectors, and performing average pooling to calculate the mean of the features of all image patch label vectors to obtain the image local features, represented as follows:

[0009] in, The feature with index 0 in the output features of the image branch is the feature corresponding to the learnable classification label vector. The output features of the image branch are indexed from 1 to the end, that is, the features corresponding to all image patch label vectors; The global features of the image are extracted for image features, that is, the features corresponding to the learnable classification label vectors obtained by splitting the image. This is the set of image patch features obtained from the splitting process; This is the subset of image blocks corresponding to the nth local feature after average segmentation; This is the subset of image blocks corresponding to the local features of the Nth image after average segmentation; This represents the nth local feature of the image. This is the Nth local feature of the image; It is a set of local features of the image; This is the average splitting function; This refers to the number of segments, i.e., the number of local image features generated. For average pooling, a text local feature extraction layer is constructed. This layer learns the weights of the text marker vectors corresponding to each local feature using learnable local text parameters, and then weights these weights to obtain the corresponding text local features. This maps image local features to text local features, achieving cross-modal fine-grained alignment. Specifically, the text local feature extraction layer learns the weights of the text marker vectors corresponding to each local feature using learnable local text parameters, and then weights these weights to obtain the corresponding text local features. This process includes: encoding the discrete word sequence input to the text branch; sorting it from largest to smallest and then splitting it to obtain the positions of the insertion start and end marker vectors and all text marker vectors; and then, based on the text marker vectors in the output features of the text branch, learning the weights of the text marker vectors corresponding to each local feature using learnable local text parameters, and then weighting these weights to obtain the corresponding text local features, as shown below:

[0010] in, The original text description is the data after encoding and insertion of start and end marker vectors; A sorting function for encoding positions from largest to smallest; This is the set of position indices for the sorted text tag vectors; This is the position index of the Lth text tag vector after sorting; The text feature corresponding to the first position index in the output features of the text branch is the feature corresponding to the insertion end marker vector; The feature corresponding to the second position index in the output features of the text branch; The feature set indexed from the 3rd position to the end of the output features of the text branch; For text global features; Features for the insertion start marker vector position obtained from the split; The feature set of the sorted text marker vector positions obtained after splitting; This is a cross-modal attention module used to learn the weights of the text tag vectors corresponding to each local feature, and then weight them to obtain the corresponding local text features. Randomly generated local text centers serve as learnable local text parameters during the training of the cross-modal attention module, determining the number of local text features, defined as being the same as the number of local image features. ; This is the weighted set of local text features; the cross-modal attention module first randomly initializes a text center for each local image, then calculates attention weights on all text tag vectors based on the text center, and finally extracts the local text features corresponding to each locality, as shown below:

[0011] in, For local text center The scaled similarity score vector of features relative to the sorted text tag vector positions; The attention weight vector is the feature of the sorted text tag vector position relative to the current local text center. This represents the set of local features of the text; the superscript T is the transpose symbol.

[0012] Optionally, an attribute alignment network is constructed for coarse-grained cross-modal attribute alignment, including: constructing a text attribute label extraction layer to extract attribute matching labels, including: using a natural language processing library to split the input text into sentences, obtaining word segmentation, part-of-speech tagging, dependency relation labels, and lexical vectors; selecting nouns and adjectives to obtain candidate attribute words; constructing a lexical-attribute dictionary; if the extracted candidate attribute words in the text description exist in the dictionary, then labeling the corresponding attribute to obtain attribute matching labels; constructing an image attribute feature extraction layer to extract image attribute features, including: using learnable attribute location parameters and image patch label vectors, calculating the attention position of each attribute through a cross-modal attention module; and then using a fully connected layer as a task-specific head to obtain the corresponding attribute features, as shown below:

[0013] in, Randomly generated attribute centers serve as learnable attribute location parameters during the training of the cross-modal attention module, used to learn the image patches that each attribute focuses on. , The number of attributes; To learn the set of all attribute features of the corresponding image patch features, To learn the s-th attribute feature of the corresponding image patch features; It is a linear transformation function; It is a set of image attribute features.

[0014] Optionally, cross-modal local fine-grained alignment can be performed by mapping local image features and local text features to each other, including: mapping local image feature sets and local text feature sets to each other by calculating a loss function to perform fine-grained alignment.

[0015] Optionally, the loss function for constructing the text-image cross-modal person re-identification model includes: jointly optimizing the loss function of the text-image cross-modal person re-identification model using similarity distribution matching loss, identity classification loss, and attribute recognition loss, as shown below:

[0016] in, The loss function for a text-image based cross-modal person re-identification model; similarity distribution matching loss. The KL divergence between image features and text features represents the identity classification loss. This represents the attribute recognition loss, used to measure the difference in identity prediction using image features and text features. This indicates the difference in attribute prediction results.

[0017] Optionally, the similarity distribution matching loss for the text-image based cross-modal person re-identification model is constructed, including using gradient descent to represent image-text pairs and using cosine similarity to calculate the cosine similarity between each image feature and each text feature, as shown below:

[0018] Among them, the gradient descent method uses Represents a set of image features. B represents the set of text features; B represents the number of image-text pairs. Cosine similarity; For the i-th image feature, including global image features and image local feature set ; For the j-th text feature, including global text features and image local feature set For each image-text pair cosine distance, calculate the similarity distribution with all image-text pairs cosine distances, as shown below:

[0019] in, Let be the probability that the i-th image and the j-th text match in a text-image based cross-modal person re-identification model; It is a temperature hyperparameter; This is the k-th text feature; Let be the probability distribution of the matching tags from the i-th image to the j-th text. For the matching label of the k-th text that belongs to the same identity as the i-th image; for each text-image pair cosine distance, calculate the similarity distribution with all text-image pairs cosine distances, as shown below:

[0020] in, Let be the probability that the j-th text and the i-th image are matched by the text-image based cross-modal person re-identification model; For the k-th image feature; Let be the probability distribution of the matching labels from the j-th text to the i-th image; The matching label for the k-th image that shares the same identity as the j-th text; similarity distribution matching loss. , means as follows:

[0021] in, The loss is for matching similar distributions; The loss is the similarity distribution matching loss from image to text direction; The loss is the similarity distribution matching loss from text to image direction; It is a constant; the identity classification loss of the text-image based cross-modal person re-identification model includes obtaining the probability prediction value of each identity after passing the global image features and global text features through the identity classifier, as shown below:

[0022] in, Here is the weight matrix for the identity classifier. ID represents the number of identities; For identity classifier; and Let be the predicted identity probabilities of the i-th image and the j-th text, respectively, and let be the features of the i-th image at this time. Includes only global image features The j-th text feature Includes only global text features Identity classification loss , means as follows:

[0023] in, The identity classification loss is used for the i-th image; The identity classification loss is used for the j-th text. Let i be the predicted probability of the i-th image corresponding to the id-th identity; Let be the predicted probability of the j-th text corresponding to the id-th identity; The matching label for the i-th image corresponding to the id-th identity; Let be the matching tag for the j-th text corresponding to the id-th identity.

[0024] The attribute recognition loss of the text-image-based cross-modal person re-identification model is constructed as follows: Image attribute features are mapped to an attribute space; when the text description and the pedestrian image have the same identity, the attributes are also the same; attribute labels are obtained using the text descriptions of pedestrians in the dataset to supervise the attribute recognition of the image; and attribute information generated during the text description process is learned, represented as follows:

[0025] in, For attribute classifiers, ; Let be the probability set of the i-th image recognition attribute; The attribute features of the i-th image; attribute recognition loss. , means as follows:

[0026] in, The label for the s-th attribute obtained through a text description that shares the same identity as the i-th image; Let be the attribute recognition loss for the s-th attribute of the i-th image; where s is the attribute index. Let be the predicted probability of the s-th attribute of the i-th image.

[0027] A text-image-based cross-modal person re-identification system is provided to perform the text-image-based cross-modal person re-identification method described above. The system includes: a data processing module for preparing a text-image person re-identification dataset and constructing attribute labels using text descriptions; a model building module for incorporating a local alignment network into a pre-trained language-image model, learning the weights of the text tag vectors corresponding to each local feature, weighting the corresponding text local features, and mapping image local features to text local features for cross-modal fine-grained local alignment; and an attribute alignment network for calculating the attention position of each attribute based on the image patch tag vectors, using the attribute labels obtained from the attributes in the text as supervision for cross-modal coarse-grained attribute alignment, resulting in a text-image-based cross-modal person re-identification model; and a loss function construction module. The model building module constructs a loss function suitable for the text-image-based cross-modal person re-identification model, including similarity distribution matching loss, identity classification loss, and attribute recognition loss. The model training module uses the prepared text-image person re-identification dataset and iteratively updates the parameters of the text-image-based cross-modal person re-identification model based on minimizing the loss function to obtain the trained model. The person re-identification module uses the trained model to perform person re-identification, sorting the obtained global image features and global text features by cosine similarity to obtain the person re-identification retrieval results. The data processing module, model building module, loss function construction module, model training module, and person re-identification module are connected sequentially.

[0028] A computer device includes a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement the steps of the text-image-based cross-modal person re-identification method described above.

[0029] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the text-image-based cross-modal person re-identification model as described above.

[0030] This application provides a text-image-based cross-modal person re-identification method and system, computer device, and computer-readable storage medium. It proposes a cross-modal local fine-grained alignment method, which reduces the heterogeneous variance of similar cross-modal samples by deconstructing the image into human body parts and mapping them to text words. This compensates for the information loss in fine-grained discrimination caused by global alignment and solves the problem of discriminative semantics being overwhelmed by irrelevant information due to occlusion, viewpoint changes, and pose shifts. Furthermore, it proposes a cross-modal attribute coarse-grained alignment method, using explicit attribute words in the text description as weak supervision signals. This method compares the consistency of attribute features with those predicted by the image branch, constructing an attribute-visual feature mapping to improve cross-modal semantic consistency and alleviate semantic confusion between visual and linguistic cross-modal approaches.

[0031] To make the above-mentioned features and advantages of the application more apparent and understandable, specific embodiments are provided below, and detailed descriptions are given in conjunction with the accompanying drawings. Attached Figure Description

[0032] Figure 1 is a flowchart of the cross-modal pedestrian re-identification method based on text-image provided in this application.

[0033] Figure 2 is a structural diagram of the cross-modal pedestrian re-identification model based on text-image proposed in this application.

[0034] Figure 3 is a structural diagram of the local alignment network proposed in this application.

[0035] Figure 4 is a structural diagram of the attribute alignment network proposed in this application.

[0036] Figure 5(a) is a schematic diagram of the pedestrian re-identification results obtained using the language-image pre-trained model.

[0037] Figure 5(b) is a schematic diagram of the pedestrian re-identification result obtained by using the text-image based cross-modal pedestrian re-identification model proposed in this application. Detailed Implementation

[0038] To make the objectives and technical solutions of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the described embodiments of this application without creative effort are within the scope of protection of this application.

[0039] In the first embodiment of this application, please refer to Figure 1. Figure 1 is a flowchart of the cross-modal pedestrian re-identification method based on text-image provided in this application, which includes the following steps: S1~S6.

[0040] Step S1: Prepare a text-image person re-identification dataset and construct attribute labels using text descriptions; Step S2: Based on a language-image pre-trained model (Contrastive Language-Image...) Pre-training (CLIP) incorporates a local alignment network to learn the weights of text token vectors corresponding to each local feature, weighting them to obtain the corresponding text local features. Image local features and text local features are mapped to each other for cross-modal fine-grained local alignment. An attribute alignment network is added to calculate the attention position of each attribute based on the image patch token vector. Attribute labels obtained from attributes in the text are used as supervision for cross-modal coarse-grained attribute alignment, resulting in a text-image-based cross-modal person re-identification model. Step S3: Construct a loss function suitable for the text-image-based cross-modal person re-identification model. Based on minimizing the loss function, iteratively update the parameters of the text-image-based cross-modal person re-identification model to obtain the trained model. Step S4: Use the trained text-image-based cross-modal person re-identification model for person re-identification. Rank the obtained global image features and global text features by cosine similarity to obtain the person re-identification retrieval results.

[0041] This application provides a text-image-based cross-modal person re-identification method. It proposes a cross-modal local fine-grained alignment method, which reduces the heterogeneous variance of similar cross-modal samples by deconstructing the image into human body parts and mapping them to text words. This compensates for the information loss in fine-grained discrimination caused by global alignment and solves the problem of discriminative semantics being overwhelmed by irrelevant information due to occlusion, viewpoint changes, and pose shifts. Furthermore, it proposes a cross-modal attribute coarse-grained alignment method, using explicit attribute words in the text description as weak supervision signals. This method compares the consistency of attribute features with those predicted by the image branch, constructing an attribute-visual feature mapping to improve cross-modal semantic consistency and alleviate semantic confusion between visual and linguistic cross-modal approaches.

[0042] In step S1, please refer to step S1 in Figure 1 to prepare the text-image pedestrian re-identification dataset.

[0043] As an example, the text-image person re-identification dataset used includes the publicly available text-image person re-identification dataset RSTPReid. This dataset contains pedestrian images and corresponding text descriptions.

[0044] Specifically, the text description organically integrates various attributes of a pedestrian, such as gender, age, height, clothing, hairstyle, accessories, actions, and posture, to construct the pedestrian's overall image. Since the vast majority of this information comes from pedestrian attributes, the corresponding attribute labels can be directly obtained by extracting attributes from the text.

[0045] In step S2, please refer to step S2 in Figure 1. A local alignment network is added based on the language-image pre-trained model to learn the weights of the text tag vectors corresponding to each local feature. The corresponding text local features are obtained by weighting and mapping the image local features and text local features to each other for cross-modal local fine-grained alignment. An attribute alignment network is added to calculate the attention position of each attribute based on the image patch tag vector. The attribute labels obtained from the attributes in the text are used as supervision for cross-modal attribute coarse-grained alignment, resulting in a text-image based cross-modal person re-identification model.

[0046] As an example, please refer to Figure 2, which is a structural diagram of the text-image-based cross-modal person re-identification model proposed in this application. The text-image-based cross-modal person re-identification model includes: an image branch, a text branch, a local alignment network, and an attribute alignment network, which are used for image feature extraction, text feature extraction, local fine-grained alignment, and attribute coarse-grained alignment, respectively.

[0047] As an example, a cross-modal pedestrian re-identification model based on text and image is constructed by adding a local alignment network and an attribute alignment network to the language-image pre-trained model, including steps S21 to S24.

[0048] Step S21: Construct image branch output image features. The image branch includes: image segmentation layer, image patch linear mapping layer, location embedding layer and self-attention-based deep learning module (Transformer) encoding layer, which are used for image segmentation, image patch linear mapping, location embedding and self-attention-based encoding output image features, respectively.

[0049] As an example, image segmentation includes: segmenting the input pedestrian image data into several fixed-size image patches, transforming a continuous two-dimensional pixel array into a discrete sequence of image patches, laying the foundation for subsequent image patch modeling, as shown below:

[0050] in, The input is pedestrian image data. These are the height and width of the image, respectively; The size of the image after segmentation; The k-th image block after segmentation, the image block number. , This is the index of the image patch in the height direction. This is the index of the image patch in the width direction. .

[0051] As an example, image patch linear mapping includes: flattening each image patch and performing an image patch linear mapping operation to obtain an image patch label vector; projecting the image patch label vector onto an embedding space with the same dimension as the self-attention-based deep learning module; and converting the visual signal into an image patch label vector that can participate in attention calculation, as shown below:

[0052] in, For convolutional layers with a kernel size of P×P; Let k be the label vector of the k-th image patch after linear mapping. , For self-attention-based deep learning module dimensions; Label all linearly mapped image patches with vector sequences. , This represents the total number of image blocks after segmentation. .

[0053] As an example, location embedding includes adding corresponding location codes to the image patch label vectors to compensate for the spatial location information lost due to flattening and projection, ensuring that the text-image-based cross-modal person re-identification model can still perceive the relative coordinates of each image patch in the original image, as shown below:

[0054] in, It is a learnable classification label vector (CLS token). ; A sequence of image patch label vectors after adding learnable classification labels; For the first Each location is encoded; For a set of location codes, ; Label the image patch with a vector sequence after location embedding. .

[0055] As an example, the self-attention-based encoding output includes: passing a sequence of image patch label vectors carrying location priors layer by layer through a self-attention-based deep learning module encoding layer composed of stacked multi-head self-attention and feedforward networks, continuously completing cross-image encoding between each self-attention-based deep learning module encoding layer. piece The contextual interaction and feature fusion are used to output image features, as shown below:

[0056] in, The output features of the encoding layer for the self-attention-based deep learning module of the image branch are used. ; These represent the encoding layers of each self-attention-based deep learning module; These represent the intermediate results output by the encoding layer of each self-attention-based deep learning module.

[0057] Specifically, taking the first layer of the self-attention-based deep learning module encoding layer as an example, each self-attention-based deep learning module encoding layer, for the received... indivual The label vector, which contains the learnable classification label vector and all image patch label vectors, is normalized using LayerNorm to obtain a distributionally stable feature representation, as shown below:

[0058] in, This represents the normalized image features.

[0059] Furthermore, the normalized image features are simultaneously fed into 12 parallel attention heads. Each parallel attention head generates a dimension-1 matrix through an independent query, key, and value linear transformation matrix. The query vector, key vector, and value vector of the h-th parallel attention head, and the image feature query vector, image feature key vector, and image feature value vector are represented as follows:

[0060] in, These are the weight matrices for generating image feature queries, key-value linear transformations, and other functions, respectively. These are the image feature query vector, image feature key vector, and image feature value vector of the h-th parallel attention head, respectively. This indicates the number of parallel attention heads.

[0061] Furthermore, the image feature query vector of the h-th parallel attention head is calculated. Image feature key vector of the h-th parallel attention head The attention score is obtained by scaling the dot product between the transposes of the vectors, and then normalized by the Softmax exponential function. This score is then compared with the corresponding image feature vector of the attention head. The weighted features are obtained by multiplication, and then the outputs of all parallel attention heads are concatenated by channel and restored to the C dimension through an output projection matrix to form the multi-head self-attention result, as shown below:

[0062] in, The image attention score for the h-th parallel attention head; The image attention score after normalization of the h-th parallel attention head; Weighted features of the image for the h-th parallel attention head; Concatenate the image weighted features of all parallel attention heads into a sequence; To output the projection matrix; This represents the result of multi-head self-attention for the image; the superscript T is the transpose symbol.

[0063] Furthermore, residual connections are used to add the multi-head self-attention result of the image to the original image patch label vector sequence that has not undergone attention, in order to obtain the intermediate features of the image. To prevent gradient vanishing and preserve low-level information, it is represented as follows:

[0064] Furthermore, the intermediate features of the image were analyzed again. LayerNorm is applied to normalize the intermediate feature distribution before it is input into the feedforward network. The feedforward network first uses a weight matrix to expand the normalized image's intermediate features from C-dimensional to 4C-dimensional, then introduces nonlinearity through the GELU activation function, and finally uses another weight matrix to reduce the features back to C-dimensional, completing the expansion-contraction transformation and obtaining the output of the feedforward network. , means as follows:

[0065] in, To standardize intermediate features of the image; These are the up-dimensional weight matrix and down-dimensional weight matrix of the feedforward network, respectively.

[0066] Furthermore, the output of the feedforward network... Image intermediate features in residual branches The features are added together again to form the image features output by the encoding layer of this self-attention-based deep learning module. This allows the updated label vector to incorporate both global context information and maintain a stable gradient path, as shown below:

[0067] Furthermore, the output of the self-attention-based deep learning module encoding layer is directly fed into the next self-attention-based deep learning module encoding layer to continue repeating the process of formulas (5) to (10) above, thereby constructing increasingly abstract and semantically rich image features layer by layer.

[0068] Finally, an additional linear projection layer is used to force the number of channels of the image features to align with the dimensions of the text branch, resulting in the output features of the image branch. To ensure that the image and text features maintain consistency on a numerical scale, as shown below:

[0069] in, The weight matrix of the feature map. D represents the dimension of the text branch.

[0070] Step S22: Construct a text branch to output text features. The text branch includes a text mapping layer, a word embedding layer, a position embedding layer, and a deep learning module encoding layer based on self-attention, which are used for text mapping, word embedding, position embedding, and encoding based on self-attention to output text features, respectively.

[0071] As an example, text mapping includes: segmenting the input text description into several character-level merge (BPE) sub-word sequences to obtain a variable-length original sub-word sequence, and inserting [SOT] and [EOT] special markers at the beginning and end of the original sub-word sequence to obtain a sub-word sequence with the special markers inserted. This transforms any natural language string into a discrete, modelable sub-word sequence, as shown below:

[0072] in, This represents the index of the vocabulary corresponding to the l-th word in the sentence describing the text, which is used by the language-image pre-trained model to call the corresponding text tag vector by index; Indicates the length of the actual sentence; The word indicating the beginning of a sentence corresponds to its index in the vocabulary. The word indicating the end of a sentence corresponds to the index of the vocabulary, and the corresponding vocabulary indices are the last two digits, i.e., the two largest numbers; The original sub-word sequence; The original text description is the data after encoding and insertion of start and end marker vectors, i.e., a discrete sequence of subwords.

[0073] Furthermore, sentences of varying lengths in all text descriptions are truncated or padded to a uniform length, ensuring that the sub-word sequences within a batch are of equal length for parallel processing, as shown below:

[0074] in, The sequence of subwords with uniform length. ; This represents the unified length. If the original sentence is too long, it will be truncated; if it is too short, it will be padded with zeros. In practical applications, the size of the unified length L can be set, so L will always be greater than L0+2. This represents the length of the original sentence after shortening.

[0075] As an example, word embedding includes: creating a learnable parametric word embedding matrix. Each subword is mapped to a vector space, and the semantic symbols are converted into text tag vectors that can participate in attention calculation, as shown below:

[0076] in, Let be the l-th text tag vector after word embedding. ; This is a text tag vector representing the start of a sentence after word embedding; This is a marker vector indicating the end of a sentence after word embedding; is the last token vector after word embedding and unification of length; T is the sequence of text token vectors after all word embeddings. ; This refers to the size of the vocabulary.

[0077] In one specific embodiment of this application, NL=49408.

[0078] As an example, positional embedding includes: superimposing learnable positional encodings onto text tag vectors to ensure that the order of words in a sentence is perceived based on a language-image pre-trained model, as shown below:

[0079] in, Encode the Lth position; A set of location codes; This is a sequence of text tag vectors embedded at the location.

[0080] Furthermore, a two-dimensional attention-efficient mask M is generated to mask the future text tag vector portion and padding portion. Its mapping can be formally expressed as:

[0081] Where a is the current position of the text tag vector used to calculate attention; b is the future position of the text tag vector used to calculate attention. This is the effective mask in the a-th row and b-th column of the two-dimensional attention mask effective matrix.

[0082] As an example, the self-attention-based encoding output includes: passing a sequence of text tag vectors carrying positional priors and effective masks layer by layer through a self-attention-based deep learning module encoding layer composed of multi-head self-attention and feedforward networks; capturing global contextual dependencies and semantic fusion between each self-attention-based deep learning module encoding layer; and thus outputting refined text features for downstream tasks to use, as shown below:

[0083] in, The output features of the encoding layer of the self-attention-based deep learning module for text branches. ; These represent the intermediate results output by the encoding layer of each self-attention-based deep learning module.

[0084] Specifically, taking the first layer of the self-attention-based deep learning module encoding layer as an example, each self-attention-based deep learning module encoding layer, for the received... indivual The tag vector containing the sentence's beginning and end markers and all text marker vectors is normalized using LayerNorm to obtain a distributionally stable feature representation, as shown below:

[0085] in, This represents the normalized text features.

[0086] Furthermore, the normalized text features are simultaneously fed into 12 parallel attention heads. Each parallel attention head generates a dimension-1 matrix through an independent query, key, and value linear transformation matrix. The query vector, key vector, and value vector of the h-th parallel attention head are represented as follows:

[0087] in, These are the weight matrices for generating text feature queries, key-value linear transformations, and other functions, respectively. These are the text feature query vector, text feature key vector, and text feature value vector of the h-th parallel attention head, respectively.

[0088] Furthermore, the text feature query vector of the h-th parallel attention head is calculated. Text feature key vector with the h-th parallel attention head Scaling dot product between transposes plus effective mask The attention score is obtained, and after normalization by the Softmax function, it is compared with the text feature vector of the corresponding attention head. The weighted features are obtained by multiplying, and then the outputs of all attention heads are concatenated by channel and restored to D dimensions through an output projection matrix to form the multi-head self-attention result, as shown below:

[0089] in, The text attention score for the h-th parallel attention head; The text attention score after normalization of the h-th parallel attention head; The text weighted features for the h-th parallel attention head; Concatenate the text weighted features of all parallel attention heads into a sequence; The result is for multi-head self-attention of the text.

[0090] Furthermore, a residual connection is used to add the multi-head self-attention result of the text to the original text tag vector sequence that has not undergone attention, in order to obtain the intermediate text features. To prevent gradient vanishing and preserve low-level information, it is represented as follows:

[0091] Furthermore, we will again examine the intermediate features of the text. The execution layer normalization layer (LayerNorm) re-standardizes the intermediate feature distribution before inputting it into the feedforward network. The feedforward network first uses a weight matrix to expand the standardized text intermediate features from D-dimensional to 4D-dimensional, then introduces non-linearity through the GELU activation function, and subsequently uses another weight matrix to reduce the features back to D-dimensional, completing the expansion-reduction transformation and obtaining the output of the feedforward network. , means as follows:

[0092] in, Standardize text intermediate features; These are the up-dimensional weight matrix and down-dimensional weight matrix of the feedforward network, respectively.

[0093] Furthermore, the output of the feedforward network... Intermediate text features in residual branches These features are added together again to form the text features output by the encoding layer of this self-attention-based deep learning module. This allows the updated label vector to incorporate both global context information and maintain a stable gradient path, as shown below:

[0094] Furthermore, the output of the encoding layer of the self-attention-based deep learning module is directly fed into the next encoding layer of the self-attention-based deep learning module to repeat the process of formulas (18) to (23) above, thereby constructing the output features of the text branch layer by layer. .

[0095] Step S23: Construct a local alignment network for cross-modal local fine-grained alignment. The local alignment network includes an image local feature extraction layer and a text local feature selection layer, which are used for local feature extraction of the image and local feature extraction of the text, respectively.

[0096] Specifically, please refer to Figure 3, which is a structural diagram of the local alignment network proposed in this application. Constructing the local alignment network for cross-modal local fine-grained alignment includes: steps S231 to S232.

[0097] Step S231: Construct an image local feature extraction layer. The output features obtained from the image branch, including the learnable classification label vector and all image patch label vectors, are split. The average pooling of the features from all image patch label vectors is then performed to calculate the mean, yielding the image local features, as shown below:

[0098] in, The feature with index 0 in the output features of the image branch is the feature corresponding to the learnable classification label vector. The output features of the image branch are indexed from 1 to the end, that is, the features corresponding to all image patch label vectors; The global features of the image are extracted for image features, that is, the features corresponding to the learnable classification label vectors obtained by splitting the image. This is the set of image patch features obtained from the splitting process; This is the subset of image blocks corresponding to the nth local feature after average segmentation; This is the subset of image blocks corresponding to the local features of the Nth image after average segmentation; This represents the nth local feature of the image. This is the Nth local feature of the image; It is a set of local features of the image; is the average splitting function; N is the number of splits, i.e., the number of local image features generated. For average pooling; Step S232: Construct a text local feature extraction layer, learn the weight of the text label vector corresponding to each local feature through learnable local text parameters, obtain the corresponding text local features by weighting, and match the image local features with the text local features to achieve cross-modal local fine-grained alignment.

[0099] As an example, a text local feature extraction layer is constructed. The weights of the text marker vectors corresponding to each local feature are learned using learnable local text parameters, and the corresponding local text features are obtained by weighting. This includes: encoding the discrete word sequence input to the text branch; sorting it according to a descending rule and then splitting it to obtain the positions of the inserted start and end marker vectors and all text marker vectors; based on the text marker vectors in the output features of the text branch, the weights of the text marker vectors corresponding to each local feature are learned using learnable local text parameters, and the corresponding local text features are obtained by weighting, as shown below:

[0100] in, The original text description is the data after encoding and insertion of start and end marker vectors; A sorting function for encoding positions from largest to smallest; This is the set of position indices for the sorted text tag vectors; This is the position index of the Lth text tag vector after sorting; The feature corresponding to the first position index in the output features of the text branch, that is, the feature corresponding to the insertion end marker vector; The feature corresponding to the second position index in the output features of the text branch; The feature set indexed from the 3rd position to the end of the output features of the text branch; The feature at the position of the inserted end marker vector is considered as a global text feature extracted from the text features and is used for subsequent feature retrieval; Features for inserting the start marker vector position; The feature set of the sorted text marker vector positions obtained after splitting; This is a cross-modal attention module used to learn the weights of the text tag vectors corresponding to each local feature, and then weight them to obtain the corresponding local text features. The randomly generated local text centers serve as learnable local text parameters during the training of the cross-modal attention module, determining the number of local text features, and are defined as having the same number of local features as the image. ; It is a set of local features of the text.

[0101] Among them, the cross-modal attention module First, randomly initialize a text center for each local area of ​​the image. Then, based on the text center, attention weights are calculated on all text tag vectors, and the word vectors are weighted and summed to finally extract the local text features corresponding to each part, as shown below:

[0102] in, For local text center The scaled similarity score vector of features relative to the sorted text tag vector positions; The attention weight vector is the feature of the sorted text tag vector position relative to the current local text center. This refers to the set of local text features that match the local text center set, i.e., the set of local text features; the superscript T is the transpose symbol. Unlike Figure 2, in Figure 3... That is, a transpose was performed first.

[0103] Furthermore, the local features of the image and the local features of the text are mapped to each other to perform cross-modal local fine-grained alignment, including: by subsequently calculating the loss function, mapping the set of local features of the image and the set of local features of the text to each other to perform cross-modal local fine-grained alignment.

[0104] Step S24: Construct an attribute alignment network for cross-modal attribute coarse-grained alignment. The attribute alignment network includes an image attribute feature extraction layer and a text attribute label extraction layer, which are used for extracting image attribute features and extracting attribute matching labels, respectively.

[0105] Specifically, please refer to Figure 4, which is a structural diagram of the attribute alignment network proposed in this application. Constructing the attribute alignment network for cross-modal attribute coarse-grained alignment includes: steps S241 to S242.

[0106] Step S241: Construct a text attribute label extraction layer to extract attribute matching labels. This includes: using the SpaCy model from the natural language processing library to segment the input text into sentences, obtaining word segmentation, part-of-speech tagging (POS), dependency relation labels (dep), and lexical vectors; then, obtaining candidate attribute words by selecting nouns (clothing, accessories, hairstyle) and adjectives (color, style). Since the same attribute can be expressed in different ways, for example, the gender "female" can be expressed as "she," "women," "woman," etc., a "lexical-attribute" dictionary is constructed. If a candidate attribute word in the extracted text description exists in the dictionary, the corresponding attribute label is marked to obtain the attribute matching label. For example, the text description "The woman's hair is dark brown. She wears a light gray coat. She has a pair of dark graytights. She wears a pair of white sneakers. She has a green..." In the phrase "backpack," the word "Bag" corresponds to the attribute "backpack," the word "Shoes" corresponds to the attribute "sneakers," the word "Gender" corresponds to the attribute "woman," the word "L.up" corresponds to the attribute "coat," the word "C.up" corresponds to the attribute "gray," the word "L.low" corresponds to the attribute "tights," and the word "C.low" corresponds to the attribute "grey." Since the text description and the pedestrian image represent the same pedestrian, the attribute labels for the pedestrian's text description and the pedestrian image should also be the same. Therefore, the text attribute labels obtained from the pedestrian's text description in the dataset can be used to supervise image feature extraction.

[0107] Step S242: Construct an image attribute feature extraction layer to extract image attribute features, including: Since the various attributes of pedestrians in the image are not uniformly distributed across the entire image, but rather fixed on the body parts of the pedestrians in the image, the attention position of each attribute is calculated through a cross-modal attention module using learnable attribute location parameters and image patch label vectors. Then, a fully connected layer is used as a task-specific head to obtain the corresponding attribute features. Cross-modal attribute coarse-grained alignment is represented as follows:

[0108] in, Randomly generated attribute centers serve as learnable attribute location parameters during the training of the cross-modal attention module, used to learn the image patches that each attribute focuses on. , For the number of attributes, cross-modal attention module The calculation process is the same as that of formula (26), and will not be repeated here; To learn the set of all attribute features of the corresponding image patch features, To learn the s-th attribute feature of the corresponding image patch features; It is a linear transformation function, i.e., a fully connected layer; It is a set of image attribute features. Let s be the s-th attribute feature of the image. .

[0109] As an example, the two alignment methods in step S2 are additional tasks performed during the training of the text-image cross-modal person re-identification model. Local alignment plus attribute alignment enables the text-image cross-modal person re-identification model to distinguish details, but is not used for the final person re-identification.

[0110] Furthermore, an attribute alignment network will be constructed to perform coarse-grained cross-modal attribute alignment, including: performing coarse-grained cross-modal attribute alignment by subsequently calculating the loss function.

[0111] In step S3, please refer to step S3 in Figure 1 to construct a loss function suitable for the text-image cross-modal person re-identification model; based on minimizing the loss function, iteratively update the parameters of the text-image cross-modal person re-identification model to obtain the trained text-image cross-modal person re-identification model.

[0112] As an example, the loss function for constructing a text-image cross-modal person re-identification model includes: jointly optimizing similarity distribution matching loss, identity classification loss, and attribute recognition loss. The loss function for constructing the text-image cross-modal person re-identification model is expressed as follows:

[0113] in, The loss function for a text-image based cross-modal person re-identification model; similarity distribution matching loss. The KL divergence between image features and text features represents the identity classification loss. This represents the attribute recognition loss, which measures the difference between predicting identities using image features and text features. This indicates the difference in attribute prediction results.

[0114] As an example, the similarity distribution matching loss for constructing a text-image-based cross-modal person re-identification model is discussed. This includes using gradient descent (mini-batch) to represent image-text pairs, and using cosine similarity to calculate the cosine similarity between each image feature and each text feature, as shown below:

[0115] Among them, the gradient descent method uses Represents a set of image features. B represents the set of text features; B represents the number of image-text pairs. Cosine similarity; For the i-th image feature, including global image features and image local feature set ; For the j-th text feature, including global text features and image local feature set .

[0116] Furthermore, for each image-text pair cosine distance, the similarity distribution with the cosine distances of all image-text pairs is calculated, as shown below:

[0117] in, Let be the probability that the i-th image and the j-th text match in a text-image based cross-modal person re-identification model; It is a temperature hyperparameter; This is the k-th text feature; Let be the probability distribution of the matching tags from the i-th image to the j-th text. To set a matching label for the k-th text that shares the same identity as the i-th image, when the i-th image and the j-th text share the same identity, set a matching label. The rest are set to 0.

[0118] Furthermore, for each text-image pair cosine distance, the similarity distribution with all text-image pairs cosine distances is calculated, as shown below:

[0119] in, Let be the probability that the j-th text and the i-th image are matched by the text-image based cross-modal person re-identification model; For the k-th image feature; Let be the probability distribution of the matching labels from the j-th text to the i-th image; The matching label for the k-th image that shares the same identity as the j-th text.

[0120] Further, referring to Figure 3, based on the similarity distribution of the cosine distance between each image-text pair and the cosine distances of all image-text pairs, and the similarity distribution of the cosine distance between each text-image pair and the cosine distances of all text-image pairs, the similarity distribution matching loss is obtained. , means as follows:

[0121] in, The loss is the similarity distribution matching loss from image to text direction; The loss is the similarity distribution matching loss from text to image direction; It is a very small constant to prevent division by zero.

[0122] As an example, we construct the identity classification loss for a text-image-based cross-modal person re-identification model. This includes, including, global image features and global text features After passing through the identity classifier, the predicted probability value for each identity is obtained, as shown below:

[0123] in, Here is the weight matrix for the identity classifier. ID represents the number of identities; This is an identity classifier used to map image features and text features to probability distributions of identity categories. Since the identity of each input pedestrian image in the training set is known, identity classification is performed based on the number of identities of pedestrian images in the training set. and Let be the predicted identity probabilities for the i-th image and the j-th text, respectively. Each image is assigned a probability for its identity. During training, the probability of identities corresponding to matching tags is increased, while the probabilities of other non-matching identities are decreased. At this point, the features of the i-th image... Includes only global image features The j-th text feature Includes only global text features .

[0124] Furthermore, based on the probability predictions of all obtained identities, cross-entropy is used to calculate the identity classification loss. , means as follows:

[0125] in, The identity classification loss is used for the i-th image; The identity classification loss is used for the j-th text. Let i be the predicted probability of the i-th image corresponding to the id-th identity; Let be the predicted probability of the j-th text corresponding to the id-th identity; The matching label for the i-th image corresponding to the id-th identity; Let be the matching tag for the j-th text corresponding to the id-th identity.

[0126] Please refer to Figure 4. The image's attribute features are mapped to the attribute space. When the text description and the pedestrian image have the same identity, the attributes should also be the same. Therefore, attribute labels obtained from the pedestrian text descriptions in the dataset can be used to supervise the attribute recognition of images, thereby learning the attribute information generated during the text description process, as shown below:

[0127] in, For attribute classifiers, ; Let be the probability set of the i-th image recognition attribute; The attribute features of the i-th image are the set of image attribute features extracted by formula (27). Furthermore, based on the probability set of each image recognition attribute, the attribute recognition loss is calculated using binary cross-entropy. , means as follows:

[0128] in, The label for the s-th attribute obtained through a text description that shares the same identity as the i-th image; Let be the attribute recognition loss for the s-th attribute of the i-th image; where s is the attribute index. Let be the predicted probability of the s-th attribute of the i-th image.

[0129] As an example, after obtaining the loss function applicable to the text-image-based cross-modal person re-identification model, the parameters of the text-image-based cross-modal person re-identification model are iteratively updated based on minimizing the loss function to obtain the trained text-image-based cross-modal person re-identification model. This includes: training the text-image-based cross-modal person re-identification model established in step S2 using the training set in the text-image person re-identification dataset prepared in step S1; during the training iteration process, the loss function of the text-image-based cross-modal person re-identification model is used as the basis for training. Based on minimization, the parameters of the text-image-based cross-modal person re-identification model are continuously updated iteratively to obtain the trained text-image-based cross-modal person re-identification model.

[0130] Specifically, when the loss function is minimized, the local feature set of the image and the local feature set of the text correspond to each other, achieving cross-modal fine-grained local alignment and cross-modal attribute coarse-grained alignment.

[0131] In step S4, please refer to step S4 in Figure 1. The trained text-image based cross-modal person re-identification model is used to perform person re-identification. The obtained global image features and global text features are sorted by cosine similarity to obtain the person re-identification retrieval results.

[0132] As an example, the test set in the text-image person re-identification dataset prepared in step S1 is input into the cross-modal person re-identification model based on text-image trained in step S3 to obtain the corresponding global image feature set and global text feature set. The cosine similarity of each global image feature and each global text feature is calculated and sorted to obtain the person re-identification retrieval result.

[0133] In a specific embodiment of this application, please refer to Figure 5. Figure 5(a) is a schematic diagram of the person re-identification result obtained using the language-image pre-trained model, and Figure 5(b) is a schematic diagram of the person re-identification result obtained using the text-image based cross-modal person re-identification model proposed in this application. To intuitively see the performance of the text-image based cross-modal person re-identification model, two typical images from the dataset are used. Input text description 1: "The man with the bright blue hood is wearing a black overcoat and a piece of blue clothing inside. His pants are lacking while shoes are brown. And he is wearing a pair of dark gloves." and text description 2: "The woman is wearing a white coat with the hood. She wears black pants and gray boots. And she is wearing a white woolly hat." The detection results of the language-image pre-trained model as the base model and the text-image based cross-modal person re-identification model proposed in this application are shown in Figure 5. "√" indicates that the image matches the text description, and "×" indicates that the image does not match the text description. The text-image-based cross-modal person re-identification model proposed in this application can match the corresponding pedestrian images using the input text description as the retrieval condition. Compared with the basic model, it has a significant performance improvement, which proves the effectiveness of the cross-modal local fine-grained alignment method and the cross-modal attribute coarse-grained alignment method proposed in this application. It achieves cross-modal person re-identification results with higher accuracy and stronger robustness.

[0134] This application also provides a text-image-based cross-modal person re-identification system for performing the aforementioned text-image-based cross-modal person re-identification method. The system includes: a data processing module for preparing a text-image person re-identification dataset and constructing attribute labels using text descriptions; a model building module for incorporating a local alignment network into a language-image pre-trained model, learning the weights of the text tag vectors corresponding to each local feature, obtaining the corresponding local text features through weighted summation, and performing cross-modal fine-grained local alignment by mapping the image local features to the weighted local text features; and an attribute alignment network for calculating the attention position of each attribute based on the image patch tag vectors, using the attribute labels obtained from the attributes in the text as supervision for cross-modal coarse-grained attribute alignment. A text-image-based cross-modal person re-identification model is obtained; Loss function construction module: constructs a loss function suitable for the text-image-based cross-modal person re-identification model, including similarity distribution matching loss, identity classification loss, and attribute recognition loss; Model training module: using the prepared text-image person re-identification dataset, iteratively updates the parameters of the text-image-based cross-modal person re-identification model based on minimizing the loss function, obtaining the trained text-image-based cross-modal person re-identification model; Person re-identification module: performs person re-identification using the trained text-image-based cross-modal person re-identification model, sorts the obtained global image features and global text features by cosine similarity, and obtains the person re-identification retrieval results.

[0135] As an example, the data processing module, model building module, loss function building module, model training module, and pedestrian re-identification module are connected in sequence.

[0136] This application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the above-described text-image-based cross-modal person re-identification method.

[0137] This application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-described text-image-based cross-modal pedestrian re-identification model.

[0138] This application provides a text-image-based cross-modal person re-identification method and system, computer device, and computer-readable storage medium. It proposes a cross-modal local fine-grained alignment method, which reduces the heterogeneous variance of similar cross-modal samples by deconstructing the image into human body parts and mapping them to text words. This compensates for the information loss in fine-grained discrimination caused by global alignment and solves the problem of discriminative semantics being overwhelmed by irrelevant information due to occlusion, viewpoint changes, and pose shifts. Furthermore, it proposes a cross-modal attribute coarse-grained alignment method, using explicit attribute words in the text description as weak supervision signals. This method compares the consistency of attribute features with those predicted by the image branch, constructing an attribute-visual feature mapping to improve cross-modal semantic consistency and alleviate semantic confusion between visual and linguistic cross-modal approaches.

[0139] Although this application has been disclosed above with reference to embodiments, it is not intended to limit this application. Anyone skilled in the art may make some modifications and refinements without departing from the spirit and scope of this application. Therefore, the scope of protection of this application shall be determined by the appended claims.

Claims

1. A cross-modal person re-identification method based on text-image, characterized in that, include, Prepare a text-image person re-identification dataset and construct attribute labels using text descriptions. Add a local alignment network to the language-image pre-trained model to learn the weights of the text tag vectors corresponding to each local feature, and obtain the corresponding text local features by weighting them. Map the image local features and text local features to each other for cross-modal fine-grained local alignment. Add an attribute alignment network to calculate the attention position of each attribute based on the image patch tag vectors. Use the attribute labels obtained from the attributes in the text as supervision for cross-modal coarse-grained attribute alignment, resulting in a text-image-based cross-modal person re-identification model. A loss function is constructed for the cross-modal person re-identification model based on the text-image. Based on minimizing the loss function, the parameters of the cross-modal person re-identification model based on the text-image are iteratively updated to obtain the trained cross-modal person re-identification model based on the text-image. The trained cross-modal person re-identification model based on the text-image is used for person re-identification. The obtained global image features and global text features are sorted by cosine similarity to obtain the person re-identification retrieval results.

2. The cross-modal person re-identification method based on text-image as described in claim 1, characterized in that, A text-image-based cross-modal person re-identification model is constructed by incorporating a local alignment network and an attribute alignment network into a language-image pre-trained model. The model includes: constructing an image branch to output image features, wherein the image branch includes: an image segmentation layer, an image patch linear mapping layer, a position embedding layer, and a self-attention-based deep learning module encoding layer, used for image segmentation, image patch linear mapping, position embedding, and self-attention-based encoding to output image features, respectively; constructing a text branch to output text features, wherein the text branch includes: a text mapping layer, a word embedding layer, a position embedding layer, and a self-attention-based deep learning module encoding layer, used for text mapping, word embedding, position embedding, and self-attention-based encoding to output text features, respectively; constructing a local alignment network for cross-modal fine-grained local alignment, wherein the local alignment network includes: an image local feature extraction layer and a text local feature selection layer, used for image local feature extraction and text local feature extraction, respectively; and constructing an attribute alignment network for cross-modal coarse-grained attribute alignment, wherein the attribute alignment network includes: The image attribute feature extraction layer and the text attribute label extraction layer are used for extracting image attribute features and attribute matching labels, respectively.

3. The text-image-based cross-modal person re-identification method as described in claim 2, characterized in that, A local alignment network is constructed for cross-modal local fine-grained alignment, including: constructing an image local feature extraction layer, splitting the output features obtained from the image branch, including the learnable classification label vector and all image patch label vectors, and performing average pooling to calculate the mean of the features of all image patch label vectors to obtain the image local features, as shown below: in, The feature with index 0 in the output features of the image branch is the feature corresponding to the learnable classification label vector. The output features of the image branch are indexed from 1 to the end, that is, the features corresponding to all image patch label vectors; The global features of the image are extracted for image features, that is, the features corresponding to the learnable classification label vectors obtained by splitting the image. This is the set of image patch features obtained from the splitting process; This is the subset of image blocks corresponding to the nth local feature after average segmentation; This is the subset of image blocks corresponding to the local features of the Nth image after average segmentation; This represents the nth local feature of the image. This is the Nth local feature of the image; It is a set of local features of the image; This is the average splitting function; This refers to the number of segments, i.e., the number of local image features generated. For average pooling, a text local feature extraction layer is constructed. This layer learns the weights of the text marker vectors corresponding to each local feature using learnable local text parameters, and then weights these weights to obtain the corresponding text local features. This maps image local features to text local features, achieving cross-modal fine-grained alignment. Specifically, the text local feature extraction layer learns the weights of the text marker vectors corresponding to each local feature using learnable local text parameters, and then weights these weights to obtain the corresponding text local features. This process includes: encoding the discrete word sequence input to the text branch; sorting it from largest to smallest and then splitting it to obtain the positions of the insertion start and end marker vectors and all text marker vectors; and then, based on the text marker vectors in the output features of the text branch, learning the weights of the text marker vectors corresponding to each local feature using learnable local text parameters, and then weighting these weights to obtain the corresponding text local features, as shown below: in, The original text description is the data after encoding and insertion of start and end marker vectors; A sorting function for encoding positions from largest to smallest; This is the set of position indices for the sorted text tag vectors; This is the position index of the Lth text tag vector after sorting; The text feature corresponding to the first position index in the output features of the text branch is the feature corresponding to the insertion end marker vector; The feature corresponding to the second position index in the output features of the text branch; The feature set indexed from the 3rd position to the end of the output features of the text branch; For text global features; Features for the insertion start marker vector position obtained from the split; The feature set of the sorted text marker vector positions obtained after splitting; This is a cross-modal attention module used to learn the weights of the text tag vectors corresponding to each local feature, and then weight them to obtain the corresponding local text features. Randomly generated local text centers serve as learnable local text parameters during the training of the cross-modal attention module, determining the number of local text features, defined as being the same as the number of local image features. ； This is the weighted set of local text features; the cross-modal attention module first randomly initializes a text center for each local image, then calculates attention weights on all text tag vectors based on the text center, and finally extracts the local text features corresponding to each locality, as shown below: in, For local text center The scaled similarity score vector of features relative to the sorted text tag vector positions; The attention weight vector is the feature of the sorted text tag vector position relative to the current local text center. This represents the set of local features of the text; the superscript T is the transpose symbol.

4. The text-image-based cross-modal person re-identification method as described in claim 3, characterized in that, A coarse-grained cross-modal attribute alignment network is constructed, including: a text attribute label extraction layer to extract attribute matching labels, which involves: segmenting the input text into sentences using a natural language processing library to obtain word segmentation, part-of-speech tagging, dependency relation labels, and lexical vectors; selecting nouns and adjectives to obtain candidate attribute words; constructing a lexical-attribute dictionary; and labeling the corresponding attribute words in the extracted text description if they exist in the dictionary, thus obtaining attribute matching labels. An image attribute feature extraction layer is also constructed to extract image attribute features, which involves: calculating the attention position of each attribute using learnable attribute location parameters and image patch label vectors through a cross-modal attention module; and then using a fully connected layer as a task-specific head to obtain the corresponding attribute features, as shown below: in, Randomly generated attribute centers serve as learnable attribute location parameters during the training of the cross-modal attention module, used to learn the image patches that each attribute focuses on. ， The number of attributes; To learn the set of all attribute features of the corresponding image patch features, To learn the s-th attribute feature of the corresponding image patch features; It is a linear transformation function; It is a set of image attribute features.

5. The cross-modal person re-identification method based on text-image as described in claim 4, characterized in that, Cross-modal local fine-grained alignment is achieved by mapping local image features to local text features, including: mapping local image feature sets to local text feature sets through a loss function for fine-grained alignment.

6. The cross-modal person re-identification method based on text-image as described in claim 5, characterized in that, The loss function for constructing a text-image cross-modal person re-identification model includes: jointly optimizing similarity distribution matching loss, identity classification loss, and attribute recognition loss. The loss function for constructing the text-image cross-modal person re-identification model is expressed as follows: in, Loss function for a text-image cross-modal person re-identification model; similarity distribution matching loss. The KL divergence between image features and text features represents the identity classification loss. This represents the attribute recognition loss, used to measure the difference in identity prediction using image features and text features. This indicates the difference in attribute prediction results.

7. The text-image-based cross-modal person re-identification method as described in claim 6, characterized in that, The similarity distribution matching loss for constructing a text-image-based cross-modal person re-identification model includes: representing image-text pairs using gradient descent; and calculating the cosine similarity between each image feature and each text feature using cosine similarity, as shown below: Among them, the gradient descent method uses Represents a set of image features. B represents the set of text features; B represents the number of image-text pairs. Cosine similarity; For the i-th image feature, including global image features and image local feature set ； For the j-th text feature, including global text features and image local feature set For each image-text pair cosine distance, calculate the similarity distribution with all image-text pairs cosine distances, as shown below: in, Let be the probability that the i-th image and the j-th text match in a text-image based cross-modal person re-identification model; It is a temperature hyperparameter; This is the k-th text feature; Let be the probability distribution of the matching tags from the i-th image to the j-th text. For the matching label of the k-th text that belongs to the same identity as the i-th image; for each text-image pair cosine distance, calculate the similarity distribution with all text-image pairs cosine distances, as shown below: in, Let be the probability that the j-th text and the i-th image are matched by the text-image based cross-modal person re-identification model; For the k-th image feature; Let be the probability distribution of the matching labels from the j-th text to the i-th image; The matching label for the k-th image that shares the same identity as the j-th text; similarity distribution matching loss. , means as follows: in, The loss is for matching similar distributions; The loss is for matching similarity distributions from image to text direction; The loss is the similarity distribution matching loss from text to image direction; It is a constant; the identity classification loss of the text-image based cross-modal person re-identification model includes obtaining the probability prediction value of each identity after passing the global image features and global text features through the identity classifier, as shown below: in, Here is the weight matrix for the identity classifier. ID represents the number of identities; For identity classifier; and Let be the predicted identity probabilities of the i-th image and the j-th text, respectively, and let be the features of the i-th image at this time. Includes only global image features The j-th text feature Includes only global text features Identity classification loss , means as follows: in, The identity classification loss is used for the i-th image; The identity classification loss is used for the j-th text. Let i be the predicted probability of the i-th image corresponding to the id-th identity; Let be the predicted probability of the j-th text corresponding to the id-th identity; The matching label for the i-th image corresponding to the id-th identity; Let be the matching label for the j-th text corresponding to the id-th identity. Construct the attribute recognition loss for a text-image-based cross-modal person re-identification model, including mapping image attribute features to an attribute space. When the text description and the identity of the pedestrian image are the same, the attributes are also the same. Use the text descriptions of pedestrians in the dataset to obtain attribute labels to supervise image attribute recognition. Learn the attribute information generated during the text description process, represented as follows: in, For attribute classifiers, ； Let be the probability set of the i-th image recognition attribute; The attribute features of the i-th image; attribute recognition loss. , means as follows: in, The label for the s-th attribute obtained through a text description that shares the same identity as the i-th image; Let be the attribute recognition loss for the s-th attribute of the i-th image; where s is the attribute index. Let be the predicted probability of the s-th attribute of the i-th image.

8. A text-image-based cross-modal person re-identification system, used to perform the text-image-based cross-modal person re-identification method as described in any one of claims 1 to 7, characterized in that, include: Data processing module: Prepare text-image pedestrian re-identification dataset and construct attribute labels using text descriptions; Model building module: Based on the language-image pre-trained model, a local alignment network is added to learn the weights of the text tag vectors corresponding to each local feature, and the corresponding text local features are obtained by weighting. The image local features and text local features are mapped to each other to perform cross-modal local fine-grained alignment; an attribute alignment network is added to calculate the attention position of each attribute based on the image patch tag vectors. The attribute labels obtained from the attributes in the text are used as supervision to perform cross-modal attribute coarse-grained alignment, resulting in a text-image based cross-modal person re-identification model; Loss function construction module: Constructs a loss function suitable for the text-image based cross-modal person re-identification model, including similarity distribution matching loss, identity classification loss, and attribute recognition loss; Model training module: Using the prepared text-image person re-identification dataset, the parameters of the text-image-based cross-modal person re-identification model are iteratively updated based on minimizing the loss function to obtain the trained text-image-based cross-modal person re-identification model; Person re-identification module: The trained text-image-based cross-modal person re-identification model is used to perform person re-identification. The obtained global image features and global text features are sorted by cosine similarity to obtain the person re-identification retrieval results; The data processing module, model building module, loss function construction module, model training module, and person re-identification module are connected in sequence.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the text-image-based cross-modal pedestrian re-identification method as described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the text-image-based cross-modal pedestrian re-identification model as described in any one of claims 1 to 7.

Images