Re-identification method based on deep text-visual modal alignment and fusion

Abstract

The invention discloses a re-identification method based on deep text-visual modal alignment and fusion, and belongs to the field of computer vision and pattern recognition, the method designs a deep neural network for text-visual modal alignment and fusion, and the method comprises a text description generation module and a visual feature coding module. A text feature coding module and a text-visual feature fusion module; the method comprises the following steps: firstly, generating instance-level text description corresponding to an image by utilizing visual questions and answers, then respectively generating strong-aligned visual features and text features through two feature extraction branches of a visual encoder and a text encoder which are deeply coupled, and finally, combining domain-independent normalization by adopting a text-visual modal alignment fusion technology, so as to obtain a text-visual modal alignment fusion model. Image target foreground feature representation is enhanced, final features are extracted in combination with instance normalization, and then a re-identification task is realized by using loss function model training. According to the method provided by the invention, the cross-domain generalization performance of the target re-identification model can be effectively enhanced.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision and pattern recognition, and specifically relates to a re-recognition method based on deep text-visual modality alignment and fusion. Background Art

[0002] Computer vision has been a hot topic and a difficult topic in recent years. As a sub-problem of image retrieval, re-identification has important application significance in the security field and smart city. Re-identification requires the function of cross-camera retrieval of targets in the monitoring screen. The object of target re-identification research is the overall characteristics of the target. It can be combined with target recognition and target tracking, and can be widely used in camera monitoring, smart city, smart retail and other fields. It has important significance in theory and practice.

[0003] The research on supervised learning and same-domain methods for re-identification is very mature and has achieved good retrieval results on existing datasets. However, cross-domain target re-identification technology is still a big challenge. There are large differences between different re-identification datasets. These differences can be divided into differences between different camera devices and differences in target characteristics. Differences between different camera devices include differences in lighting, differences in indoor and outdoor scenes, low resolution of surveillance images, blurred images, and different camera shooting angles. Differences in target characteristics include line occlusion, posture changes, clothing accessories, etc.

[0004] Due to the significant differences between different datasets, the performance of a model trained on a single dataset will usually drop significantly when it is directly tested on other datasets. In addition, traditional object re-identification methods mainly rely on the visual encoding of images, ignoring the rich semantic information contained in the images. This method easily causes the model to learn background noise, making it impossible to effectively obtain a robust feature representation of the target, resulting in overfitting of the model during training. Summary of the invention

[0005] The purpose of the present invention is to provide a re-identification method based on deep text-visual modality alignment and fusion.

[0006] The technical solution to achieve the purpose of the present invention is: a re-identification method based on deep text-visual modality alignment and fusion, comprising the following steps:

[0007] Step 1: Collect a source domain RGB image dataset D with no less than N images from different camera perspectives s , N represents the source domain dataset D S The size of the target domain dataset D T; Each image is a target ID image with only one target. The dataset is named pid_camid_index.jpg, where pid is the target label, camid is the camera label, and index is the image index. During training, only the source domain dataset D can be accessed. s , the target domain dataset D is used for testing T ;

[0008] Step 2: Pass the dataset into the instance-level description generator, obtain the corresponding description of the image through different prompts and inquiry methods, and obtain text modality input;

[0009] Step 3: Perform data preprocessing on the visual images in step 1. A variety of image enhancement methods are used for training images, and only general cropping and normalization are used for test images.

[0010] Step 4: After encoding the learnable cue words, the image preprocessed in step 3 and the description generated in step 2 are input into the text-visual modality model, and deep coupling is performed between the network layers to obtain strongly aligned image features and text features;

[0011] Step 5: Input the image features and text features obtained in step 4 into the text-visual modality fusion network to obtain the image features after text enhancement;

[0012] Step 6: The features after the text-visual modality fusion in step 5 are passed through a domain-independent normalization module to obtain the final feature representation;

[0013] Step 7: The final feature representation obtained in step 6 calculates the distance between features to measure their similarity for re-identification ranking.

[0014] An electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the above-mentioned re-identification method based on deep text-visual modality alignment and fusion is implemented.

[0015] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the above-mentioned re-identification method based on deep text-visual modality alignment and fusion.

[0016] A computer program product includes a computer program, which, when executed by a processor, implements the above-mentioned re-identification method based on deep text-visual modality alignment and fusion.

[0017] Compared with the prior art, the present invention has the following significant advantages:

[0018] (1) Currently, the object re-identification task only uses single-modal image input. This method overcomes the problem that the re-identification task can only be trained in a single mode. It uses visual question-answering technology to generate detailed image target descriptions, which not only replaces the inefficient manual annotation, but also enables the network learning to focus more on the target foreground, so that the model learns enhanced image representation, which makes it more robust across domains.

[0019] (2) The text and image encoders use deep coupling to influence and connect each other. The text-visual modality fusion module aligns the text and image features, so that the text can effectively enhance the image.

[0020] (3) The text-visual modality fusion module added after the network extracts the visual language space features aligns the text image features, allowing the features represented by the text and the features extracted from the image to be deeply integrated, so that the text can effectively enhance the image.

[0021] (4) Images taken by different cameras have large style differences, including lighting differences, color differences, indoor and outdoor scene differences, shooting angle differences, and target feature differences. Adding normalization can better adapt to the variable target domain distribution, thereby improving the cross-domain generalization ability of target re-identification. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 This is a flow chart of the target re-identification method of the present invention. DETAILED DESCRIPTION

[0023] The present invention is further described in detail below in conjunction with the accompanying drawings.

[0024] A deep neural network for text-visual modality alignment and fusion proposed in the present invention mainly comprises a text description generation module, a visual feature encoding module, a text feature encoding module and a text-visual feature fusion module; the method firstly generates an instance-level text description corresponding to an image by using visual question answering, and then generates strongly aligned visual features and text features respectively through two feature extraction branches of a deeply coupled visual encoder and a text encoder, finally adopts a text-visual alignment fusion technology, combines domain-independent normalization, enhances the image target foreground feature representation, and then combines instance normalization to extract the final feature, uses a cross entropy loss function to constrain model training, calculates the distance between a to-be-retrieved image and an image in a database, measures similarity based on the distance, and outputs N images that are most similar to the to-be-retrieved image.

[0025] Some exemplary embodiments of the present invention are described below in conjunction with the accompanying drawings.

[0026] According to the example of the present invention, a method for constructing a target re-identification model based on deep text-visual modality alignment and fusion is proposed to improve the cross-domain generalization performance of target re-identification. Figure 1 As shown, the implementation of this method generally includes the following steps:

[0027] Step 1: Collect the re-identification dataset and divide the dataset according to the re-identification training and testing criteria.

[0028] Step 2: Pass the re-identification dataset into the instance-level description generator, and obtain the corresponding description of the image through visual question answering through different prompts and inquiry methods.

[0029] Step 3: Preprocess the re-identification dataset, perform various image enhancement methods on the training dataset, and use general cropping and normalization on the test images.

[0030] In step 4, the image preprocessed in step 3 and the description generated in step 2 are encoded with learnable prompt words and then input into the text-visual modality model. Deep coupling is performed between the network layers to obtain strongly aligned image features and text features.

[0031] Step 5: Input the image features and text features obtained in step 4 into the text-visual modality fusion network to obtain the image features after text enhancement.

[0032] Step 6: After the text-visual modality fusion in step 5, the features are passed through a domain-independent normalization module to obtain the final feature representation.

[0033] In step 7, the final feature representation obtained in step 5 and step 6 calculates the distance between the features to measure their similarity for re-identification ranking.

[0034] Furthermore, in step 1, a source domain RGB image dataset of no less than 10,000 images from different camera perspectives is collected. in Denotes the source domain dataset D S Image data in, N represents the source domain dataset D S The size of the image, H represents the height of the image, W represents the width of the image, and the target domain dataset needs to be collected Each image is an ID image with only one target. Each ID image comes from a different camera. The dataset is named pid_camid_index.jpg, where pid is the target label, camid is the camera label, and index is the image index. s and target domain data D TThe data sets must be divided into training data sets and test data sets, where the test data sets are divided into Gallery sets and Query sets. The ID of the Gallery set needs to include the ID in the Query set to ensure that when using the Query set data for testing, it matches the data of different images with the same ID in the Gallery. During training, only the source domain training data set can be accessed, and during testing, the target domain test data set is used.

[0035] Furthermore, the step 2 is specifically as follows:

[0036] Using visual question answering, we can pair an image x with an open-ended language question surrounding the image. Input description generator G, output image description Where N I Indicates the total number of language questions.

[0037] Furthermore, in step 3, the image is enhanced for cross-domain adaptation. The specific method is as follows: first, a basic resize operation is performed to resize the original image of size H×W using the bicubic interpolation method using the 4×4 pixels in the image that are nearest to the target position for cubic interpolation calculation, and the original image is resized to H'×W'. Even the training data set can ensure the uniformity of the input size of different networks; in addition, multiple cross-domain data enhancements are performed as follows: to increase image diversity, this method uses random cropping, random elimination, random flipping, padding, and filtering to enhance image features (bilateral filtering, sobel filtering, etc.). To ensure the effectiveness of image enhancement, in the training stage, the image is randomly flipped, padding, randomly cropped, color calibrated, randomly replaced with image blocks, bilaterally filtered, and Sobel filtered, and then converted into a tensor and normalized, followed by random elimination, and finally the preprocessed training data set D' is obtained. S ={x' 1,s ,x' 2,s ,…,x' N,s},x' i ∈R 3 ×H′×W′ ; In the test phase, only the general cropping is used to convert it into a tensor for normalization to obtain the preprocessed test data set D' T ={x' 1,T ,x' 2,T ,…,x' N,T},x' i ∈R 3×H'×W' .

[0038] In the above method, the specific steps in step 4 are:

[0039] Step 4-1, first use the user-defined or fully automatically generated initialization learnable prompt words [V] = [V1] [V2] ... [V M ], the initial text code W0 corresponding to the description C generated in step A2 is dynamically encoded

[0040] p i =[V1][V2]…[V M ][W0],i∈(1,2,…,N I )

[0041] For instance-level learning, use each image x in the preprocessed dataset in step A3 to apply a lightweight module h to the learnable prompt word θ (·) Further encoding, further strengthening the association between modalities, between prompt words and description words:

[0042] [V] i (x) = [V] i +h θ (x),i∈(1,2,…,M)

[0043] Get dynamic text encoding

[0044] p i =[V1(x)][V2(x)]…[V M (x)][W0],i∈(1,2,…,N I )

[0045] Step 4-2, first divide the image x into blocks, add class coding to obtain the image block coding E0 = PatchEmb (x), take the dynamic text coding p (x) obtained in step 1) and the image block coding E0 as input, and couple the network layers. First, before the first layer input of the visual encoder and the text encoder, use the learnable prompt word [V] obtained in step A4-1 to share it with the visual modality, and use the function proj to generate the visual class prompt word [V']:

[0046] [V′]=proj([V])

[0047] The visual cue word [V'] is merged with the image block code E0 into [E0, V'] as the final input of the first layer of the visual encoder.

[0048] Step 4-3: In the multimodal model, users can customize the number of deep coupling text layers K L and the number of visual layers K V , K-layer visual encoder With K-layer text encoder Each layer of the text encoder inputs [V(x)][C] and the output is as follows:

[0049]

[0050] Each layer of the visual encoder input [E i ,V'], the output is as follows:

[0051]

[0052] Furthermore, in step 4, the visual language model is used as the base network, which includes K layers of visual encoders and K-layer text encoder The visual encoder uses two structures, one is ModifiedResNet and the other is VisionTransformer. The text encoder and image encoder use K layers of d l dimensional model, each layer of the model uses the same residual structure H-head attention module. First, the input of each layer is normalized and then passed through the H-head attention, then normalized again, and finally passed through a multi-layer perceptron layer consisting of a linear projection layer and an activation layer to obtain the input of each layer. The model performs lowercase byte pair encoding (BPE) representation of the text. First, the input text is token encoded, then position encoded, and then passed through the H-head attention module of each layer of the text encoder mentioned above, and the embedded text at each layer can learn to prompt this token. The image encoder uses K layers of d l dimensional model, each layer of the model is a residual structure H-head attention module; for the image branch, the input image is encoded by a 2D convolutional layer for image block encoding, and the position encoding is added after adding the token-like encoding. Then, it passes through each layer of the H-head attention module of the image encoder mentioned above, and the visual learnable prompt of this token with deep text coupling is embedded in each layer of the input.

[0053] In the above method, the specific steps of step 5 are:

[0054] Step 5-1, image feature E is obtained in step 4-3 K and text features W K , using the cross attention mechanism, first the image-text cross attention part, through three mapping matrices Generate image Query feature Q i , text key feature K t , text Value feature V t ,in Transform image features from dimension d i Mapping to Q i Dimensions The text features are transformed from dimension dt Mapping to K t Dimensions The text features are transformed from dimension d t Mapping to V t Dimensions Final fusion image-text feature Attn i2t :

[0055]

[0056]

[0057] f i2t =softmax(Attn i2t )·V t

[0058] Step 5-2: Text-image cross attention part, through three mapping matrices Generate text query feature Q t , generate image Key feature K i , generate image Value feature V i ,in The text features are transformed from dimension d t Mapping to Q t Dimensions Transform image features from dimension d i Mapping to K i Dimensions Transform image features from dimension d i Mapping to V i Dimensions Finally, the text-image feature Attn is fused t2 i:

[0059]

[0060]

[0061] f t2i =softmax(Attn t2i )·V i

[0062] Finally, the residual structure is needed to fuse the features:

[0063] f=W K +f t2i +f i2t

[0064] Furthermore, in step 6, after obtaining the fusion feature f through text-enhanced image feature representation, the statistical parameters are adjusted by domain-agnostic normalization. The user can dynamically adjust the perturbation coefficient (α, β) and calculate the statistical parameter γ da and β da ,

[0065] γ da =ασ(f)

[0066] β da =βσ(f)

[0067] Among them, α~uniform(1-ω,1+ω), ω∈(0,0.5) are hyperparameters; the mean μ(f) and standard deviation σ(f) of the source domain features are calculated, and the source domain distribution is adjusted by normalizing DaFN(·) to obtain the final feature representation f global :

[0068]

[0069] Furthermore, the loss function of the deep neural network with cross-domain generalization capability in step 7 is constructed by: using the cross entropy loss function as the classification loss function, the cross entropy describes the distribution between two probabilities; using the triplet loss function as the difficult sample learning loss function, the triplet loss can learn subtle features; using the text image contrast loss, it is possible to align the text image feature space.

[0070] For the cross-domain target re-identification task, there is only visual modality data, and the background-irrelevant information is learned during the training process, which leads to the problem of model overfitting. Based on the idea of ​​multimodal learning, the present invention actively generates corresponding text descriptions based on visual modality data, thereby aligning the two modalities of vision and text into the deeply coupled text-visual modality network, and enhancing the image representation through the text-visual fusion module after the network, and finally better adapting to the changing domain style through cross-domain normalization. Specifically, the method first uses visual question answering to generate the instance-level text description corresponding to the image, and then generates strongly aligned visual features and text features respectively through two feature extraction branches of the deeply coupled visual encoder and text encoder, and finally uses the text-visual modality fusion technology to combine domain-irrelevant normalization to enhance the image target foreground feature representation, and then combines instance normalization to extract the final features, and then uses the loss function model for training to achieve the re-identification task. The method proposed in the present invention can effectively enhance the cross-domain generalization performance of the re-identification model.

Claims

1. A re-identification method based on deep text-visual modality alignment and fusion, characterized in that: The steps include: Step 1: Collect a source domain RGB image dataset D with no less than N images from different camera perspectives s , N represents the source domain dataset D S The size of the target domain dataset D T ; Each image is a target ID image with only one target. The dataset is named pid_camid_index.jpg, where pid is the target label, camid is the camera label, and index is the image index. During training, only the source domain dataset D can be accessed. s , the target domain dataset D is used for testing T ; Step 2: Pass the dataset into the instance-level description generator, obtain the corresponding description of the image through different prompts and inquiry methods, and obtain text modality input; Step 3: Perform data preprocessing on the visual images in step 1. A variety of image enhancement methods are used for training images, and only general cropping and normalization are used for test images. Step 4: After encoding the learnable cue words, the image preprocessed in step 3 and the description generated in step 2 are input into the text-visual modality model, and deep coupling is performed between the network layers to obtain strongly aligned image features and text features; Step 5: Input the image features and text features obtained in step 4 into the text-visual modality fusion network to obtain the image features after text enhancement; Step 6: The features after the text-visual modality fusion in step 5 are passed through a domain-independent normalization module to obtain the final feature representation; Step 7: The final feature representation obtained in step 6 calculates the distance between features to measure their similarity for re-identification ranking.

2. The method according to claim 1, characterized in that In step 1, a source domain RGB image dataset of no less than 10,000 images from different camera perspectives is collected. in Denotes the source domain dataset D S Image data in, N represents the source domain dataset D S The size of the image, H represents the height of the image, W represents the width of the image, and the target domain dataset needs to be collected Each image is an ID image with only one target. Each ID image comes from a different camera. The dataset is named pid_camid_index.jpg, where pid is the target label, camid is the camera label, and index is the image index. s and target domain data D T The data sets must be divided into training data sets and test data sets, where the test data sets are divided into Gallery sets and Query sets. The ID of the Gallery set needs to include the ID in the Query set to ensure that when using the Query set data for testing, it matches the data of different images with the same ID in the Gallery. During training, only the source domain training data set can be accessed, and during testing, the target domain test data set is used.

3. The method according to claim 1, characterized in that The step 2 is specifically as follows: Using visual question answering, we can pair an image x with an open-ended language question surrounding the image. Input description generator G, output image description Where N I Indicates the total number of language questions.

4. The method according to claim 1, characterized in that: In step 3, the image is enhanced for cross-domain adaptation. The specific method is as follows: first, a basic resize operation is performed on the original image of size H×W. The image is interpolated using the 4×4 pixels in the image that are closest to the target position through the bicubic interpolation method. The original image is resized to H'×W'. Even the training data set can ensure the uniformity of the input size of different networks; in the training stage, the image is randomly flipped, padding, randomly cropped, color calibrated, randomly replaced with image blocks, bilaterally filtered, and Sobel filtered, and then converted into a tensor and normalized, and then randomly eliminated, and finally the preprocessed training data set D' is obtained. S ={x' 1,s ,x' 2,s ,…,x' N,s },x' i ∈R 3×H′×W′ ; In the test phase, only the general cropping is used to convert it into a tensor for normalization to obtain the preprocessed test data set D' T ={x' 1,T ,x' 2,T ,…,x'N, T },x' i ∈R 3×H'×W' .

5. The method according to claim 1, characterized in that The step 4 is specifically as follows: Step 4-1: First, use the user-defined or fully automatically generated initial M learnable prompt words [V] = [V1] [V2] ... [V M ], the initial text encoding W0 corresponding to the description C generated in step 2 is dynamically encoded <h2 style=";text-align:left;direction:ltr">p<h2 style=";text-align:left;direction:ltr"> i <h2 style=";text-align:left;direction:ltr"> (B1) [V2]…[V]<h2 style=";text-align:left;direction:ltr"> M <h2 style=";text-align:left;direction:ltr"> ][W0],i∈(1,2,…,N<h2 style=";text-align:left;direction:ltr"> I <h2 style=";text-align:left;direction:ltr"> ) Use each image x in the preprocessed dataset in step 3 to apply a lightweight module h to the learnable prompt word θ (·) Further encoding, further strengthening the association between modalities, between prompt words and description words: [V] i (x)=[V] i +h θ (x),i∈(1,2,…,M) Get dynamic text encoding p i =[V1(x)][V2(x)]…[V M (x)][W0],i∈(1,2,…,N I ) Step 4-2: First, divide the image x into blocks, add class tokens to obtain the image block code E0 = PatchEmb (x), take the dynamic text code p (x) obtained in step 4-1 and the image block code E0 as input, and couple the network layers. First, before the first layer input of the visual encoder and the text encoder, use the learnable prompt word [V] obtained in step 4-1 to share it with the visual modality, and use the function proj to generate the visual class prompt word [V']: [V′]=proj([V]) The visual cue word [V'] is merged with the image block code E0 into [E0, V'] as the final input of the first layer of the visual encoder; Step 4-3: In the text-visual modality model, the user defines the number of deeply coupled text layers K L and the number of visual layers K V , K-layer visual encoder With K-layer text encoder Each layer of the text encoder inputs [V(x)][C] and the output is as follows: Each layer of the visual encoder input [E i ,V'], the output is as follows:

6. The method according to claim 1, characterized in that The step 5 is specifically as follows: Step 5-1: Get the image feature E in step 4-3 K and text features W K , using the cross attention mechanism, first the image-text cross attention part, through three mapping matrices Generate image Query feature Q i , text key feature K t , text Value feature V t ,in Transform image features from dimension d i Mapping to Q i Dimensions The text features are transformed from dimension d t Mapping to K t Dimensions The text features are transformed from dimension d t Mapping to V t Dimensions Final fusion image-text feature Attn i2t : f i2t =softmax(Attn i2t )·V t Step 5-2: Text-image cross attention part, through three mapping matrices Generate text query feature Q t , generate image Key feature K i , generate image Value feature V i ,in The text features are transformed from dimension d t Mapping to Q t Dimensions Transform image features from dimension d i Mapping to K i Dimensions Transform image features from dimension d i Mapping to V i Dimensions Finally, the text-image feature Attn is fused t2i : f t2i =softmax(Attn t2i )·V i Finally, the residual structure is needed to fuse the features: f=W K +f t2i +f i2t 。 7. The method according to claim 1, characterized in that In step 6, after obtaining the fusion feature f through text-enhanced image feature representation, the statistical parameters are adjusted by domain-agnostic normalization. The user can dynamically adjust the perturbation coefficient (α, β) and calculate the statistical parameter γ da and β da , c da =ασ(f) b da =βσ(f) Among them, α~uniform(1-βω,1+βω), ω∈(0,0.5) are hyperparameters; the mean μ(f) and standard deviation σ(f) of the source domain features are calculated, and the source domain distribution is adjusted by normalizing DaFN(·) to obtain the final feature representation f global :

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the method according to any one of claims 1 to 7 is implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method described in any one of claims 1 to 7 is implemented.

Images