A re-identification method based on deep text-visual modal alignment and fusion

Abstract

The application discloses a re-identification method based on deep text-visual modal alignment and fusion, belongs to the field of computer vision and pattern recognition, and designs a deep neural network for text-visual modal alignment and fusion, comprising a text description generation module, a visual feature coding module, a text feature coding module and a text-visual feature fusion module; firstly, the method generates an instance-level text description corresponding to an image by using a visual question and answer; then, two feature extraction branches of a deep-coupled visual encoder and a text encoder are used to generate strongly aligned visual features and text features; finally, a text-visual modal alignment and fusion technology is adopted, domain-independent normalization is combined, image target foreground feature representation is enhanced, instance normalization is combined to extract final features, a loss function model is trained, and a re-identification task is realized. The method disclosed by the application can effectively enhance the cross-domain generalization performance of a target re-identification model.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of computer vision and pattern recognition, and particularly relates to a re-identification method based on deep text-visual modal alignment and fusion. BACKGROUND

[0002] Computer vision has been a hot and difficult research topic in recent years, and re-identification, as a sub-problem of image retrieval, has important application significance in the field of security and smart city. Re-identification needs to realize the function of cross-camera retrieval of targets in the monitoring picture. The object of target re-identification research is the overall features of the target, which can be combined with target recognition, target tracking, etc., and can be widely applied in the fields of camera monitoring, smart city, intelligent retail, etc., and has important significance in theory and practice.

[0003] The supervised learning and same-domain method research of re-identification have been very mature, and good retrieval effect has been achieved on existing data sets, but the cross-domain target re-identification technology is still a big challenge. The differences between different re-identification data sets are large, which can be divided into two kinds: the difference between different camera equipment and the difference in target characteristics. The difference between different camera equipment includes light difference, indoor and outdoor scene difference, low resolution of monitoring image, blurred image, different camera shooting angle, etc. The difference in target characteristics includes row occlusion, posture change, clothing accessories, etc.

[0004] Due to the significant differences between different data sets, the performance of the model will usually decrease significantly when directly tested on other data sets after training on a single data set. In addition, the traditional target re-identification method mainly relies on the visual coding of the image, ignoring the rich semantic information contained in the image. This method is easy to make the model learn the background noise, so as to fail to effectively obtain the robust feature representation of the target, resulting in overfitting phenomenon of the model in the training process. SUMMARY

[0005] The purpose of the present application is to provide a re-identification method based on deep text-visual modal alignment and fusion.

[0006] The technical solution for achieving the purpose of the present application is: a re-identification method based on deep text-visual modal alignment and fusion, comprising the following steps:

[0007] Step 1: Collect source domain RGB image data set D s with not less than N camera perspectives S , N represents the size of the source domain data set D T; each image is the target corresponding ID image that has retrieved only one target, the data set naming method is pid_camid_index.jpg, wherein pid is the target label, camid is the camera label, and index is the image index; only the source domain data set D s can be accessed during training T ;

[0008] Step 2: pass the data set into the instance-level description generator to obtain the image corresponding description through different prompting and inquiry methods, and obtain the text modal input;

[0009] Step 3: data pre-processing is performed on the visual image in step 1, the training image adopts a plurality of image enhancement methods, and the test image only adopts general cropping and normalization;

[0010] Step 4: after the image pre-processed in step 3 and the description generated in step 2 are encoded by a learnable prompt word, the text-visual modal model is input, and deep coupling is performed between network layers to obtain strongly aligned image features and text features;

[0011] Step 5: the image features and the text features obtained in step 4 are input into the text-visual modal fusion network to obtain the text enhanced image features;

[0012] Step 6: the features after text-visual modal fusion in step 5 are subjected to a domain-independent normalization module to obtain the final feature representation;

[0013] Step 7: the final feature representation obtained in step 6 is used to calculate the distance between the features to measure the similarity and perform re-identification sorting.

[0014] An electronic device includes a memory, a processor, and a computer program stored on the memory and executable on the processor, and the processor implements the above-mentioned re-identification method based on deep text-visual modal alignment and fusion when executing the program.

[0015] A computer readable storage medium has a computer program stored thereon, and the program is executed by a processor to implement the above-mentioned re-identification method based on deep text-visual modal alignment and fusion.

[0016] A computer program product includes a computer program, and the computer program is executed by a processor to implement the above-mentioned re-identification method based on deep text-visual modal alignment and fusion.

[0017] Compared with the prior art, the present application has the following advantages:

[0018] (1) The current target re-identification task only uses image single modal input, and the method overcomes the problem that the re-identification task can only be trained in single mode, generates detailed image target descriptions by using the visual question answering technology, not only replaces the low-efficiency manual labeling, but also enables the network to learn more on the target foreground, so that the model learns enhanced image representation, thereby having stronger robustness when crossing the domain.

[0019] (2) The text images between the text image encoders are deeply coupled by mutual influence and mutual connection; the text-visual modal fusion module aligns the text image features, so that the text effectively enhances the image.

[0020] (3) The text-visual modal fusion module added after the network extracts the visual language space features aligns the text image features, so that the text representation features and the image extraction features can be deeply fused, and the text effectively enhances the image.

[0021] (4) The images taken by different cameras have great style difference, and the style difference includes light difference, color difference, indoor and outdoor scene difference, shooting angle difference and target feature difference, and the addition of normalization can better adapt to the variable target domain distribution, thereby improving the cross-domain generalization ability of target re-identification. BRIEF DESCRIPTION OF DRAWINGS

[0022] Figure 1 The target re-identification method flowchart of the application. DETAILED DESCRIPTION

[0023] The application will be described in further detail below with reference to the accompanying drawings.

[0024] The text-visual modal alignment and fusion deep neural network provided by the application mainly comprises a text description generation module, a visual feature encoding module, a text feature encoding module and a text-visual feature fusion module; the method first generates an instance-level text description corresponding to an image by using visual question answering, then generates strongly aligned visual features and text features by using two feature extraction branches of a deeply coupled visual encoder and a text encoder, finally uses text-visual alignment and fusion technology, jointly domain-independent normalization, enhances image target foreground feature representation, then jointly instance normalization extracts the final features, uses a cross-entropy loss function to constrain model training, calculates the distance between the picture to be searched and the pictures in the database, measures the similarity according to the distance, and outputs the N pictures most similar to the picture to be searched.

[0025] Some exemplary embodiments of the application will be described below with reference to the accompanying drawings.

[0026] According to the example of the application, a target re-identification model construction method based on deep text-visual modal alignment and fusion is proposed to improve the cross-domain generalization performance of target re-identification. Figure 1 As shown in the figure, the implementation of the method generally includes the following steps:

[0027] Step 1, collect the re-identification dataset, and divide the dataset according to the re-identification training test criteria.

[0028] Step 2, input the re-identification dataset into the instance-level description generator, obtain the description corresponding to the image through visual question answering through different prompts and inquiry methods.

[0029] Step 3, data preprocessing is performed on the re-identification dataset, and multiple image enhancement methods are performed on the training dataset, and the test image adopts general cropping and normalization.

[0030] Step 4, after the image preprocessed in step 3 and the description generated in step 2 are encoded with learnable prompts, they are input into the text-visual modal model, and deep coupling is performed between network layers to obtain strongly aligned image features and text features.

[0031] Step 5, input the image features and text features obtained in step 4 into the text-visual modal fusion network to obtain text-enhanced image features.

[0032] Step 6, the features after text-visual modal fusion in step 5 are input into the domain-independent normalization module to obtain the final feature representation.

[0033] Step 7, calculate the distance between the final feature representations obtained in steps 5 and 6 to measure their similarity and perform re-identification sorting.

[0034] Further, in step 1, a source domain RGB image dataset with no less than 10,000 images from different camera angles is collected Wherein represents the image data in the source domain dataset D S , N represents the size of the source domain dataset D S , H represents the height of the image, and W represents the width of the image. Meanwhile, a target domain dataset Each image has retrieved only one corresponding ID image of the target, and each ID image comes from a different camera. The dataset naming method is pid_camid_index.jpg, wherein pid is the target label, camid is the camera label, and index is the image index; the source domain dataset D s and the target domain D TEach dataset must be divided into a training dataset and a test dataset. The test dataset is further divided into a Gallery set and a Query set. The IDs in the Gallery set must contain the IDs in the Query set to ensure that the data used in the Query set matches the data of different images with the same ID in the Gallery during the query phase. During training, only the source domain training dataset can be accessed, while the target domain test dataset is used during testing.

[0035] Furthermore, step 2 specifically includes:

[0036] Using visual question answering, the image x and open-ended language questions that expand around the image are combined. Input a description generator G, output an image description Where N I This indicates the total number of language problems.

[0037] Furthermore, in step 3, cross-domain adaptive image enhancement is performed. The specific method is as follows: First, a basic resize operation is performed. The original image of size H×W is resized to H'×W' using bicubic interpolation, employing the nearest 4×4 pixels to the target location. This ensures uniformity of input size across different networks, even in the training dataset. Additionally, several cross-domain data enhancements are performed. To increase image diversity, this method employs random cropping, random elimination, random flipping, padding, and image feature enhancement filtering (bilateral filtering, Sobel filtering, etc.). To ensure the effectiveness of image enhancement, during the training phase, images are sequentially subjected to random flipping, padding, random cropping, color calibration, random image patch replacement, bilateral filtering, and Sobel filtering, then converted to tensors and normalized. Random elimination is then performed, finally yielding the preprocessed training dataset D'. S ={x' 1,s ,x' 2,s ,…,x' N,s},x' i ∈R 3 ×H′×W′ During the testing phase, only a general method of cropping and converting the dataset into tensors for normalization is used to obtain the preprocessed test dataset D'. T ={x' 1,T ,x' 2,T ,…,x' N,T},x' i ∈R 3×H'×W' .

[0038] In the above method, the specific steps in step 4 are as follows:

[0039] Step 4-1, first use the user-defined or fully automatically initialized learnable prompt words [V] = [V1] [V2]…[V M ], the initial text encoding W0 corresponding to the description C generated in step A2 is dynamically text encoded

[0040] p i = [V1] [V2]…[V M ][W0], i ∈ (1, 2, …, N I )

[0041] For instance-level learning, use each image x in the preprocessed data set in step A3 to further encode the learnable prompt words with a lightweight module h θ (·) to further enhance the association between modalities, prompt words and description words:

[0042] [V] i (x) = [V] i +h θ (x), i ∈ (1, 2, …, M

[0043] Get dynamic text encoding

[0044] p i = [V1(x)] [V2(x)]…[V M (x)][W0], i ∈ (1, 2, …, N I )

[0045] Step 4-2, first block the image x, increase the class encoding to get the image block encoding E0 = PatchEmb(x), take the dynamic text encoding p(x) obtained in step 1) and the image block encoding E0 as input, and perform coupling between network layers, first use the learnable prompt words [V] obtained in step A4-1 to share to the visual modality before the first layer input of the visual encoder and the text encoder, and use the function proj to generate the visual class prompt words [V']:

[0046] [V'] = proj([V])

[0047] Merge the visual class prompt words [V'] and the image block encoding E0 into [E0, V'] as the final input of the first layer of the visual encoder.

[0048] Step 4-3, in the multi-modal model, the user can customize the number of deep coupling text layers K L and the number of visual layers K V , the K-layer visual encoder and the K-layer text encoder The text encoder inputs [V(x)] [C] at each layer and outputs as follows:

[0049]

[0050] The visual encoder inputs [E i , V'] at each layer and outputs as follows:

[0051]

[0052] Further, the visual language model in step 4 is based on a network containing K layers of a visual encoder and K layers of a text encoder The visual encoder adopts two structures, one is ModifiedResNet and the other is VisionTransformer. The text encoder and the image encoder adopt a K-layer d l dimensional model, and each layer of the model adopts the same residual structure H-head attention module. First, each layer of input is subjected to layer normalization, then subjected to H-head attention, then subjected to layer normalization, and finally subjected to a multi-layer perception layer composed of a linear projection layer and an activation layer to obtain each layer of input. The model performs encoding on the text in lowercase byte pair encoding (BPE) representation. First, the input text is tokenized, then position encoded, and then subjected to the H-head attention module of each layer of the text encoder mentioned above, and a learnable prompt for this token is embedded in each layer of input. The image encoder adopts a K-layer d l dimensional model, and each layer of the model is a residual structure H-head attention module. For the image branch, the input image is subjected to 2-dimensional convolutional layer image block encoding, added with class token encoding, then position encoded, and then subjected to the H-head attention module of each layer of the image encoder mentioned above, and a deep text-coupled visual learnable prompt for this token is embedded in each layer of input.

[0053] In the above method, the specific steps of step 5 are as follows:

[0054] Step 5-1, the image feature E K and the text feature W K obtained in step 4-3 are subjected to cross-attention mechanism. First, the image-text cross-attention part generates image Query feature Q i , text Key feature K t , and text Value feature V t through three mapping matrices wherein the image feature is mapped from dimension d i to the dimension of Q i the text feature is mapped from dimension d​t Mapping to K t Dimensions Text features from dimension d t Mapping to V t Dimensions Finally, the image-text feature Attn is fused. i2t :

[0055]

[0056]

[0057] f i2t =softmax(Attn) i2t )·V t

[0058] Step 5-2: Text-image cross-attention part, through three mapping matrices Generate text query features Q t Generate image key features K i Generate image Value features V i ,in Text features from dimension d t Mapping to Q t Dimensions Image features from dimension d i Mapping to K i Dimensions Image features from dimension d i Mapping to V i Dimensions Finally, the text-image feature Attn is fused. t2 i:

[0059]

[0060]

[0061] f t2i =softmax(Attn) t2i )·V i

[0062] Ultimately, a residual structure is needed to fuse the features:

[0063] f = W K +f t2i +f i2t

[0064] Further, in step 6, after obtaining the fusion feature f through text enhancement image feature representation, the domain-agnostic normalization adjustment statistical parameter is adopted, the user can dynamically adjust the disturbance coefficient (alpha, beta), and the statistical parameter gamma is calculated da And beta da ,

[0065] Gamma da = alpha * sigma (f)

[0066] Beta da = beta * sigma (f)

[0067] Where alpha ~ uniform (1-omega, 1+omega), omega is a hyperparameter in (0, 0.5); the source domain feature mean mu (f) and standard deviation sigma (f) are calculated, the source domain distribution is adjusted through normalization DaFN (·), and the final feature representation f is obtained global :

[0068]

[0069] Further, the loss function of the deep neural network with cross-domain generalization ability in step 7 is constructed, and the construction method is specifically: using cross-entropy loss function as the classification loss function, cross-entropy describes the distribution between two probabilities; using a three-tuple loss function as a difficult sample learning loss function, the three-tuple loss can learn subtle features; using a text-image contrast loss, which can align the text-image feature space.

[0070] In view of the problem that only visual modal data exists in the cross-domain target re-identification task, and background-independent information is learned in the training process, leading to model overfitting, the present application is based on the multi-modal learning idea, and generates corresponding text description based on visual modal data, so that the visual and text two modalities are input into the deep coupled text-visual modal network for alignment, and the image representation is enhanced through the text-visual fusion module after the network, and finally through the normalization for the cross-domain, better adapt to the variable domain style. Specifically, the method first generates the corresponding instance-level text description of the image by using visual question answering, then generates strongly aligned visual features and text features through a deep coupled visual encoder and a text encoder two feature extraction branches respectively, finally adopts text-visual modal fusion technology combined with domain-independent normalization to enhance image target foreground feature representation, and then extracts the final feature through instance normalization, and then trains the loss function model to realize the re-identification task. The method proposed by the present application can effectively enhance the cross-domain generalization performance of the re-identification model.

Claims

1. A re-identification method based on deep text-visual modality alignment and fusion, characterized in that, Comprising the following steps: Step 1: Collect source domain RGB image dataset with at least 10,000 images from different camera views wherein represents the image data in the source domain dataset D S , N represents the size of the source domain dataset D S , H represents the height of the image, and W represents the width of the image. The target domain dataset is also required to be collected, with each image having only one corresponding ID image of the target retrieved, each ID image coming from a different camera. The dataset naming method is pid_camid_index.jpg, where pid is the target label, camid is the camera label, and index is the image index. The source domain dataset D s and the target domain dataset D T are both divided into training and testing datasets, with the testing dataset divided into Gallery and Query sets. The IDs in the Gallery set need to include the IDs in the Query set, ensuring that in the query phase using the Query set data, the data matches the same ID but different images in the Gallery. During training, only the source domain dataset can be accessed, and during testing, the target domain dataset is used. Step 2: The data set is transmitted into the instance-level description generator to obtain the image corresponding description through different prompting and inquiry methods, and the text modal input is obtained; Step 3: The visual image in step 1 is preprocessed, the training image adopts a plurality of image enhancement methods, and the test image only adopts general cropping and normalization; Step 4: After the image in step 3 is preprocessed and the description generated in step 2 is encoded, the text-visual modal model is input, and the deep coupling between network layers is performed to obtain the strongly aligned image features and text features; Step 5: The image features and text features obtained in step 4 are input into the text-visual modal fusion network to obtain the text enhanced image features; Step 6: The features after text-visual modal fusion in step 5 are subjected to domain-independent normalization module to obtain the final feature representation; After the text enhanced image feature representation is obtained, the fusion feature f, the domain-agnostic normalization adjusts statistical parameters, and the user can dynamically adjust the disturbance coefficients (a, b), and the statistical parameters g are calculated da and b da , γ da = αs(f) β da = βσ(f) where a ~ uniform(1 - ω, 1 + ω), ω e (0, 0.5) is a hyper-parameter; compute the source domain feature mean μ(f) and standard deviation σ(f), adjust the source domain distribution by normalizing DaFN(·) to get the final feature representation f global : Step 7: The final feature representation obtained in step 6 is used to calculate the distance between the features to measure the similarity for re-identification sorting.

2. The method of claim 1, wherein, The step 2 is specifically: adopting visual question answering, an image x and an open-ended language question posed around the image an input description generator G, outputting an image description where N I denotes the total number of language questions.

3. The method of claim 1, wherein, The image enhancement of cross-domain adaptation of the image in step 3 is specifically as follows. First, a basic Resize operation is performed to Resize the original image with a size of HxW to H'xW' through bicubic interpolation using 4x4 pixels closest to the target position in the image, so that the original image is Resize to H'xW', that is, the uniformity of different network input sizes is ensured even for the training data set; in the training stage, the image is sequentially subjected to random flipping, padding, random cropping, color calibration, random replacement of image blocks, bilateral filtering, Sobel filtering, conversion into a tensor and normalization, then random elimination, and finally the preprocessed training data set D' is obtained S ={x' 1,s ,x' 2,s ,…,x' N, s},x' i ∈R 3×H′×W′ ; in the test stage, only the general cropped tensor is converted for normalization to obtain the preprocessed test data set D' T ={x' 1,T ,x' 2,T ,…,x' N,T},x' i ∈R 3×H′×W′ .

4. The method of claim 1, wherein, The step 4 is specifically: Step 4-1: First, use M learnable prompt words [V] = [V1] [V2]…[V M ] initialized by user customization or fully automatic generation, and the initial text encoding W0 generated in step 2 corresponding to the description C to dynamically encode the text p i = [V1] [V2]... [V M ][W0], i e (1, 2,..., N I ) Using each image x in the pre-processed dataset from step 3 to learn the prompt words with a light-weight module h θ (·) Further encoding, further enhancing the association between modalities, prompt words and description words: [V] i (x) = [V] i + h θ (x), i e (1,2,..., M) obtaining dynamic text encoding p i = [V1(x)] [V2(x)]... [V M (x)] [W0], i e (1, 2,..., N I ) Step 4-2: First, the image x is divided into blocks to increase the class token to obtain the image block encoding E0=PatchEmb(x), the dynamic text encoding p(x) obtained in step 4-1 and the image block encoding E0 are input to couple between network layers, first, before the first layer input of the visual encoder and the text encoder, the learnable prompt [V] obtained in step 4-1 is shared to the visual modal, and the visual class prompt [V'] is generated using the function proj: [V'] = proj([V]) The visual class prompt [V'] is merged with the image block encoding E0 to form [E0, V'] as the final input of the first layer of the visual encoder; Step 4-3: User-defined depth K of text modality in the text-visual modality model L with visual modality K V K layers of visual encoder with K layers of text encoder Input of each layer of text encoder [V(x)][C], output as follows: Visual encoder input to each layer [E i , V'] is output as follows:

5. The method of claim 1, wherein, The step 5 is specifically: Step 5-1: Obtain image feature E from step 4-3 K and text feature W K , using cross-attention mechanism, first image-text cross-attention part, through three mapping matrices generate image Query feature Q i , text Key feature K t , text Value feature V t , where map image feature from dimension d i to dimension of Q i map text feature from dimension d t to dimension of K t map text feature from dimension d t to dimension of V t finally fuse image-text feature Attn i2t :​​​ f i2t = softmax(Attn i2t ) · V t Step 5-2: Text-Image Cross Attention Part, through three mapping matrices Generate text Query feature Q t , generate image Key feature K i , generate image Value feature V i , where Map text feature from dimension d t to dimension of Q t Map image feature from dimension d i to dimension of K i , Map image feature from dimension d i to dimension of V i Final fusion text-image feature Attn t2i :​​​ f t2i = softmax(Attn t2i ) · V i Finally, the residual structure is used to fuse the features: f = W k + f t2i + f i2t .

6. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the program to realize the method of any one of claims 1-5.

7. A computer-readable storage medium having stored thereon a computer program, characterized in that, The program is executed by the processor to realize the method of any one of claims 1-5.

8. A computer program product comprising a computer program, characterized in that, The computer program is executed by the processor to realize the method of any one of claims 1-5.

Images