Image change description method based on visual language model

Through a visual language model-based method, combined with Transformer structure and coding module, the problem of fine-grained semantic understanding and lack of data in image change description task is solved, and high-accuracy image change description is achieved under different viewing angles and lighting conditions.

CN118918336APending Publication Date: 2024-11-08INSTITUTE OF INFORMATION ENGINEERING CHINESE ACADEMY OF SCIENCES
View PDF 0 Cites 1 Cited by

Patent Information

Application Number
CN202310507784.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2023-05-08
Publication Date
2024-11-08

AI Technical Summary

Technical Problem

Image variation description tasks require fine-grained semantic understanding and high annotation costs. Existing methods cannot effectively capture subtle differences between images and are limited by data volume, especially applications that are difficult to scale under different viewing angles and lighting conditions.

Method used

Using a visual language model-based method, the Transformer structure, viewpoint coding module and change coding module in deep neural networks are used to generate a description of image changes, and the cross entropy loss function is used to optimize the model and design the visual language model. To improve the performance of image variation description on limited data.

Benefits of technology

It improves the accuracy of image change description and the ability to identify fine-grained content, solves the problem of data scarcity, realizes effective training and fine-tuning of the model on less data, and improves the performance of image change description.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118918336A_ABST
    Figure CN118918336A_ABST
Patent Text Reader

Abstract

The invention discloses an image change description method based on a visual language model, and the method comprises the following steps: 1) network inputs are a pre-change image, a post-change image and a text for describing the change, and the text is converted into a feature vector as the first input of the visual language model; 2) performing feature extraction on the image before change and the image after change to preliminarily extract features, and inputting the features corresponding to the two images into a viewpoint coding module; (3) the image features passing through the viewpoint coding module are then input into a change coding module, and image features after different points are enhanced are obtained; 4) taking the image features of the two images after different point enhancement as the second input and the third input of the visual language model, and training the visual language model; and 5) processing a pair of to-be-processed images A and B through the viewpoint coding module and the change coding module in sequence to obtain image features after different point enhancement of the images A and B, inputting the image features into the visual language model, and predicting change description information of the output images A and B.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the field of computer vision technology, and specifically relates to a method for describing image changes based on a visual language model, which can identify the changes from one image to another and describe them in natural language. Background Art

[0002] Endowing machines with the ability to automatically perceive and understand visual information and express it in natural language has long been a goal that researchers have longed to achieve. The image description task aims to generate a natural language description of a given image and has become one of the classic research tasks. Image change description, which generates a natural description of the difference between two similar images, is more challenging and is a further extension of the general image description task. The task of image change description aims to describe the changes between two images by generating detailed sentences about the changes in objects in these images. This task has rich potential in real-world applications, such as assisting ornithologists to distinguish species with similar appearances, assisting medical experts to automatically detect and describe the location of human lesions, and describing scene changes in surveillance tasks. In this task, researchers are only interested in entity-level changes (e.g., color, position, etc. of entities in the image), and do not generate descriptions for changes in perspective, lighting conditions, etc.

[0003] In recent years, with the progress in the field of artificial intelligence and the development of deep learning models, scholars have become increasingly interested in the problem of combining language and visual information. At the same time, a large amount of data combining text and visual information has appeared on the Internet, such as: labeled photos, newspaper illustrations, and multimodal information on social media. The emergence of a large amount of data has brought more possibilities for image change description. At present, the main challenges of the image change description task involve two aspects. First, the image change description task requires fine-grained semantic understanding. Unlike the general image description task that describes a single image, the image change description task requires perceiving the fine-grained content of two similar images to identify the usually subtle differences. For example, the same category of birds in different scenes, regardless of the complex natural environment, the task only focuses on whether the appearance of the birds is the same type of bird. In addition, fine-grained visual differences can be very diverse in different situations, and the appearance of the same type of bird from different perspectives can also be slightly different due to perspective and lighting. Second, the annotation cost of the image change description task is particularly high. Compared with general image description task annotation, it requires higher cognitive load from the annotator, including first observing two images, then comparing the differences, and then describing these differences using natural language to generate annotations in triple format (image 1, image 2, description). Therefore, the existing manually annotated benchmark datasets have limited data size.

[0004] Most of the early methods to solve this problem were based on pixel-level differences in images, and approximated object-level differences by clustering pixels. This method cannot extract deep semantic features of images, and clusters information on shallow features of images, but the effect is not ideal. After that, researchers tried to obtain the changed areas by generating attention maps for paired input images. Although this type of method obtains the key areas that the model needs to pay attention to, it cannot have a deeper understanding of the area, so the effect of converting it into a language description is also not ideal. In addition, most of the aforementioned methods are limited by the data set, that is, the data set used for the experiment does not have a change in perspective, and all experiments and verifications are conducted at the same perspective, so it is difficult to apply them in practice.

[0005] With the popularization of big data, researchers have proposed datasets of images with different viewpoints and lighting conditions, which have enabled the task of describing image changes to be more widely applied in practical applications. Based on this type of dataset, a number of studies have begun to address the problem of different viewpoints and different lighting conditions. The researchers took viewpoint and lighting issues into account in the experiment, trained a more robust neural network by calculating the difference in paired inputs, and achieved good results on this dataset.

[0006] In addition, in order to further enhance the correlation between visual information and generated words between different images in the encoding-decoding process, researchers explored the visual attention mechanism. Today, the image change description network designed based on the attention-enhanced encoder-decoder architecture has achieved a leading position in performance. The introduction of the deep self-attention transformer (Transformer) architecture has brought significant improvements to the performance of various tasks (such as translation, text generation and language understanding). However, the image change description method is still limited by the lack of data. With the popularity of the Transformer model, a series of visual language models have emerged. This type of model is pre-trained on a large amount of data and migrated to specific tasks, which largely solves the problem of missing data. This scheme is also applicable to the image change description task. Many researchers have further improved the performance of this task by applying the visual language model. However, the application of this type of pre-trained model is still at a relatively early stage. Existing studies have failed to explicitly encode the differences between images by inputting the triple form of (image 1, image 2, description) into the network for model building and training. To this end, the method of the invention designs an effective difference encoding scheme based on the introduction of the visual language model to solve the problem of lack of data, thereby improving the performance of image change description. Summary of the invention

[0007] In view of the problem that the image change description task has a high demand for fine-grained information and a lack of data, the purpose of the present invention is to provide an image change description method based on a visual language model. The present invention simultaneously inputs text and two different images into a large visual language model, and designs a pre-training task to generate descriptions of different points for the two different images, that is, descriptions of the changes of the two images.

[0008] The present invention adopts the Transformer structure in the deep neural network, and through the input of the triple (image 1, image 2, description) consisting of the encoded paired image and its change description, the network has the function of generating the description of its change, and can still reconstruct the content of the description when there is no description input in the test phase. In particular, due to the small amount of data, the present invention is based on the visual language model and designs a training method that is consistent with it, so that the model can still make full use of the data for large-scale training and fine-tuning on less data. In addition, in order to generate a description that can capture more differences between the two images in details, the present invention designs a viewpoint encoding module and a change encoding module to jointly enhance the semantic features of the different points, and then generate a description that conforms to the changes in the two images.

[0009] The technical solution adopted by the present invention is as follows:

[0010] A method for describing image changes based on a visual language model, the steps of which include:

[0011] 1) Select multiple images, change each image, generate the changed image and text describing the image change, use the images before and after the image change and the corresponding text as a training sample, and train the selected visual language model;

[0012] 2) The text in the training sample is converted into a feature vector R as the input of the visual language model through the word vector embedding module;

[0013] The features of the training sample before and after the change are extracted respectively, and the feature X corresponding to the image before the change is bef The feature X corresponding to the changed image aft Input the viewpoint encoding module, calculate the similarity between the two images and classify the feature X according to the similarity bef Enhance the features For feature X aft Enhance the features After inputting the change encoding module;

[0014] 3) The change encoding module is characterized in Fusion Features The viewpoint information within is encoded to obtain the image features after enhancement at different points As the input of the visual language model; the variation encoding module Fusion Features The viewpoint information within is encoded to obtain the image features after enhancement at different points As input to the visual language model;

[0015] 4) The visual language model is based on the input feature vector R and image features Output predicted change information Y; optimize the visual language model using a cross entropy loss function according to the change information Y and the text in the corresponding training sample;

[0016] 5) For a pair of images A and B to be processed, the image features of different points of images A and B are obtained by processing them with the viewpoint coding module and the change coding module in turn and input into the visual language model to predict and output the change description information of images A and B.

[0017] Furthermore, the viewpoint encoding module encodes the feature X bef With X aft Perform dot product calculation to obtain the similarity S and input it into the softmax function to obtain the weight matrix α, and then use the weight matrix α to respectively calculate the feature X bef , X aft Perform weighted calculation to obtain the corresponding features

[0018] Furthermore, the viewpoint encoding module is based on the input feature X bef , X aft Get Features and The process is: calculate similarity S = U T bef U aft , then calculate the weight matrix α = softmax(S), Among them, U bef =W bef X bef , U aft =W aft X aft , W bef With W aft is the learnable parameter matrix.

[0019] Furthermore, the change encoding module is a transformer-based encoding module to obtain the image feature The method is:

[0020] 31) The variation encoding module uses a self-attention mechanism to focus on the input features and Encode and obtain a corresponding output vector respectively;

[0021] 32) The corresponding output vector is used as the query vector V. The corresponding output vector is used as the key vector K and the value vector V, and the attention mechanism is used to output the first attention (W q Q, W k K, W v V); then the multi-head self-attention mechanism is used to process the first attention (W q Q, W k K,W v V), obtain multiple different attention results and concatenate them as the output of the unit multi-head attention mechanism; use the residual connection unit to add the input and output of the multi-head attention mechanism to strengthen the output features and obtain the image features

[0022] 33) The corresponding output vector is used as the query vector V. The corresponding output vector is used as the key vector K and the value vector V, and the attention mechanism is used to output the second attention (W q Q, W k K, W v V); then the multi-head self-attention mechanism is used to process the second attention (W q Q, W k K,W v V), obtain multiple different attention results and concatenate them as the output of the unit multi-head attention mechanism; use the residual connection unit to add the input and output of the multi-head attention mechanism to strengthen the output features and obtain the image features Among them, W q , W k With W v are the learnable parameters corresponding to the query vector V, key vector K, and value vector V respectively.

[0023] Furthermore, the word vector embedding module is a pre-trained word vector model word2vec.

[0024] Furthermore, the pre-trained ResNet101 is used to extract the grid features of the image and convert them into feature sequences to be input into the viewpoint encoding module.

[0025] Furthermore, the visual language model includes 4 self-attention transformation layers; the visual language model is trained using three methods; the first method is word coverage training; the second method is region replacement training; and the third method is sequence replacement training.

[0026] Furthermore, the trained visual language model is fine-tuned by masking the text in the training sample in the form of a one-way mask, predicting only one word each time the input is made, and when predicting the word, only the word before the word in the text of the current training sample and the images before and after the two changes in the current training sample are input.

[0027] The present invention also provides a server, characterized in that it includes a memory and a processor, the memory stores a computer program, the computer program is configured to be executed by the processor, and the computer program includes instructions for executing each step in the above method.

[0028] The present invention also provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the steps of the above method when executed by a processor.

[0029] The change coding module of the present invention will and As input, after fusing the viewpoint information of the other image, the self-attention mechanism is used for encoding, with the purpose of further fusing the features of the two images. Specifically, the change encoding module is a transformer-based encoding module with the attention mechanism as its core. and Together, they are input into the attention-based variation encoding module network. and The two matrices are regarded as a 49-dimensional vector representation, and the input change encoding module network has the same output size as the input vector group. Each input vector corresponds to an output vector. For each corresponding output vector, The corresponding vector in is V, The output vector of is used as Q and K, and the attention mechanism is used to output attention (W q Q, W k K, W v V). Where W q , W k With W v are the learnable parameters corresponding to the query vector V, key vector K, and value vector V respectively. The attention mechanism of the query vector Q, key vector K, and value vector V is calculated by the following formula:

[0030]

[0031] Then, a multi-head self-attention mechanism is used to process the results of the attention mechanism. The query vector Q, key vector K, and value vector V in the attention mechanism are projected through h different linear transformations, and finally the different attention results are spliced ​​together:

[0032] MultiHead(Q,K,V)=Concat(head1,...,head h )W o

[0033] Where headn is the output result of the nth attention mechanism. The output result is residually connected and regularized. Residual connection refers to adding the input and output of the multi-head attention mechanism to strengthen the output characteristics. Regularization converts the input of each layer of neurons into the same mean and variance to speed up the convergence of the training process. Therefore, the output of the sub-layer can be expressed as a formula:

[0034] sub_layer_output=LayerNorm(x+(MultiHead(x))

[0035] Where x is the set of Q, K, and V input to the multi-head attention mechanism, and LayerNorm is regularization. The multi-head form of the attention mechanism with residual connections and regularization is called an attention transformation layer. The network contains three layers of self-attention transformation layers. The fully convolutional forward network encodes the output of the multi-head self-attention mechanism sublayer through the fully convolutional forward network, and after the above-mentioned residual connection and regularization, it is used as the output of the entire single-layer self-attention transformation layer.

[0036] where d k is the dimension of vectors Q, K, and V, and the three vectors have the same dimension.

[0037] After changing the encoding module, the output The splicing form.

[0038] The input of the visual language model is the word vector R and the image features after the two images are enhanced. Stitched together: The description information Y of the two images is predicted and output by the visual language model. Then the network is pre-trained using the cross entropy loss function to generate the final result.

[0039] The visual language model is a 4-layer self-attention transformation layer. There are three types of network pre-training, the first of which is word coverage pre-training. The present invention follows the existing visual language model pre-training work, shielding 15% of the words in the input R, of which 80% are replaced with special markers [mask], 10% are replaced with random words, and 10% remain unchanged. The entire model is used to restore the input R for training.

[0040] The second is region replacement pre-training. In step 4) input X bef With X aft In the proposed method, 15% of the input image features are randomly replaced with blocks of other images, and the image features not covered in the batch are defined as negative samples. The contrast loss is used to encourage the model to identify positive samples from negative samples in the batch, thereby forcing the model to have the ability to distinguish image blocks.

[0041] The third type is sequence replacement pre-training. In step 4), the present invention exchanges the order of the two images in the text and image input, and exchanges the corresponding entities in the two images in the text description to obtain a new description, thereby performing the first type of pre-training.

[0042] Furthermore, after pre-training, the present invention performs fine-tuning for the task of describing image changes. Specifically, a one-way mask is used, and only one word is predicted each time input, and when predicting the word, only the words before the word and two images can be input. After fine-tuning, the model training is completed.

[0043] In summary, the present invention designs a method for describing image changes based on a visual language model, which has a high accuracy result. Compared with the existing technology, the advantages of the present invention are:

[0044] 1. Based on the transformer network, the granularity of identifying entities and the accuracy of differences between paired images are improved.

[0045] 2. The visual language model is used to solve the problem of data scarcity and complete the model training using limited data. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 This is a diagram of the scene shadow detection network architecture. DETAILED DESCRIPTION

[0047] The present invention is further described in detail below through specific embodiments and drawings.

[0048] The process of the image change description method based on the visual language model of the present invention is as follows: Figure 1 As shown in the figure, it is mainly divided into two stages: training stage and testing stage.

[0049] The steps of the training phase are as follows:

[0050] 1) During training, generating words is one step and generating sentences is one stage. The annotated text is used as network input and converted into word vectors through the word vector embedding module.

[0051] The processing process of step 1) is as follows: the training is carried out in steps of generating words, and when generating words in each step, only the generated words are used as input, and word vectors are generated through a highly trained word embedding module.

[0052] 2) The image before and after the change are subjected to feature extraction to initially extract features, and the features corresponding to the two images are input into the viewpoint coding module.

[0053] The core of the viewpoint coding module in step 2 is to calculate the similarity of the two input images. First, the pre-trained ResNet101 is used to extract the grid features of the two images. The two input images are extracted into (7, 7, 2048) dimensions by ResNet101, and then converted into (49, 2048) dimensional feature sequences for subsequent input. Specifically, the (49, 2048) dimensional feature sequences of the two images are represented as X bef With X aft , the similarity map S is calculated by dot product, the weight matrix α is obtained by softmax function, and the weight matrix is ​​weighted by the two images to highlight the parts of the image with higher attention. The input of the whole process is X bef With X aft , the output is and As shown in the following formula:

[0054] S=U T bef U aft , where U bef =W bef X bef , U aft =W aft X aft

[0055] α=softmax(S)

[0056]

[0057]

[0058] Among them, W bef With W aft is a learnable parameter, (·) Trepresents the matrix transpose. The viewpoint encoding module calculates the similarity of the two input images and enhances them separately. The similarity map calculates all pairwise similarities between the points of the two feature maps. This essentially represents the position of the object in the previous image in the next image, assuming that the same object still has similar features when viewed from different angles. Therefore, the output and This can be explained as follows: The viewpoint information of the subsequent image is injected into the previous image.

[0059] 3) The image features after the viewpoint coding module are then input into the change coding module to obtain the image features after enhancement at different points.

[0060] The change coding module in step 3) draws on the idea of ​​transformer, and the entire architecture uses the transformer architecture to encode the two images. and As input, the self-attention mechanism is used to calculate the relationship between all blocks of the two images. The purpose is to enhance the fusion features of different points between the two images by calculating the similarity of the inner products of the blocks through the self-attention mechanism. and They are input into the attention-based variation coding module network. The network contains three layers of self-attention transformation layers, each of which consists of two sub-layers, namely the multi-head self-attention mechanism and the fully convolutional forward network. Each sub-layer is added with residual connection and regularization, so the output of the sub-layer can be expressed as the formula:

[0061] sub_layer_output=LayerNorm(x+(SubLayer(x))

[0062] Here x is the result of the multi-head self-attention mechanism, which projects the query vector Q, key vector K, and value vector V in the attention mechanism through h different linear transformations, and finally concatenates the different attention results:

[0063] MultiHead(Q,K,V)=Concat(head1,...,head h )W o

[0064] The attention mechanism of query vector Q, key vector K, and value vector V is calculated by the following formula:

[0065]

[0066] In the present invention, the query vector Q, the key vector K, and the value vector V are all input sequences of this step. Then, the output of the multi-head self-attention mechanism sublayer is encoded by the fully convolutional forward network, and after the above residual connection and regularization, it is used as the output of the entire single-layer self-attention transformation layer. k is the dimension of vectors Q, K, and V, and the three vectors have the same dimension.

[0067] After changing the encoding module, the output The splicing form.

[0068] The visual language model in step 4) also uses the transformer architecture, with the input being R from step 1) and Stitched together: The visual language model outputs the final result Y. The designed loss function is then used to pre-train the network and generate the final result. The visual language model consists of 4 layers of the aforementioned self-attention transformation layers.

[0069] There are three types of network pre-training. The first type is word coverage pre-training. The present invention follows the existing visual language model pre-training work, masks 15% of the words in the input R, of which 80% are replaced with special markers [mask], 10% are replaced with random words, and 10% remain unchanged. The entire model is used to restore the input R for training.

[0070] The second is region replacement pre-training. In step 4) input X bef With X aft In the process, 15% of the input image features are randomly replaced with blocks from other images, and the image features not covered in the batch are defined as negative samples. Contrastive loss is used to encourage the model to identify positive samples from negative samples in the batch, thereby forcing the model to have the ability to distinguish image blocks. Due to the imbalance of training data, the training method uses contrastive learning training, and contrastive loss is used for the constructed positive and negative samples:

[0071]

[0072] where d(·) represents the cosine similarity, τ represents the temperature hyperparameter, and x + Represents the features of the original image x before replacement. The present invention defines the features of the images that are not replaced in the batch as negative samples. The contrast loss encourages the model to identify positive samples from negative samples in the batch.

[0073] The third type is sequence replacement pre-training. In step 4), in the text and image input, the present invention exchanges the order of the two images and simultaneously exchanges the corresponding subject and object in the text, thereby performing the first type of pre-training.

[0074] Furthermore, after pre-training, the present invention performs fine-tuning for the task of describing image changes. Specifically, a one-way mask is used, and only one word is predicted each time input, and when predicting the word, only the words before the word and two images can be input. After fine-tuning, the model training is completed.

[0075] The steps of the testing phase are as follows:

[0076] 1) Input the paired test images into the trained network, and input the words predicted at the beginning of this step into the trained network. The first time text is input, the input is used as a marker for the beginning of a sentence. <bos>.

[0077] 2) Repeat step 1) multiple times until the sentence is predicted to end. <eos>, the generation phase of the entire sentence ends, that is, the description of the image changes ends.

[0078] (1) Test environment:

[0079] System environment: Ubuntu 16.04;

[0080] Hardware environment: Memory: 32GB, GPU: 4 TITIANXP, Hard disk: 1TB;

[0081] (2) Experimental data:

[0082] The CLEVR-Change dataset is used. The dataset contains 67660, 3976, and 7970 image pairs for training, validation, and testing, respectively. The images in the dataset are all geometric figures with clean backgrounds, including different shooting angles, colors, textures, movements, and other changes.

[0083] Training Optimization Method: ADAM

[0084] (3) Experimental results:

[0085] In order to illustrate the effect of the present invention, the present invention compares this method with other methods.

[0086] The test comparison results of the present invention are shown in Table 1 below:

[0087] Table 1. Comparison of test results between the existing method and the present invention

[0088] Serial number method B4 R C M 1 IFDC 49.2 69.1 118.7 32.5 2 DUDA+Aux 51.2 70.5 115.4 37.7 3 The present invention 51.2 71.3 129.4 37.8

[0089] It can be clearly seen from the table that the image change description method based on the visual language model involved in the present invention is generally at a high level.

[0090] The above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit the same. A person skilled in the art may modify or make equivalent substitutions for the technical solutions of the present invention without departing from the spirit and scope of the present invention. The protection scope of the present invention shall be subject to the claims.< / eos> < / bos>

Claims

1. A method for describing image changes based on a visual language model, the steps comprising: 1) Select multiple images, change each image, generate the changed image and text describing the image change, The images before and after the image change and the corresponding text are used as a training sample to train the selected visual language model; 2) The text in the training sample is converted into a feature vector R as the input of the visual language model through the word vector embedding module; the features of the image before and after the training sample are extracted respectively, and the feature X corresponding to the image before the change is extracted. bef The feature X corresponding to the changed image aft Input the viewpoint encoding module, calculate the similarity between the two images and classify the feature X according to the similarity bef Enhance the features For feature X aft Enhance the features After inputting the change encoding module; 3) The change encoding module is characterized in Fusion Features The viewpoint information within is encoded to obtain the image features after enhancement at different points As the input of the visual language model; the variation encoding module Fusion Features The viewpoint information within is encoded to obtain the image features after enhancement at different points As input to the visual language model; 4) The visual language model is based on the input feature vector R and image features Output predicted change information Y; optimize the visual language model using a cross entropy loss function according to the change information Y and the text in the corresponding training sample; 5) For a pair of images A and B to be processed, the image features of different points of images A and B are obtained by processing them with the viewpoint coding module and the change coding module in turn and input into the visual language model to predict and output the change description information of images A and B.

2. The method according to claim 1, characterized in that The viewpoint encoding module performs the feature X bef With X aft Perform dot product calculation to obtain the similarity S and input it into the softmax function to obtain the weight matrix α, and then use the weight matrix α to respectively calculate the feature X bef , X aft Perform weighted calculation to obtain the corresponding features 3. The method according to claim 2, characterized in that The viewpoint encoding module is based on the input feature X bef , X aft Get Features and The process is: calculate similarity S = U T bef U aft , then calculate the weight matrix α = softmax(S), Among them, U bef =W bef X bef , U aft =W aft X aft , W bef With W aft is the learnable parameter matrix.

4. The method according to claim 1, 2 or 3, characterized in that: The change coding module is a transformer-based coding module, which obtains the image features The method is: 31) The variation encoding module uses a self-attention mechanism to focus on the input features and Encode and obtain a corresponding output vector respectively; 32) The corresponding output vector is used as the query vector V. The corresponding output vector is used as the key vector K and the value vector V, and the attention mechanism is used to output the first attention (W q Q,W k K,W v V); then the multi-head self-attention mechanism is used to process the first attention (W q Q,W k K,W v V) Obtain multiple different attention results and concatenate them as the output of the unit multi-head attention mechanism; The residual connection unit is used to add the input and output of the multi-head attention mechanism to strengthen the output features and obtain image features. 33) The corresponding output vector is used as the query vector V. The corresponding output vector is used as the key vector K and the value vector V, and the attention mechanism is used to output the second attention (W q Q,W k K,W v V); then the multi-head self-attention mechanism is used to process the second attention (W q Q,W k K,W v V) Obtain multiple different attention results and concatenate them as the output of the unit multi-head attention mechanism; The residual connection unit is used to add the input and output of the multi-head attention mechanism to strengthen the output features and obtain image features. Among them, W q , W k With W v are the learnable parameters corresponding to the query vector V, key vector K, and value vector V respectively.

5. The method according to claim 1, 2 or 3, characterized in that: The word vector embedding module is a pre-trained word vector model word2vec.

6. The method according to claim 1, 2 or 3, characterized in that: The pre-trained ResNet101 is used to extract the grid-like features of the image and convert them into feature sequences to be input into the viewpoint encoding module.

7. The method according to claim 1, characterized in that The visual language model includes 4 self-attention transformation layers; the visual language model is trained in three ways; the first way is word coverage training; the second way is region replacement training; and the third way is sequence replacement training.

8. The method according to claim 7, characterized in that The trained visual language model is fine-tuned by masking the text in the training sample in the form of a one-way mask, predicting only one word each time the input is made, and when predicting the word, only the word before the word in the text of the current training sample and the images before and after the two changes in the current training sample are input.

9. A server, characterized in that: The method comprises a memory and a processor, wherein the memory stores a computer program, the computer program is configured to be executed by the processor, and the computer program comprises instructions for executing each step in the method according to any one of claims 1 to 8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

Citation Information

Cited By

  • Adaptive balanced perception multi-change subtitle method

    CN119206743A