An adaptive balanced perception method for multiple changes in subtitles

Abstract

The present invention relates to the technical field of variable subtitles, and in particular, provides a method for adaptive balanced perception of multiple variable subtitles. The method comprises obtaining visual features of a training set of variable subtitles; constructing a vocabulary and a label file; obtaining image features in the first stage of the encoder; obtaining common features in the second stage of the encoder; extracting difference features of the image in the third stage of the encoder; splicing two sets of image features to construct a comparative representation of the two differences; finally, modeling the overall difference and the original image respectively; in the first stage of the decoder, establishing grammatical correlations for each word in the text data through a text dependency model; and obtaining a set of subtitle information describing the image differences in the second stage of the decoder. In the process of generating variable subtitles, the method ensures the completeness and smoothness of sentences according to grammatical rules and context information, effectively combines visual information and language information, and improves the accuracy of semantic information.

Description

Technical Field

[0001] The present invention relates to the technical field of variable subtitles, and in particular to a method for adaptive balanced perception of multiple variable subtitles. Background Art

[0002] Change captioning aims to describe the semantic changes between a pair of similar images in concise natural language. This task has many practical applications, such as environmental monitoring, pathological changes, and traditional street scene changes. Therefore, describing the differences in positioning in natural language is a very challenging problem in the multimodal field.

[0003] Previous work on solving the problem of changing subtitles mainly compared and processed two images in global and local space, so as to locate the changed area in the encoder. The most intuitive global method is to directly perform a difference, subtracting the two images to obtain the changed features, but usually the two images are not aligned, and the objects are even unclear or blocked; the local method is based on feature matching between the two images, because usually the same areas in similar images account for the majority and are easy to extract. The common method uses cross-attention to extract common features, and then extracts the changed features by difference, but it is easily affected by extreme viewpoint changes. Pseudo-changes in the appearance and position of objects will mask the real changes. The drawbacks of this process result in the acquisition of difference features with noise, which is not conducive to the generation of sentences in the subsequent decoder. Summary of the invention

[0004] In view of this, the present invention provides an adaptive balanced perception multi-variable subtitle method, which is used to improve the accuracy of semantic information while ensuring the complete and smooth sentences in the process of generating variable subtitles.

[0005] In a first aspect, the present invention provides a method for adaptive equalization-aware multi-variable subtitles, the method comprising:

[0006] Step 1: Obtain image data and text data of changing subtitles to obtain visual features of the training set of changing subtitles and ;

[0007] Step 2: Take image data and text data as input and build vocabulary and label files;

[0008] Step 3: Visual features of the training set and The i-th image features and Input to the encoder, in the first stage of the encoder, the image features and After the context fusion module, the feature representation of each pixel is enhanced by utilizing the context information in the horizontal and vertical directions to obtain the image features. and ;

[0009] Step 4: In the second stage of the encoder, image features and After the context fusion module, and Take them as query features and perform two common feature extractions; and As query features, the common features obtained are and ;

[0010] Step 5: In the third stage of the encoder, the common features obtained based on the differential expression are and , extract the difference features of the image; is based on The image differences obtained as query features; is based on The image difference obtained as the query feature; the two sets of image features and Splice and construct a comparative representation of the two differences to obtain ; Finally, the overall difference With the original image and Modeling separately, we get ;

[0011] Step 6: In the first stage of the decoder, the text-dependent model is used to establish grammatical relevance for each word in the text data;

[0012] Step 7: In the second stage of the decoder, the context fusion module combines the processed text information with Put it into the transformer and get a set of subtitle information describing the image differences.

[0013] Optionally, step 1 includes:

[0014] The image data consists of two sets of similar image data, namely and , and Respectively represent the image before and after the i-th change, ; The text data includes the description information of the image and the part of speech of the description information; the two sets of image data, images and sc_images, are divided into training set, validation set and test set in the ratio of 8:1:1. The text information corresponds to the image data and is also divided into three parts: training set, validation set and test set; after data preprocessing of the image data and text data, the visual features are extracted to obtain the visual features of the training set , ;

[0015] The process of extracting visual features is as follows: read images in batches from the specified directory, normalize the input images, scale the pixel values ​​to the range [0, 1] and apply mean and standard deviation normalization, extract features using the pre-trained ResNet101 model, convert to NumPy arrays, and save these features as .npy files.

[0016] Optionally, the step 2 includes:

[0017] Step 21, load the description of the image data and the dependency of the description in the text data and the json file of the divided data set, and then save the name of the image data into the all_imgs file, and obtain the following files: Totalimages, Total captioned images, Total captions, Total train images, Total traincaptions;

[0018] Step 22: Build the required vocabulary dictionary. First, traverse the text sequence and segment it according to the given delimiter, selectively retain and remove the specified symbols, and add special tags at the beginning and end of the list. <start>and <end>; According to each word segmentation result, count the frequency of each word segmentation to update the dictionary; sort the words according to the statistics, set a custom minimum frequency, add values ​​greater than or equal to the minimum frequency to the dictionary, and assign an index to it; build a vocabulary dictionary with the key as the word and the value as its corresponding index. The vocabulary is used for subsequent text processing tasks;

[0019] Step 23: Build the required dependency vocabulary. The head_tags dictionary is used to define dependency tags and their indexes. Each tag represents a different grammatical role. The stoi dictionary is used to store special tags and their indexes. Add dependency tags to the stoi dictionary and assign indexes to each tag. The completed dependency tag vocabulary is in the format of {label: index}. The vocabulary dependency parsing helps the model understand the structure and grammatical relationship of the sentence. By converting tags into indexes, subsequent model input and processing are facilitated.

[0020] Step 24: Use the Encode function to convert the token list into a corresponding index list, and check whether each token exists in the vocabulary. If the token does not exist in the vocabulary, decide whether to use it according to the settings. <unk>Instead; finally, the encoded description, dependency information, and index information are saved to the HDF5 file.

[0021] Optionally, the step 3 includes:

[0022] Step 31: Input image features The dimension is (B, C, H, W), and the feature map obtained by view has a dimension of (B, H×W, C); and then the preprocessed feature map is obtained by nn.Sequential. , and the feature map dimension is , the input feature of nn.Sequential, which contains the fully connected layer, layer standardization, and Dropout linear transformation, is C, and the output feature is ; According to image features Similarly, we get the feature map ;

[0023] Step 32: Preprocess feature map Processed to dimension As input, weighted features are generated through convolution and cosine similarity calculation of the context fusion module; it is divided into five steps:

[0024] (321) Generate query, key, value; As input, query, key, and value are obtained through convolution calculation, and the feature dimensions are ; Then, through dimension transformation, the dimension is converted into ; The number of input channels for convolution calculation is , the number of output channels is , the convolution kernel size is 1;

[0025] (322) Calculate the horizontal and vertical attention scores and apply a negative infinity mask. First, calculate the horizontal and vertical attention scores using the query and key. Calculate the horizontal attention score and obtain the similarity between the query and the key by matrix multiplication and add a negative infinity mask. After the calculation is completed, reshape the result and rearrange the dimensions so that the horizontal and vertical dimensions are (B, H, W, W).

[0026] (323) Merge the attention scores. Merge the horizontal and vertical attention scores through torch.cat and normalize them using the softmax function to obtain the attention weight of each position. The dimension is ;

[0027] (324) Separate the attention results. Separate the merged attention scores into horizontal and vertical attention scores, and adjust the shapes for subsequent processing. The horizontal attention score is converted into a suitable shape through permute and contiguous. The vertical attention score is processed similarly, with a dimension of (B×W, H, W).

[0028] (325) Weighted value calculation: weight the value according to the calculated attention score, multiply the value by the attention score through matrix multiplication to obtain the horizontal and vertical outputs out_H and out_W. After the calculation is completed, reshape the result and rearrange the dimensions to finally obtain and , the dimension is .

[0029] Optionally, step 4 includes:

[0030] Extracting common features in the context fusion module is divided into three steps:

[0031] (41) Feature processing, As a query, Use the layer_module function as the key and value; then perform convolution calculations on (Q, K, V) respectively, and the dimensions of the feature maps obtained are ; and then adjust the dimension to Get a new feature map (Q, K, V); the number of input channels for convolution calculation is , the number of output channels is , the convolution kernel size is 3, the padding is 1, the step size is 1, the bias is False, and the grouping is 512;

[0032] (42) Calculate the attention score, calculate the attention score att_scores = Q× , which is used to measure the similarity or correlation between the query and all keys; in order to avoid the gradient of the softmax function disappearing due to the dot product result being too large, the attention score is scaled , is the dimension of the key, the scores are converted to probability distributions using the softmax function, and dropout is applied;

[0033] (43) Calculate the context representation. According to the weighted value of the attention probability distribution, the context representation is obtained. Then the weight is summed with the query to obtain the common feature. ; Similarly, As a query, As key and value, we get ; Their expressions are:

[0034] ;

[0035] .

[0036] Optionally, step 5 includes:

[0037] Step 51: The process of extracting differential features from differential expressions is as follows: based on the extracted and ,Will Dimensions are converted to ,get , is based on The image difference obtained as the query feature; similarly Dimensions are converted to , can be obtained , is based on The image differences obtained as query features;

[0038] Step 52, data preprocessing before the difference information is passed to the decoder, is divided into three steps:

[0039] (521) Difference feature concatenation: two sets of difference image features and Concatenate on the dim=-1 dimension to get , the dimension is (B, H×W, C); use the fully connected layer to Perform linear transformation, the input feature is C, and the output feature is ; To prevent the model from overfitting, add a Dropout layer, set a dropout rate of 10%, and then use the ReLU activation function;

[0040] (522) Combining input and difference features, and and Connect on dim=1 to form new features and , the dimensions are all (B, C, H, W);

[0041] (523) Convolution calculation, new features and The dimension calculated by convolution is , the input channel is C, and the output channel is , the convolution kernel size is 1, the padding is 0, and then the group normalization layer is used to divide the channels into 32 groups for normalization to obtain the final result .

[0042] Optionally, step 6 includes:

[0043] Step 61. Before establishing the relevance of the text, it is necessary to preprocess the text data. The dimensions of the image description information seq, dependency information dep and mask information mask are all (N, L). First, the image description information in the text data is word-embedded. The vocabulary size is 76, the size of each embedding vector is 300, and the index 0 is specified for padding to obtain a seq with a dimension of (N, L, D). Then, the position information is added to the description information seq to help the model understand the order of elements in the sequence. During initialization, a position encoding tensor of size (128, 500) is created, which is gradually extracted and added to the input seq to keep the shape of the final input unchanged. Finally, a linear transformation is performed through the fully connected layer, with an input feature of D and an output feature of 512, to obtain a seq tensor of dimension (N, L, 512).

[0044] Step 62: The description information seq is processed by self-attention to establish the connection between words, which is divided into four steps:

[0045] (621) Map seq to query, key and value and adjust the dimension. Map the input seq to the query, key and value space. The linear transformation of the fully connected layer is adopted. The input feature is 512, the output feature is 512, and the output dimension is (N, L, 512). Use the dimension transformation method to convert the query, key and value tensors into a format suitable for attention calculation. The method is to extract the first two dimensional information of the tensor, and then add new dimensional information to it, and concatenate the two.

[0046] (622) Calculate the attention score by performing a dot product of the query and the key, i.e., att_scores = Q× , which is used to measure the similarity or correlation between the query and all keys; in order to avoid the gradient of the softmax function disappearing due to the dot product result being too large, the attention score is scaled , is the dimension of the key, and then continue to use the softmax function to convert the score into a probability distribution, and apply dropout to discard some attention weights;

[0047] (623) Context representation calculation, according to the weighted value of the attention probability distribution, the context representation is obtained;

[0048] (624) Reshape adjustment: adjust the shape of the context layer to the form of (N, L, 512) and return the result.

[0049] Optionally, step 7 includes:

[0050] Step 71: Before establishing association with text data, the description information seq of the specific internal correlation is added to the hidden state of the decoder, and then the sum of the two is layered to improve the convergence speed and stability of the model. The resulting seq dimension is (N, L, 512);

[0051] Step 72: Use cross attention to establish the relationship between text and image difference features, which is divided into four steps:

[0052] (721) Map the sequence to a query and Map into keys and values, and adjust the dimensions accordingly; perform linear transformation of the query, key, and value in the fully connected layer, with the input feature being 512 and the output feature also being 512, so the output dimension is (N, L, 512); use the dimension transformation method to convert the query, key, and value tensors into a format suitable for attention calculation, which includes extracting the first two dimensional information of the tensor, adding new dimensional information, and finally performing concatenation processing;

[0053] (722) Calculate the attention score by performing a dot product of the query and the key to get att_scores = Q× , which is used to measure the similarity or correlation between the query and all keys; in order to avoid the gradient disappearance caused by the dot product result being too large, the attention score is scaled, that is , is the dimension of the key; if att_mask is not empty, add it to att_scores, and then use the softmax function to convert the score into a probability distribution. By calculating the average of the attention scores along the second dimension as att_scores1, we get the average attention score of each query for all keys, and apply dropout to the attention score to discard some of the attention weights;

[0054] (723) Calculate the context representation and weight the values ​​according to the attention probability distribution to obtain the context representation att; finally, adjust the shape of the context layer to (N, L, 12), perform a residual connection between att and the hidden state of the decoder, and then perform layer-1 processing on the sum of the two;

[0055] (724) Feature processing: apply linear transformation to att, whose input and output features are both 512, and use GELU activation function for activation; then apply linear transformation to the input again, whose input and output features are both 512, and perform dropout; perform residual connection with the result itself, then perform layer normalization, and finally return the result.

[0056] In a second aspect, an embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium includes a stored program, wherein when the program is running, the computer-readable storage medium is controlled to execute the adaptive equalization-aware multi-variable subtitle method in the first aspect or any possible implementation of the first aspect.

[0057] In a third aspect, an embodiment of the present invention provides an electronic device, comprising: one or more processors; a memory; and one or more computer programs, wherein the one or more computer programs are stored in the memory, and the one or more computer programs include instructions, which, when executed by the device, enable the device to perform the adaptive equalization-aware multi-variable subtitle method in the first aspect or any possible implementation of the first aspect.

[0058] In the technical solution provided by the present invention, the method includes obtaining image data and text data of changing subtitles to obtain training set visual features of changing subtitles; taking the image data and text data as input to construct a vocabulary and a label file; inputting the i-th image feature of the training set visual feature into an encoder, and in the first stage of the encoder, the image feature passes through a context fusion module, and by utilizing the context information in the horizontal and vertical directions, the feature representation of each pixel is enhanced to obtain the image feature. and ; In the second stage of the encoder, image features and After the context fusion module; and As query features, the common features obtained are and ; In the third stage of the encoder, the common features based on differential expression are and , extract the difference features of the image; is based on The image differences obtained as query features; is based on The image difference obtained as the query feature; the two sets of image features and Splice and construct a comparative representation of the two differences to obtain ; Finally, the overall difference With the original image and Modeling separately, we get In the first stage of the decoder, the text dependency model is used to establish grammatical relevance between words in the text data; in the second stage of the decoder, the context fusion module processes the obtained text information and By putting it into the transformer, we can get a set of subtitle information describing the image differences. In the process of generating subtitles, this method ensures the completeness and smoothness of the sentences based on grammatical rules and contextual information, effectively combines visual information and language information, and improves the accuracy of semantic information. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0060] Figure 1 A flowchart of a method for adaptively balancing and perceiving multiple changes in subtitles provided by an embodiment of the present invention;

[0061] Figure 2 A schematic diagram of the architecture of an adaptive balance-aware network provided in an embodiment of the present invention;

[0062] Figure 3 A schematic diagram of a context fusion module provided in an embodiment of the present invention;

[0063] Figure 4 A schematic diagram of a common feature fusion module provided in an embodiment of the present invention;

[0064] Figure 5 A schematic diagram of inputting changing subtitles provided by an embodiment of the present invention;

[0065] Figure 6 A schematic diagram of outputting changing subtitles provided by an embodiment of the present invention;

[0066] Figure 7 A schematic diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0067] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0068] It should be clear that the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0069] The terms used in the embodiments of the present invention are only for the purpose of describing specific embodiments, and are not intended to limit the present invention. The singular forms "a", "said" and "the" used in the embodiments of the present invention are also intended to include plural forms, unless the context clearly indicates other meanings.

[0070] It should be understood that the term "and / or" used in this article is only a description of the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0071] The word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrases "if it is determined" or "if (stated condition or event) is detected" may be interpreted as "when determining" or "in response to determining" or "when detecting (stated condition or event)" or "in response to detecting (stated condition or event)", depending on the context.

[0072] Figure 1 The flowchart of the adaptive balanced perception multi-variable subtitle method provided by the embodiment of the present invention is as follows Figure 1 As shown, the method includes:

[0073] Step 1: Obtain image data and text data of changing subtitles to obtain visual features of the training set of changing subtitles and .

[0074] In the embodiment of the present invention, Figure 2 As shown, step 1 includes:

[0075] The image data consists of two sets of similar image data, namely and , and Respectively represent the image before and after the i-th change, ; The text data includes the description information of the image and the part of speech of the description information; the two sets of image data, images and sc_images, are divided into training set, validation set and test set in the ratio of 8:1:1. The text information corresponds to the image data and is also divided into three parts: training set, validation set and test set; after data preprocessing of the image data and text data, the visual features are extracted to obtain the visual features of the training set , ;

[0076] The process of extracting visual features is as follows: read images in batches from the specified directory, normalize the input images, scale the pixel values ​​to the range [0, 1] and apply mean and standard deviation normalization, extract features using the pre-trained ResNet101 model, convert to NumPy arrays, and save these features as .npy files.

[0077] Step 2: Take image data and text data as input and build vocabulary and label files.

[0078] In the embodiment of the present invention, step 2 includes:

[0079] Step 21, load the description of the image data and the dependency of the description in the text data and the json file of the divided data set, and then save the name of the image data into the all_imgs file, and obtain the following files: Totalimages, Total captioned images, Total captions, Total train images, Total traincaptions;

[0080] Step 22: Build the required vocabulary dictionary. First, traverse the text sequence and segment it according to the given delimiter, selectively retain and remove the specified symbols, and add special tags at the beginning and end of the list. <start>and <end>; According to each word segmentation result, count the frequency of each word segmentation to update the dictionary; sort the words according to the statistics, set a custom minimum frequency, add values ​​greater than or equal to the minimum frequency to the dictionary, and assign an index to it; build a vocabulary dictionary with the key as the word and the value as its corresponding index. The vocabulary is used for subsequent text processing tasks;

[0081] Step 23: Build the required dependency vocabulary. The head_tags dictionary is used to define dependency tags and their indexes. Each tag represents a different grammatical role, such as punctuation (punct), subject (nsubj), direct object (dobj), etc. The stoi dictionary is used to store special tags and their indexes, such as <pad> 、 <unk>etc.; add dependent labels to the stoi dictionary and assign indexes to each label; build a complete dependent label vocabulary in the format of {label: index}; vocabulary dependency parsing helps the model understand the structure and grammatical relationship of sentences; by converting labels into indexes, it facilitates subsequent model input and processing;

[0082] Step 24: Use the Encode function to convert the token list into a corresponding index list, and check whether each token exists in the vocabulary. If the token does not exist in the vocabulary, decide whether to use it according to the settings. <unk>Instead; because the model can usually only process numeric inputs, the text data needs to be converted into the corresponding index representation. Finally, the encoded description, dependency information, and index information are saved to the HDF5 file.

[0083] Step 3: Visual features of the training set and The i-th image features and Input to the encoder, in the first stage of the encoder, the image features and After the context fusion module, the feature representation of each pixel is enhanced by utilizing the context information in the horizontal and vertical directions to obtain the image features. and .

[0084] In the embodiment of the present invention, step 3 includes:

[0085] Step 31: Input image features The dimension is (B, C, H, W), and the feature map obtained by view has a dimension of (B, H×W, C); and then the preprocessed feature map is obtained by nn.Sequential. , and the feature map dimension is , the input feature of nn.Sequential, which contains the fully connected layer, layer standardization, and Dropout linear transformation, is C, and the output feature is ; According to image features Similarly, we get the feature map ;

[0086] Step 32: Preprocess feature map Processed to dimension As input, the weighted features are generated through convolution and cosine similarity calculation of the context fusion module; Figure 3 As shown, it is divided into five steps:

[0087] (321) Generate query, key, value; As input, query, key, and value are obtained through convolution calculation, and the feature dimensions are ; Then, through dimension transformation, the dimension is converted into ; The number of input channels for convolution calculation is , the number of output channels is , the convolution kernel size is 1;

[0088] (322) Calculate the horizontal and vertical attention scores and apply a negative infinity mask. First, calculate the horizontal and vertical attention scores using the query and key. Calculate the horizontal attention score, obtain the similarity between the query and the key through matrix multiplication, and add a negative infinity mask to ensure that certain positions do not affect the attention. After the calculation is completed, reshape the result and rearrange the dimensions so that the horizontal and vertical dimensions are (B, H, W, W).

[0089] (323) Merge the attention scores. Merge the horizontal and vertical attention scores through torch.cat and normalize them using the softmax function (dim=3) to get the attention weight of each position. The dimension is ;

[0090] (324) Separate the attention results and separate the merged attention scores into horizontal and vertical attention, and adjust the shapes for subsequent processing. The horizontal attention is converted into a suitable shape through permute and contiguous, and the vertical attention is processed similarly, with the dimension of (B×W, H, W).

[0091] (325) Weighted value calculation: weight the value according to the calculated attention score, multiply the value by the attention score through matrix multiplication to obtain the horizontal and vertical outputs out_H and out_W. After the calculation is completed, reshape the result and rearrange the dimensions to obtain and , the dimension is .

[0092] Step 4: In the second stage of the encoder, image features and After the context fusion module, and Take them as query features and perform two common feature extractions; and As query features, the common features obtained are and .

[0093] In the embodiment of the present invention, a lot of similar information has been integrated in the second stage of the encoder, and its correlation is made stronger, overcoming the influence of extreme viewpoints, so the extracted common features are highly accurate.

[0094] In the embodiment of the present invention, Figure 4 As shown, step 4 includes:

[0095] Extracting common features in the context fusion module is divided into three steps:

[0096] (41) Feature processing, As a query, Use the layer_module function as the key and value; then perform convolution calculations on (Q, K, V) respectively, and the dimensions of the feature maps obtained are ; and then adjust the dimension to Get a new feature map (Q, K, V); the number of input channels for convolution calculation is , the number of output channels is , the convolution kernel size is 3, the padding is 1, the step size is 1, the bias is False, and the grouping is 512;

[0097] (42) Calculate the attention score, calculate the attention score att_scores = Q× , which is used to measure the similarity or correlation between the query and all keys; in order to avoid the gradient of the softmax function disappearing due to the dot product result being too large, the attention score is scaled , is the dimension of the key, the scores are converted to probability distributions using the softmax function, and dropout is applied;

[0098] (43) Calculate the context representation. According to the weighted value of the attention probability distribution, the context representation is obtained. Then the weight is summed with the query to obtain the common feature. ; Similarly, As a query, As key and value, we get ; Their expressions are:

[0099] ;

[0100] .

[0101] Step 5: In the third stage of the encoder, the common features obtained based on the differential expression are and , extract the difference features of the image; is based on The image differences obtained as query features; is based on The image difference obtained as the query feature; the two sets of image features and Splice and construct a comparative representation of the two differences to obtain ; Finally, the overall difference With the original image and Modeling separately, we get .

[0102] In the embodiment of the present invention, step 5 includes:

[0103] Step 51: The process of extracting differential features from differential expressions is as follows: based on the extracted and ,Will Dimensions are converted to ,get , is based on The image difference obtained as the query feature; similarly Dimensions are converted to , can be obtained , is based on The image differences obtained as query features;

[0104] Step 52, data preprocessing before the difference information is passed to the decoder, is divided into three steps:

[0105] (521) Difference feature concatenation: two sets of difference image features and Concatenate on the dim=-1 dimension to get , the dimension is (B, H×W, C); use the fully connected layer to Perform linear transformation, the input feature is C, and the output feature is ; To prevent the model from overfitting, add a Dropout layer, set a dropout rate of 10%, and use the ReLU activation function so that the model can learn more complex features;

[0106] (522) Combining input and difference features, and and Connect on dim=1 to form new features and , the dimensions are all (B, C, H, W);

[0107] (523) Convolution calculation, new features and The dimension calculated by convolution is , the input channel is C, and the output channel is , the convolution kernel size is 1, the padding is 0, and then the group normalization layer is used to divide the channels into 32 groups for normalization to obtain the final result .

[0108] Step 6: In the first stage of the decoder, the text-dependent model is used to establish grammatical relevance for each word in the text data.

[0109] In the embodiment of the present invention, based on these links, the model can better understand the complex structure in the sentence, thereby identifying which object has changed.

[0110] In the embodiment of the present invention, step 6 includes:

[0111] Step 61. Before establishing the relevance of the text, it is necessary to preprocess the text data. The dimensions of the image description information seq, dependency information dep and mask information mask are all (N, L). First, the image description information in the text data is word-embedded. The vocabulary size is 76, the size of each embedding vector is 300, and index 0 is specified for padding to obtain a seq with a dimension of (N, L, D), such as (128, 24, 300). Then, the position information is added to the description information seq to help the model understand the order of elements in the sequence. During initialization, a position encoding tensor of size (128, 500) is created, which is gradually extracted and added to the input seq to keep the shape of the final input unchanged. Finally, a linear transformation is performed through the fully connected layer, with an input feature of D and an output feature of 512, to obtain a seq tensor of dimension (N, L, 512).

[0112] Step 62: The description information seq is processed by self-attention to establish the connection between words, which is divided into four steps:

[0113] (621) Map seq to query, key and value and adjust the dimension. Map the input seq to the query, key and value space. The linear transformation of the fully connected layer is adopted. The input feature is 512, the output feature is 512, and the output dimension is (N, L, 512). Use the dimension transformation method to convert the query, key and value tensors into a format suitable for attention calculation. The method is to extract the first two dimensional information of the tensor, and then add new dimensional information to it, and concatenate the two.

[0114] (622) Calculate the attention score by performing a dot product of the query and the key, i.e., att_scores = Q× , which is used to measure the similarity or correlation between the query and all keys; in order to avoid the gradient of the softmax function disappearing due to the dot product result being too large, the attention score is scaled , is the dimension of the key, and then continue to use the softmax function to convert the score into a probability distribution, and apply dropout to discard some attention weights;

[0115] (623) Context representation calculation, according to the weighted value of the attention probability distribution, the context representation is obtained;

[0116] (624) Reshape adjustment: adjust the shape of the context layer to the form of (N, L, 512) and return the result.

[0117] Step 7: In the second stage of the decoder, the context fusion module combines the processed text information with Put it into the transformer and get a set of subtitle information describing the image differences.

[0118] In the embodiment of the present invention, step 7 includes:

[0119] Step 71: Before establishing association with text data, the description information seq of the specific internal correlation is added to the hidden state of the decoder, and then the sum of the two is layered to improve the convergence speed and stability of the model. The resulting seq dimension is (N, L, 512);

[0120] Step 72: Use cross attention to establish the relationship between text and image difference features, which is divided into four steps:

[0121] (721) Map the sequence to a query and Map into keys and values, and adjust the dimensions accordingly; perform linear transformation of the query, key, and value in the fully connected layer, with the input feature being 512 and the output feature also being 512, so the output dimension is (N, L, 512); use the dimension transformation method to convert the query, key, and value tensors into a format suitable for attention calculation, which includes extracting the first two dimensional information of the tensor, adding new dimensional information, and finally performing concatenation processing;

[0122] (722) Calculate the attention score by performing a dot product of the query and the key to get att_scores = Q× , which is used to measure the similarity or correlation between the query and all keys; in order to avoid the gradient disappearance caused by the dot product result being too large, the attention score is scaled, that is , is the dimension of the key; if att_mask is not empty, add it to att_scores, and then use the softmax function to convert the score into a probability distribution. By calculating the average of the attention scores along the second dimension as att_scores1, we get the average attention score of each query for all keys, and apply dropout to the attention score to discard some of the attention weights;

[0123] (723) Calculate the context representation and weight the values ​​according to the attention probability distribution to obtain the context representation att; finally, adjust the shape of the context layer to (N, L, 12), perform a residual connection between att and the hidden state of the decoder, and then perform layer-1 processing on the sum of the two;

[0124] (724) Feature processing: apply linear transformation to att, whose input and output features are both 512, and use GELU activation function for activation; then apply linear transformation to the input again, whose input and output features are both 512, and perform dropout; perform residual connection with the result itself, then perform layer normalization, and finally return the result.

[0125] The present invention consists of two parts: an encoder and a decoder. The encoder is divided into four stages. Context Fusion focuses the context on each feature, which helps to focus on inconspicuous local changes and enhance the robustness of the model to noise and interference. In addition, because feature learning is performed at different scales, local and global features complement each other, thereby better capturing information from details to the global and improving the model's perception of various changes in different scenarios. Common feature extraction captures unchanged features by matching a pair of images by similarity features, and then separates them from the original image pair to obtain stability difference features. The decoder is divided into two stages. Text-dependent model establishes directional associations between individual words in text information through grammar. The model can understand the semantics of the text more deeply, and identify verbs and their modifiers in sentences. The model can judge the change tendency of the text. Therefore, in the process of generating subtitles, the model can generate natural and fluent sentences according to grammatical rules and contextual information. Caption generation passes text information with grammatical association and difference feature maps into transformer, effectively combining visual information and language information, thereby generating more accurate and expressive subtitle information.

[0126] In the embodiment of the present invention, Figure 5 and Figure 6 As shown in the figure, the result of the change caption on CLEVR-Change, a total of 40,000 pairs of images were selected, including changes in color, position, material, and complex situations such as different vision, different object density, different occlusion degree, and different light intensity. From the perspective of generating difference positioning maps, the method of the present invention has achieved positioning effects in complex situations, and the method of the present invention can be applied to many aspects, such as: environmental monitoring, pathological changes, satellite image detection, and traditional street scene changes.

[0127] The experimental comparison results of the method of the present invention and other models on CLEVR-Change are shown in Tables 1 to 4. The evaluation indicators are Blue_4, METEOR, ROUGE_L, CIDEr, and SPICE.

[0128] Table 1

[0129] ;

[0130] Table 2

[0131] ;

[0132] Table 3

[0133] ;

[0134] Table 4

[0135] ;

[0136] Table 1 shows the evaluation results of the overall performance. The METEOR, ROUGE_L, and CIDEr indicators are all higher than the DUDA+TIGR, M-VAM, R3Net+SSP, SRDRL+AVS, and IFDC models. The ROUGE_L indicator of the present invention is improved by 0.3 compared to the highest SRDRL+AVS. The CIDEr indicator is improved by 0.6 compared to the highest R3Net+SSP. Tables 2-4 evaluate the attributes of color (C), material (T), addition (A), deletion (D), and movement (M).

[0137] Table 2 shows the evaluation indicators of CIDEr. The evaluation results of the color, material, and deletion attributes of the present invention are higher than those of other mainstream models. The color evaluation result is 1.1 higher than the highest R3Net+SSP, the material evaluation result is 0.8 higher than the highest R3Net+SSP, and the deletion evaluation result is 1.5 higher than the highest SRDRL+AVS.

[0138] Table 3 is the evaluation index of SPICE. The evaluation results of the color and deletion attributes of the present invention are higher than those of other mainstream models. The evaluation result of color is 0.3 higher than the highest SRDRL+AVS, and the evaluation result of deletion is 0.1 higher than the highest SRDRL+AVS.

[0139] Table 4 shows the evaluation index of METEOR. The evaluation results of the color, material and added attributes of the present invention are higher than those of other mainstream models. The evaluation result of color is 0.2 higher than the highest SRDRL+AVS, the evaluation result of material is 0.3 higher than the highest SRDRL+AVS, and the evaluation result of added is 0.1 higher than the highest SRDRL+AVS. The above data further illustrate that the method of the present invention has better performance.

[0140] Each step of the embodiment of the present invention may be performed by an electronic device, which includes but is not limited to a mobile phone, a tablet computer, a portable PC, a desktop computer, etc.

[0141] In the technical solution provided by the present invention, the method includes obtaining image data and text data of changing subtitles to obtain training set visual features of changing subtitles; taking the image data and text data as input to construct a vocabulary and a label file; inputting the i-th image feature of the training set visual feature into an encoder, and in the first stage of the encoder, the image feature passes through a context fusion module, and by utilizing the context information in the horizontal and vertical directions, the feature representation of each pixel is enhanced to obtain the image feature. and ; In the second stage of the encoder, image features and After the context fusion module; and As query features, the common features obtained are and ; In the third stage of the encoder, the common features based on differential expression are and , extract the difference features of the image; is based on The image differences obtained as query features; is based on The image difference obtained as the query feature; the two sets of image features and Splice and construct a comparative representation of the two differences to obtain ; Finally, the overall difference With the original image and Modeling separately, we get In the first stage of the decoder, the text dependency model is used to establish grammatical relevance between words in the text data; in the second stage of the decoder, the context fusion module processes the obtained text information and By putting it into the transformer, we can get a set of subtitle information describing the image differences. In the process of generating subtitles, this method ensures the completeness and smoothness of the sentences based on grammatical rules and contextual information, effectively combines visual information and language information, and improves the accuracy of semantic information.

[0142] An embodiment of the present invention provides a computer-readable storage medium, which includes a stored program. When the program is running, the electronic device where the computer-readable storage medium is located is controlled to execute the embodiment of the above-mentioned adaptive equalization-aware multi-variable subtitle method.

[0143] Figure 7 A schematic diagram of an electronic device provided by an embodiment of the present invention, such as Figure 7 As shown, the electronic device 21 includes: a processor 211, a memory 212, and a computer program 213 stored in the memory 212 and executable on the processor 211. When the computer program 213 is executed by the processor 211, the adaptive equalization-aware multi-variable subtitle method in the embodiment is implemented. To avoid repetition, they are not described one by one here.

[0144] The electronic device 21 includes, but is not limited to, a processor 211 and a memory 212. Those skilled in the art will appreciate that Figure 7 It is only an example of the electronic device 21 and does not constitute a limitation of the electronic device 21. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the electronic device may also include input and output devices, network access devices, buses, etc.

[0145] The processor 211 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0146] The memory 212 may be an internal storage unit of the electronic device 21, such as a hard disk or memory of the electronic device 21. The memory 212 may also be an external storage device of the electronic device 21, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card (FlashCard), etc. equipped on the electronic device 21. Further, the memory 212 may also include both an internal storage unit of the electronic device 21 and an external storage device. The memory 212 is used to store computer programs and other programs and data required by network devices. The memory 212 may also be used to temporarily store data that has been output or is to be output.

[0147] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0148] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.< / unk> < / unk> < / pad> < / end> < / start> < / unk> < / end> < / start>

Claims

1. A method for adaptive balanced perception of multiple changes in subtitles, characterized in that: The method comprises: Step 1: Obtain image data and text data of changing subtitles to obtain visual features of the training set of changing subtitles and ; Step 2: Take image data and text data as input and build vocabulary and label files; Step 3: Visual features of the training set and The i-th image features and Input to the encoder, in the first stage of the encoder, the image features and After the context fusion module, the feature representation of each pixel is enhanced by utilizing the context information in the horizontal and vertical directions to obtain the image features. and ; Step 4: In the second stage of the encoder, image features and After the context fusion module, and Take them as query features and perform two common feature extractions; and As query features, the common features obtained are and ; Step 5: In the third stage of the encoder, the common features obtained based on the differential expression are and , extract the difference features of the image; is based on The image differences obtained as query features; is based on The image difference obtained as the query feature; the two sets of image features and Splice and construct a comparative representation of the two differences to obtain ; Finally, the overall difference With the original image and Modeling separately, we get ; Step 6: In the first stage of the decoder, the text-dependent model is used to establish grammatical relevance for each word in the text data; Step 7: In the second stage of the decoder, the context fusion module combines the processed text information with Put it into the transformer to get a set of subtitle information describing the image differences; The step 3 comprises: Step 31: Input image features The dimension is (B, C, H, W), and the feature map obtained by view has a dimension of (B, H×W, C); and then the preprocessed feature map is obtained by nn.Sequential. , the feature map dimension is , the input feature of nn.Sequential, which contains the fully connected layer, layer standardization, and Dropout linear transformation, is C, and the output feature is ; According to image features Similarly, we get the feature map ; Step 32: Preprocess feature map Processed to dimension As input, weighted features are generated through convolution and cosine similarity calculation of the context fusion module; it is divided into five steps: (321) Generate query, key, value; As input, query, key, and value are obtained through convolution calculation, and the feature dimensions are ; Then, through dimension transformation, the dimension is converted into ; The number of input channels for convolution calculation is , the number of output channels is , the convolution kernel size is 1; (322) Calculate the horizontal and vertical attention scores and apply a negative infinity mask. First, calculate the horizontal and vertical attention scores using the query and key. Calculate the horizontal attention score and obtain the similarity between the query and the key by matrix multiplication and add a negative infinity mask. After the calculation is completed, reshape the result and rearrange the dimensions so that the horizontal and vertical dimensions are (B, H, W, W). (323) Merge the attention scores. Merge the horizontal and vertical attention scores through torch.cat and normalize them using the softmax function to obtain the attention weight of each position. The dimension is ; (324) Separate the attention results. Separate the merged attention scores into horizontal and vertical attention scores, and adjust the shapes for subsequent processing. The horizontal attention score is converted into a suitable shape through permute and contiguous. The vertical attention score is processed similarly, with a dimension of (B×W, H, W). (325) Weighted value calculation: weight the value according to the calculated attention score, multiply the value by the attention score through matrix multiplication to obtain the horizontal and vertical outputs out_H and out_W. After the calculation is completed, reshape the result and rearrange the dimensions to finally obtain and , the dimension is ; The step 4 comprises: Extracting common features in the context fusion module is divided into three steps: (41) Feature processing, As a query, Use the layer_module function as the key and value; then perform convolution calculations on (Q, K, V) respectively, and the dimensions of the feature maps obtained are ; and then adjust the dimension to Get a new feature map (Q, K, V); the number of input channels for convolution calculation is , the number of output channels is , the convolution kernel size is 3, the padding is 1, the step size is 1, the bias is False, and the grouping is 512; (42) Calculate the attention score, calculate the attention score att_scores = Q× , which is used to measure the similarity or correlation between the query and all keys; in order to avoid the gradient of the softmax function disappearing due to the dot product result being too large, the attention score is scaled , is the dimension of the key, the scores are converted to probability distributions using the softmax function, and dropout is applied; (43) Calculate the context representation. According to the weighted value of the attention probability distribution, the context representation is obtained. Then the weight is summed with the query to obtain the common feature. ; Similarly, As a query, As key and value, we get ; Their expressions are: ； 。 2. The method according to claim 1, characterized in that The step 1 comprises: The image data consists of two sets of similar image data, namely and , and Respectively represent the image before and after the i-th change, ; The text data includes the description information of the image and the part of speech of the description information; The two sets of image data, images and sc_images, are divided into training set, validation set and test set in the ratio of 8:1:

1. The text information corresponds to the image data and is also divided into three parts: training set, validation set and test set; After data preprocessing of the image data and text data, the visual features are extracted to obtain the visual features of the training set , ; The process of extracting visual features is as follows: read images in batches from the specified directory, normalize the input images, scale the pixel values ​​to the range [0, 1] and apply mean and standard deviation normalization, extract features using the pre-trained ResNet101 model, convert to NumPy arrays, and save these features as .npy files.

3. The method according to claim 1, characterized in that The step 2 comprises: Step 21, load the description of the image data and the dependency of the description in the text data and the json file of the divided data set, and then save the name of the image data into the all_imgs file, and obtain the following files: Totalimages, Total captioned images, Total captions, Total train images, Total traincaptions; Step 22: Build the required vocabulary dictionary. First, traverse the text sequence and segment it according to the given delimiter, selectively retain and remove the specified symbols, and add special tags at the beginning and end of the list. <start>and <end> ; According to each word segmentation result, count the frequency of each word segmentation to update the dictionary; sort the words according to the statistics, set a custom minimum frequency, add values ​​greater than or equal to the minimum frequency to the dictionary, and assign an index to it; build a vocabulary dictionary with the key as the word and the value as its corresponding index. The vocabulary is used for subsequent text processing tasks;< / end> < / start> Step 23: Build the required dependency vocabulary. The head_tags dictionary is used to define dependency tags and their indexes. Each tag represents a different grammatical role. The stoi dictionary is used to store special tags and their indexes. Add dependency tags to the stoi dictionary and assign indexes to each tag. The completed dependency tag vocabulary is in the format of {label: index}. The vocabulary dependency parsing helps the model understand the structure and grammatical relationship of the sentence. By converting tags into indexes, subsequent model input and processing are facilitated. Step 24: Use the Encode function to convert the token list into a corresponding index list, and check whether each token exists in the vocabulary. If the token does not exist in the vocabulary, decide whether to use it according to the settings. <unk>Instead; finally, the encoded description, dependency information, and index information are saved to the HDF5 file.< / unk> 4. The method according to claim 1, characterized in that The step 5 comprises: Step 51: The process of extracting differential features from differential expressions is as follows: based on the extracted and ,Will Dimensions are converted to ,get , is based on The image difference obtained as the query feature; similarly Dimensions are converted to , can be obtained , is based on The image differences obtained as query features; Step 52, data preprocessing before the difference information is passed to the decoder, is divided into three steps: (521) Difference feature concatenation: two sets of difference image features and Concatenate on the dim=-1 dimension to get , the dimension is (B, H×W, C); use the fully connected layer to Perform linear transformation, the input feature is C, and the output feature is ; To prevent the model from overfitting, add a Dropout layer, set a dropout rate of 10%, and then use the ReLU activation function; (522) Combining input and difference features, and and Connect on dim=1 to form new features and , the dimensions are all (B, C, H, W); (523) Convolution calculation, new features and The dimension calculated by convolution is , the input channel is C, and the output channel is , the convolution kernel size is 1, the padding is 0, and then the group normalization layer is used to divide the channels into 32 groups for normalization to obtain the final result .

5. The method according to claim 1, characterized in that The step 6 comprises: Step 61. Before establishing the relevance of the text, it is necessary to preprocess the text data. The dimensions of the image description information seq, dependency information dep and mask information mask are all (N, L). First, the image description information in the text data is word-embedded. The vocabulary size is 76, the size of each embedding vector is 300, and the index 0 is specified for padding to obtain a seq with a dimension of (N, L, D). Then, the position information is added to the description information seq to help the model understand the order of elements in the sequence. During initialization, a position encoding tensor of size (128, 500) is created, which is gradually extracted and added to the input seq to keep the shape of the final input unchanged. Finally, a linear transformation is performed through the fully connected layer, with an input feature of D and an output feature of 512, to obtain a seq tensor of dimension (N, L, 512). Step 62: The description information seq is processed by self-attention to establish the connection between words, which is divided into four steps: (621) Map seq to query, key and value and adjust the dimension. Map the input seq to the query, key and value space. The linear transformation of the fully connected layer is adopted. The input feature is 512, the output feature is 512, and the output dimension is (N, L, 512). Use the dimension transformation method to convert the query, key and value tensors into a format suitable for attention calculation. The method is to extract the first two dimensional information of the tensor, and then add new dimensional information to it, and concatenate the two. (622) Calculate the attention score by performing a dot product of the query and the key, i.e., att_scores = Q× , which is used to measure the similarity or correlation between the query and all keys; in order to avoid the gradient of the softmax function disappearing due to the dot product result being too large, the attention score is scaled , is the dimension of the key, and then continue to use the softmax function to convert the score into a probability distribution, and apply dropout to discard some attention weights; (623) Context representation calculation, according to the weighted value of the attention probability distribution, the context representation is obtained; (624) Reshape adjustment: adjust the shape of the context layer to the form of (N, L, 512) and return the result.

6. The method according to claim 1, characterized in that The step 7 comprises: Step 71: Before establishing association with text data, the description information seq of the specific internal correlation is added to the hidden state of the decoder, and then the sum of the two is layered to improve the convergence speed and stability of the model. The resulting seq dimension is (N, L, 512); Step 72: Use cross attention to establish the relationship between text and image difference features, which is divided into four steps: (721) Map the sequence to a query and Map into keys and values, and adjust the dimensions accordingly; perform linear transformation of the query, key, and value in the fully connected layer, with the input feature being 512 and the output feature also being 512, so the output dimension is (N, L, 512); use the dimension transformation method to convert the query, key, and value tensors into a format suitable for attention calculation, which includes extracting the first two dimensional information of the tensor, adding new dimensional information, and finally performing concatenation processing; (722) Calculate the attention score by performing a dot product of the query and the key to get att_scores = Q× , which is used to measure the similarity or correlation between the query and all keys; in order to avoid the gradient disappearance caused by the dot product result being too large, the attention score is scaled, that is , is the dimension of the key; if att_mask is not empty, add it to att_scores, and then use the softmax function to convert the score into a probability distribution. By calculating the average of the attention scores along the second dimension as att_scores1, we get the average attention score of each query for all keys, and apply dropout to the attention score to discard some of the attention weights; (723) Calculate the context representation and weight the values ​​according to the attention probability distribution to obtain the context representation att; finally, adjust the shape of the context layer to (N, L, 12), perform a residual connection between att and the hidden state of the decoder, and then perform layer-1 processing on the sum of the two; (724) Feature processing: apply linear transformation to att, whose input and output features are both 512, and use GELU activation function for activation; then apply linear transformation to the input again, whose input and output features are both 512, and perform dropout; perform residual connection with the result itself, then perform layer normalization, and finally return the result.

7. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored program, wherein when the program is executed, the device where the computer-readable storage medium is located is controlled to execute the adaptive equalization-aware multi-variable subtitle method according to any one of claims 1 to 6.

8. An electronic device, characterized in that: include: one or more processors; Memory; And one or more computer programs, wherein the one or more computer programs are stored in the memory, and the one or more computer programs include instructions, when the instructions are executed by the device, the device performs the adaptive equalization-aware multi-change subtitle method according to any one of claims 1 to 6.

Images