Text generation method and device and related equipment
Through the two-way interactive calculation of image features and non-image features, more accurate predictive text is generated, which solves the problem of low accuracy in image content recognition text in the prior art and improves the recognition effect.
Patent Information
- Application Number
- CN202411052435.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-01
- Publication Date
- 2024-12-03
AI Technical Summary
In the prior art, in image content recognition, the accuracy of the recognition text is low, and the image content cannot be effectively mined, resulting in inaccurate identification text output.
By encoding the predicted image, image features are acquired, and bidirectional interactive calculations are performed in combination with non-image features (such as text features), target attention information is generated, and finally decoded to generate predicted text.
Improve the accuracy of the predicted text, making it more suitable for the image content, and enhances the understanding and description of the image context.
Smart Images

Figure CN119067178A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the technical field of artificial intelligence, and in particular to a text generation method, apparatus and related equipment. Background Art
[0002] Content recognition of images is a hot topic in the field of computer vision and natural language processing, for example: recognition of comic theme sentences of comic images, where the comic theme sentences can be understood as textual expressions of the content expressed by the comic images.
[0003] In the related art, due to the insufficient degree of mining of image content, the accuracy of the recognized text output by performing content recognition on the image is low. Summary of the invention
[0004] The purpose of the present disclosure is to provide a text generation method, apparatus and related equipment to solve the technical problem of low text recognition accuracy in the related art in image content recognition.
[0005] In a first aspect, the present application provides a text generation method, the method comprising:
[0006] Encode the image to be predicted to obtain image features;
[0007] Calculating a non-image feature associated with the image to be predicted according to the image feature to obtain first attention information, and calculating the image feature according to the non-image feature to obtain second attention information, wherein the non-image feature is used to represent the context of the image to be predicted;
[0008] fusing the first attention information and the second attention information to obtain target attention information;
[0009] The target attention information is decoded to obtain a predicted text, where the predicted text is a text used to describe the image content of the image to be predicted.
[0010] In a second aspect, the present application further provides a text generation device, the device comprising:
[0011] An encoding module, used for encoding the image to be predicted to obtain image features;
[0012] an attention calculation module, configured to calculate a non-image feature associated with the image to be predicted according to the image feature to obtain first attention information, and to calculate the image feature according to the non-image feature to obtain second attention information, wherein the non-image feature is used to represent the context of the image to be predicted;
[0013] an attention fusion module, configured to fuse the first attention information and the second attention information to obtain target attention information;
[0014] A decoding module is used to decode the target attention information to obtain a predicted text, where the predicted text is a text used to describe the image content of the image to be predicted.
[0015] In a third aspect, the present application provides an electronic device, comprising a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the steps of the method described in the first aspect.
[0016] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the method described in the first aspect are implemented.
[0017] In a fifth aspect, the present application provides a computer program product, comprising computer instructions, which, when executed by a processor, implement the steps of the method described in the first aspect.
[0018] In the present application, non-image features are introduced, and by respectively calculating the attention information of image features to non-image features, and calculating the attention information of non-image features to image features, a two-way interaction between image features and non-image features used to describe the image context is completed to obtain more accurate attention information. Decoding is performed based on this, so that the output predicted text can be more in line with the image content of the image to be predicted, and the accuracy of the predicted text is improved. BRIEF DESCRIPTION OF THE DRAWINGS
[0019] Figure 1 It is a flowchart of a text generation method provided by an embodiment of the present disclosure;
[0020] Figure 2 is a schematic diagram of an image to be predicted provided by an embodiment of the present disclosure;
[0021] Figure 3 It is a flowchart of a method for generating comic topic sentences based on multimodal fusion provided by an embodiment of the present disclosure;
[0022] Figure 4 is a structural schematic diagram of a text generation device provided by an embodiment of the present disclosure;
[0023] Figure 5 It is a schematic diagram of an electronic device provided by an embodiment of the present disclosure. DETAILED DESCRIPTION
[0024] The following will be combined with the drawings in the embodiments of the present disclosure to clearly and completely describe the technical solutions in the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, rather than all of the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present disclosure.
[0025] The present disclosure provides a text generation method, such as Figure 1 As shown, the text generation method comprises:
[0026] Step 101: Encode the image to be predicted to obtain image features.
[0027] The method described in the present application can be applied to any scenario where a pair of images are recognized and a text is outputted to describe or summarize the content expressed by the image, for example, in a comic theme sentence recognition scenario. In this case, the image to be predicted can be understood as a comic image of a comic theme sentence to be recognized, and the comic theme sentence can be understood as a descriptive text or a summary text of the comic content depicted by the comic image.
[0028] Alternatively, in the emoticon package production scenario, at this time, the image to be predicted can be understood as an emoticon package image without adding emoticon package text.
[0029] Specifically, encoding the image to be predicted to obtain image features includes: encoding the image to be predicted based on a pre-trained target encoder to obtain the image features.
[0030] In some implementations, when the application scenario is specifically a comic topic sentence recognition scenario, the training process of the target encoder may be:
[0031] Collect multiple unlabeled comic images and form a dataset D;
[0032] The initial encoder is constructed based on the ResNet-50 network structure, and the ImageNet dataset is used to pre-train the initial encoder to initialize the weights and parameters of the initial encoder.
[0033] Dividing the data set D into multiple training subsets, and iteratively training the initial encoder based on the multiple training subsets to obtain a target encoder;
[0034] Among them, when iterating to a training subset mini-batch B={Img1,Img2,...,Img N}, the training process based on the training subset includes:
[0035] For each comic image Img in the training subset mini-batch of the current iterationi Perform data augmentation (i=1, 2, ..., N) to obtain two enhanced views t(Img i ) and t'(Img i );
[0036] The two enhanced views corresponding to the same comic image are input into the encoder of the current iteration respectively, and the enhanced view t(Img i ) corresponds to the enhanced image visual feature z i and the enhanced view t'(Img i ) corresponds to the enhanced image visual feature z' i ;
[0037] Among them, z i =f(t(Img i )),z' i =f(t'(Img i )), f(*) is used to represent the image visual features obtained by performing convolution, pooling and other processing on the input image.
[0038] Calculate z i and z' i The noise contrast estimation (NCE) loss between:
[0039]
[0040] In formula (1), the value range of j is 1 to N, and j is not equal to i.
[0041] Conveniently calculate each comic image in the training subset mini-batch of the current iteration, and refer to the above process to calculate the NCE loss of each comic image, then calculate the mean of multiple NCE losses of multiple comic images in the training subset mini-batch of the current iteration, and use the optimizer to update the network parameters of the encoder of the current iteration based on the mean to obtain an encoder or target encoder for use in the next iteration;
[0042] The above process is repeated until the network parameters of the encoder converge (a plurality of training subsets equally divided into the data set D are all iteratively used, or a mean value calculated in a certain iteration is less than a preset convergence threshold), and the encoder at the time of convergence is determined as the target encoder; wherein the encoder used in the first iteration is the initial encoder.
[0043] The above settings select data augmentation for the comic image and obtain two enhanced views, and calculate the visual feature differences between the two enhanced views after being processed by the encoder, which can simulate the image differences produced in different scenes, so that the target encoder finally trained has better adaptability and generalization ability. In addition, the above self-supervised adaptation settings avoid the interference of human factors, can improve the feature encoding ability of the target encoder, and make the image features encoded by the target encoder more accurate.
[0044] Step 102: Calculate non-image features associated with the image to be predicted based on the image features to obtain first attention information, and calculate the image features based on the non-image features to obtain second attention information.
[0045] The non-image features are used to represent the context of the image to be predicted.
[0046] Exemplarily, the non-characteristic image feature may be a text feature, a video feature, a voice feature, or the like.
[0047] For example, the image to be predicted may be a comic image, and the non-image feature may be a text feature. The text feature may be obtained by encoding the comic text of a comic image previous to the image to be predicted and / or the comic text of the comic image previous to the image to be predicted. The text feature may also be obtained by encoding the comic knowledge base text corresponding to the image to be predicted, or by encoding the text contained in the image to be predicted. The text contained in the image to be predicted may be obtained by performing optical character recognition (OCR) on the image to be predicted.
[0048] In one example, a large-scale Transformer-based language model encoder can be used to obtain more accurate text features by acquiring long-range dependency information of the input text (such as the aforementioned comic text or comic knowledge base text or training text), which can obtain high-quality semantic representation.
[0049] In this example, the Transformer-based large language model encoder needs to perform word embedding and position encoding during the training phase, specifically:
[0050] First, input the training text Tokens i Each word t in (sequence form) number Through the query word embedding matrix W e Mapping to word embedding vector:
[0051] e number =W e ×t number (2)
[0052] Then add a positional code to each position to incorporate the sequence position information:
[0053] x number =e number +W p ×pos(t number )(3)
[0054] Among them, W p is the position encoding matrix, pos(t number ) is t number The position number in the input training text is used to obtain the initial word vector sequence representation: X = [x1, x2, ..., x number ].
[0055] Then, the long-range dependencies between different word units are captured through the multi-head attention mechanism, specifically:
[0056] Calculate the query Q, key K and value V based on X respectively:
[0057] Q=X*W Q ,k=X*W k ,V=X*W v (4)
[0058] In formula (4), W Q is the preset matrix corresponding to the query Q, W k is the preset matrix corresponding to key K, W v is the preset matrix corresponding to the value V;
[0059] According to the result of formula (4), the scaled dot product attention is calculated:
[0060]
[0061] In formula (5), d k is the dimension number of key K, K T is the transposed matrix of key K;
[0062] Use a feed-forward fully connected network to apply position-wise nonlinear transformation to the multi-head attention output:
[0063] FFN(x)=max(0,x*head1+b1)*W2+b2(6)
[0064] In formula (6), b1, W2, and b2 are all preset matrices of the feedforward fully connected network, and max is the activation function used to extract high-order semantic features.
[0065] Residual connections and normalization are used for the output of each layer in the multi-head attention mechanism to maintain the gradient propagation of semantic features and accelerate the convergence of related parameters in the multi-head attention mechanism.
[0066] The multi-head attention mechanism consists of multiple layers of encoding modules. Each layer of encoding modules executes the process from (4) to (6) once. The last layer of encoding modules outputs the context vector [h1,h2,...,h number ];
[0067] Concatenate the 1 to number attention heads corresponding to the input text in the training phase to obtain the text feature representation:
[0068] MultiHead(Q,K,V)=Concat(head1,…,head number )*W o (7)
[0069] In formula (7), W o is the preset matrix.
[0070] For example, if the image feature is set to E i , the non-image feature is E t , then the process of performing cross attention calculation on the non-image features associated with the image to be predicted according to the image features to obtain the first attention information may be:
[0071] First, the non-image feature E t and image features E i Through linear layer mapping, they are fused into the same semantic space:
[0072] E t ′=LayerNorm(E t *W t +b t )(8)
[0073] E i ′=LayerNorm(E i *W i +b i )(9)
[0074] In formulas (8) to (9), W i , W t 、b i 、b t All are pre-set matrices.
[0075] Then calculate the image feature E based on the following formula i For non-image features E t Cross attention C t(i.e. first attention information):
[0076] Q i =E i ′*WQ i K t =V t =E t ′(10)
[0077] C t =Attention(Q i ,K t ,V t )(11)
[0078] Similarly, the non-image feature E is calculated based on the following formula: t For image feature E i Cross attention C i (i.e. second attention information):
[0079] Q t =E t ′*WQ t K i =V i =E i ′(12)
[0080]
[0081] After introducing non-image features, the present application can complete the deep fusion of image dimension and non-image dimension (such as text semantic dimension) by interactively calculating the cross-attention between non-image features and image features, thereby fully mining the image features of the predicted image, which can improve the accuracy of the predicted text obtained by subsequent decoding.
[0082] Step 103: merge the first attention information and the second attention information to obtain target attention information.
[0083] Exemplarily, the first attention information and the second attention information may be fused through a gating mechanism to obtain the target attention information.
[0084] For example, the target attention information may be expressed as:
[0085] C=LayerNorm(C i *W i +C t *W t )(14)
[0086] In formula (14), C refers to the target attention information, C t Refers to the first attention information, Ci Refers to the second attention information.
[0087] Step 104: decode the target attention information to obtain predicted text, where the predicted text is text used to describe the image content of the image to be predicted.
[0088] In the present application, non-image features are introduced, and by respectively calculating the attention information of image features to non-image features, and calculating the attention information of non-image features to image features, a two-way interaction between image features and non-image features used to describe the image context is completed to obtain more accurate and comprehensive attention information. Decoding is performed based on this, so that the output predicted text can fit the image content of the image to be predicted in many aspects, that is, the accuracy of the predicted text is improved.
[0089] It should be noted that the above-mentioned prediction text includes not only semantic text data for representing the semantic content expressed by the image to be predicted, but also graphic text data for describing the image itself of the image to be predicted.
[0090] For example, when the image to be predicted is Figure 2 In the case of the image shown, the predicted image is processed based on the method described in the present application, and the obtained predicted text can be: "It's so funny! Hahaha", and his eyes are narrowed into a line, wherein "It's so funny! Hahaha" can be understood as the aforementioned semantic text data, and his eyes are narrowed into a line can be understood as the aforementioned graphic text data.
[0091] In one embodiment, decoding the target attention information to obtain predicted text includes:
[0092] Acquire a text decoding query vector according to the image feature;
[0093] Performing attention calculation according to the text decoding query vector and the image feature to obtain a conditional visual feature;
[0094] The target attention information is decoded according to the conditional visual features to obtain predicted text.
[0095] Furthermore, obtaining a text decoding query vector according to the image feature includes:
[0096] Pooling the image features to obtain a plurality of pooled visual features, wherein the plurality of pooled visual features correspond one-to-one to a plurality of feature scales, and the pooled visual features are features obtained by pooling the image features according to the corresponding feature scales;
[0097] Mapping each of the pooled visual features according to the self-attention mechanism to obtain detail features and semantic features of each of the pooled visual features;
[0098] The text decoding query vector is obtained according to the detail features and semantic features of each of the pooled visual features.
[0099] In this embodiment, image features are pooled based on multiple pre-set feature scales to obtain pooled visual features of the image features at each feature scale. The multiple pooled visual features form a pyramid feature structure similar to a Feature Pyramid Network (FPN). Then, the detail features and semantic features of each pooled visual feature are extracted through a self-attention mechanism, and a text decoding query vector for decoding the target attention information is formed accordingly, which can make full use of the detail information and semantic information in the image and obtain a more comprehensive and accurate decoding result.
[0100] Exemplarily, multiple pooled visual features can be expressed as:
[0101] {E1,E2,…,E k =FPN(E i )(15)
[0102] In formula (15), E1 to E k represents the multiple pooled visual features, and the subscript of each pooled visual feature indicates the sequence number or serial number in the pooled visual feature. The larger the sequence number / serial number is, the larger or smaller the feature scale corresponding to the pooled visual feature is.
[0103] Among them, the pooled visual features are mapped according to the self-attention mechanism, and the detail features and semantic features of the obtained pooled visual features can be expressed as:
[0104]
[0105] Among them, E j A pooled visual feature representing multiple pooled visual features, Refers to E j The detailed features of Refers to E j semantic features.
[0106] In one embodiment, obtaining the text decoding query vector according to the detail features and semantic features of each of the pooled visual features includes:
[0107] Performing attention calculations on the detail features and semantic features of each of the pooled visual features according to the hidden layer state of the target decoder to obtain detail attention data and semantic attention data of each of the visual features, wherein the target decoder is used to decode the target attention information;
[0108] splicing the detail attention data and the semantic attention data of each of the pooled visual features to obtain spliced data;
[0109] The hidden layer state and the concatenated data are fused to obtain the text decoding query vector.
[0110] Exemplarily, first, according to the hidden layer state h of the target decoder t Calculate the query vector for the decoding preparation phase:
[0111] Q h =W q *h t (18)
[0112] In formula (18), Q h is the query vector in the decoding preparation stage, W q Q h The corresponding preset matrix.
[0113] For each pooled visual feature, Q h Perform cross attention calculations with the detail features and semantic features of each pooled visual feature to obtain the detail attention data and semantic attention data of each visual feature:
[0114]
[0115] In the above formula, we still set E j A pooled visual feature representing multiple pooled visual features, For E j Detailed attention data, For E j Semantic attention data.
[0116] In this application, the detail attention data and semantic attention data of each pooled visual feature are spliced to obtain spliced data including:
[0117] Fusing the detail attention data of each of the pooled visual features to obtain fused detail data;
[0118] Fusing the semantic attention data of each of the pooled visual features to obtain fused semantic data;
[0119] The fused detail data and the fused semantic data are spliced to obtain the spliced data.
[0120] Exemplarily, the spliced data may be expressed as:
[0121] Exemplarily, the text decoding query vector can be expressed as:
[0122] h′ t =LayerNorm(h t +C vis +W c )(twenty one)
[0123] In formula (21), h′ t Refers to the text decoding query vector, W c Refers to the preset matrix corresponding to the fusion stage.
[0124] In one embodiment, decoding the target attention information according to the conditional visual features to obtain predicted text includes:
[0125] Iteratively process the conditional visual features and the target attention information to obtain predicted text;
[0126] The tth iteration in the iterative process includes:
[0127] Performing attention calculation on the hidden layer state of the target decoder at the t-th iteration, the conditional visual features, and the target attention information to obtain a context vector of the t-th iteration, wherein the target decoder is used to decode the target attention information;
[0128] Performing full connection processing and normalization processing in sequence according to the hidden layer state of the t-th iteration and the context vector of the t-th iteration to obtain distribution data of the t-th iteration, wherein the distribution data includes multiple word units and probability distributions of the multiple word units;
[0129] Wherein, t is a positive integer. When t is 1, the hidden layer state of the t-th iteration is determined according to the initial hidden layer state of the target decoder. When t is greater than 1, the hidden layer state of the t+1-th iteration is determined according to the word unit sampling data of the t-th iteration, the hidden layer state of the t-th iteration, and the context vector of the t-th iteration, wherein the word unit sampling data of the t-th iteration is obtained by sampling based on the distribution data of the t-th iteration;
[0130] When the iterative process satisfies the iteration termination condition, the predicted text is obtained according to the distribution data of the last iteration.
[0131] In one example, when t is 1, the hidden layer state of the t-th iteration may be an all-0 vector.
[0132] Among them, the context vector of the tth iteration can be expressed as:
[0133] c t =Attention(S t ,C,E cond )(twenty two)
[0134] In formula (22), c t Refers to the context vector of the tth iteration, S t refers to the hidden layer state of the tth iteration, C refers to the target attention information, E cond Refers to the visual characteristics of the condition.
[0135] The probability distribution of multiple word units in the distribution data of the tth iteration can be expressed as:
[0136] p t =softmax(W o (W s S t +W c c t )) (twenty three)
[0137] In formula (23), P t Refers to the probability distribution sequence of multiple word units in the distribution data of the tth iteration, W o , W s , W c is the preset matrix corresponding to the full connection and normalization processing, for P t Sampling can obtain the word unit sampling data y of the tth iteration t .
[0138] The hidden layer state at the t+1th iteration can be expressed as:
[0139] S t+1 =RNN(S t ,[y t ;c t ])(twenty four)
[0140] In formula (24), S t+1 Refers to the hidden layer state at the t+1th iteration.
[0141] The predicted text may be understood as a sequence text consisting of word unit sampling data of the first iteration to word unit sampling data of the last iteration during the above-mentioned iterative processing.
[0142] Since the above decoding process introduces the conditional visual feature E cond, which can provide key visual clues and detail information for generating topic sentences, thus improving the accuracy of the output predictive text.
[0143] In the application, special symbols can be <bos>As the initial input of the above iterative process, and when the output y t Special terminator <eos>, or when the number of iterations reaches the maximum number, terminate the generation (i.e., end the iteration) and output the entire generated sequence Y = [y1, y2,..., y T , y T refers to the token sampling data output in the last iteration.
[0144] During the training phase of the target decoder, the negative log-likelihood between the generated sequence Y and the reference answer Y' can be minimized, and this is used as the loss function to guide the training optimization of the target decoder:
[0145] Loss = -∑ t logP(y t |y * <t,C)(25)
[0146] In Equation (25), y*<t represents the prefix sequence of the reference answer.
[0147] In practical applications, a coverage penalty loss can be added to prevent the generation of duplicate content, and an n-gram copy loss can also be added to avoid generating semantically incoherent predicted text.
[0148] In some embodiments, the calculating the conditional visual feature based on the text-decoded query vector and the image feature includes:
[0149] Based on the calculating the attention based on the text-decoded query vector and the image feature, obtaining the attention weight of each feature vector in the image feature;
[0150] Performing weighted calculation on each feature vector in the image feature according to the attention weight of each feature vector in the image feature to obtain the conditional visual feature.
[0151] The above settings introduce an attention mechanism, which can adaptively focus on the visual regions related to the current generated prediction text task, avoid the noise that may be introduced by the useless background information in the image to be predicted, can adaptively focus on the visual regions related to the task, calculate the most important conditional visual feature, and make the finally generated prediction text more accurate and reliable.
[0152] Exemplarily, the attention weight of each feature vector in the image feature can be expressed as:
[0153] α j = Attention(Q, E ij , E ij )(26)
[0154] In Equation (26), E ij is the j-th feature vector of the image feature E i and α j is the image feature E i The attention weight of the j-th feature vector of , Q refers to the text decoding query vector.
[0155] The conditional visual features can be expressed as:
[0156] E cond =sum j (α j *E ij )(27)
[0157] In formula (27), E cond is the visual feature of the condition.
[0158] For ease of understanding, the following examples are provided:
[0159] See also Figure 3 ,This application also provides a method for generating comic topic sentences based on multimodal fusion, such as Figure 3 As shown, during the training phase, the method for generating comic topic sentences uses a pre-trained visual encoder to extract visual features E of the comic image used for training. i , and use the pre-trained text encoder to extract the text sentence vector E of the comic text corresponding to the comic image i , where the visual encoder can be understood as the aforementioned target encoder, and the text encoder can be understood as the aforementioned Transformer-based large language model encoder.
[0160] Then, referring to formulas (8)-(9), the visual feature E i and the text sentence vector E i Map them to the same semantic space, and refer to formulas (10)-(13) to calculate the cross attention of the two respectively, and refer to formula (14) to perform weighted fusion of the cross attention of the two through a pre-set gating mechanism. In this process, refer to formula (7) to splice 1 to number attention heads corresponding to the input text in the training stage to obtain text feature representation.
[0161] Then obtain the visual feature E i Pooled visual features at different feature scales, and referring to formulas (16)-(17), self-attention encoding is performed on each pooled visual feature to obtain detail features and semantic features of each pooled visual feature; referring to formulas (18)-(20), the cross attention of the detail features of each pooled visual feature and the cross attention of the semantic features of each pooled visual feature are calculated, the cross attention of the detail features of each pooled visual feature and the cross attention of the semantic features of each pooled visual feature are fused, and the hidden layer state h of the target decoder is updated based on the fused spliced data t .
[0162] At the same time, in order to focus on the part of the visual features related to the predicted text in the decoding stage, we can also refer to formulas (26)-(27) to introduce the text decoding query vector (that is, the hidden layer state of the updated target decoder) to perform conditional attention calculation on the visual features to obtain the conditional visual feature E cond .
[0163] Finally, the conditional visual feature E cond The first hidden state (i.e., hidden layer state) of the target decoder is initialized with the text decoding query vector C (the first hidden state is initialized to a vector of all zeros), and is sent to the decoder loop. It is iterated according to formulas (22)-(24). The word units output from the first iteration to the last iteration in the iterative process are summarized into a sequence, and the sequence is output as the comic theme sentence.
[0164] The above method realizes two-way text-visual interaction through an innovative interactive cross-attention fusion mechanism (i.e., the acquisition of the aforementioned first attention information and second attention information). At the same time, the newly proposed multi-level visual fusion strategy (i.e., the acquisition of multiple pooled visual features) makes full use of the details and semantic information of the image to capture better visual representation, thereby achieving a better fusion effect with the semantic modality, and can output a multimodal fusion representation of image + semantics, so that the final output comic theme sentence not only fits the semantics indicated by the image of the comic image, but also fits the image of the comic image itself.
[0165] It should be noted that the above encoding and decoding process can effectively capture the long-range dependencies between words at different positions through the introduction of the attention mechanism, so that the encoder and decoder can make full use of the global context information for sequence generation, which significantly improves the accuracy of the output comic theme sentence (i.e., predicted text).
[0166] Moreover, the above-mentioned interactive cross-attention fusion mechanism can realize fine-grained multi-way interaction of visual and semantic features in each step during decoding, making full use of the complementary information of the two modalities, improving the fusion effect between the modalities, and also improving the accuracy of the output comic theme sentence (i.e., predicted text).
[0167] The introduction of a multi-level visual fusion strategy can integrate visual knowledge from both global and local perspectives, capture more comprehensive and rich visual semantic information, and also improve the accuracy of the output comic theme sentence (i.e., predicted text).
[0168] See also Figure 4 , Figure 4 is a text generation device provided by an embodiment of the present disclosure, such as Figure 4 As shown, the text generation device 400 includes:
[0169] The encoding module 401 is used to encode the image to be predicted to obtain image features;
[0170] an attention calculation module 402, configured to calculate a non-image feature associated with the image to be predicted according to the image feature to obtain first attention information, and to calculate the image feature according to the non-image feature to obtain second attention information, wherein the non-image feature is used to represent the context of the image to be predicted;
[0171] An attention fusion module 403, configured to fuse the first attention information and the second attention information to obtain target attention information;
[0172] The decoding module 404 is used to decode the target attention information to obtain a predicted text, where the predicted text is a text used to describe the image content of the image to be predicted.
[0173] In one embodiment, the decoding module 404 includes:
[0174] A vector acquisition submodule, used to acquire a text decoding query vector according to the image features;
[0175] A conditional visual feature acquisition submodule, used for performing attention calculation according to the text decoding query vector and the image feature to obtain a conditional visual feature;
[0176] A decoding submodule is used to decode the target attention information according to the conditional visual features to obtain predicted text.
[0177] In one embodiment, the vector acquisition submodule includes:
[0178] A pooling unit, used for pooling the image features to obtain a plurality of pooled visual features, wherein the plurality of pooled visual features correspond to a plurality of feature scales one by one, and the pooled visual features are features obtained by pooling the image features according to corresponding feature scales;
[0179] A mapping unit, used for mapping each of the pooled visual features according to a self-attention mechanism to obtain detail features and semantic features of each of the pooled visual features;
[0180] The vector acquisition unit is used to obtain the text decoding query vector according to the detail features and semantic features of each of the pooled visual features.
[0181] In one embodiment, the vector acquisition unit includes:
[0182] an attention calculation subunit, configured to respectively perform attention calculations on the detail features and semantic features of each of the pooled visual features according to the hidden layer state of the target decoder, to obtain detail attention data and semantic attention data of each of the visual features, wherein the target decoder is configured to decode the target attention information;
[0183] A splicing subunit, used for splicing the detail attention data and the semantic attention data of each of the pooled visual features to obtain spliced data;
[0184] The fusion subunit is used to fuse the hidden layer state and the spliced data to obtain the text decoding query vector.
[0185] In one embodiment, the decoding submodule is specifically used to:
[0186] Iteratively process the conditional visual features and the target attention information to obtain predicted text;
[0187] The tth iteration in the iterative process includes:
[0188] Performing attention calculation on the hidden layer state of the target decoder at the t-th iteration, the conditional visual features, and the target attention information to obtain a context vector of the t-th iteration, wherein the target decoder is used to decode the target attention information;
[0189] Performing full connection processing and normalization processing in sequence according to the hidden layer state of the t-th iteration and the context vector of the t-th iteration to obtain distribution data of the t-th iteration, wherein the distribution data includes multiple word units and probability distributions of the multiple word units;
[0190] Wherein, t is a positive integer. When t is 1, the hidden layer state of the t-th iteration is determined according to the initial hidden layer state of the target decoder. When t is greater than 1, the hidden layer state of the t+1-th iteration is determined according to the word unit sampling data of the t-th iteration, the hidden layer state of the t-th iteration, and the context vector of the t-th iteration, wherein the word unit sampling data of the t-th iteration is obtained by sampling based on the distribution data of the t-th iteration;
[0191] When the iterative process satisfies the iteration termination condition, the predicted text is obtained according to the distribution data of the last iteration.
[0192] In one embodiment, the image to be predicted is a comic image, and the non-image feature includes at least one of the following: a text feature and an audio feature.
[0193] The text generation device 400 provided in the embodiment of the present disclosure can implement each process in the above-mentioned text generation method embodiment, and will not be described again here to avoid repetition.
[0194] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device and a readable storage medium.
[0195] Figure 5 A schematic block diagram of an example electronic device 500 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.
[0196] like Figure 5 As shown, the device 500 includes a computing unit 501, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 502 or a computer program loaded from a storage unit 508 to a random access memory (RAM) 503. In the RAM 503, various programs and data required for the operation of the device 500 can also be stored. The computing unit 501, the ROM 502, and the RAM 503 are connected to each other via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.
[0197] A number of components in the device 500 are connected to the I / O interface 505, including: an input unit 506, such as a keyboard, a mouse, etc.; an output unit 507, such as various types of displays, speakers, etc.; a storage unit 508, such as a disk, an optical disk, etc.; and a communication unit 509, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 509 allows the device 500 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.
[0198] The computing unit 501 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 501 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSP), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 501 performs the various methods and processes described above, such as the text generation method. For example, in some embodiments, the text generation method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 508. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 500 via the ROM 502 and / or the communication unit 509. When the computer program is loaded into the RAM 503 and executed by the computing unit 501, one or more steps of the text generation method described above may be performed. Alternatively, in other embodiments, the computing unit 501 may be configured to execute the text generation method in any other appropriate manner (eg, by means of firmware).
[0199] Various embodiments of the systems and techniques described above herein may be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.
[0200] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.
[0201] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
[0202] As used herein, the term "machine-readable medium" refers to any computer program product, apparatus, and / or device (e.g., disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.
[0203] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).
[0204] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communications network). Examples of communications networks include: a local area network (LAN), a wide area network (WAN), and the Internet.
[0205] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.
[0206] The present application also provides a computer program product, including computer instructions, which, when executed by a processor, implement the above Figure 1 or Figure 3 The various processes of the method embodiment shown can achieve the same technical effect, and will not be described again here to avoid repetition.
[0207] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.< / eos> < / bos>
Claims
1. A text generation method, characterized in that: The method comprises: Encode the image to be predicted to obtain image features; Calculating a non-image feature associated with the image to be predicted according to the image feature to obtain first attention information, and calculating the image feature according to the non-image feature to obtain second attention information, wherein the non-image feature is used to represent the context of the image to be predicted; fusing the first attention information and the second attention information to obtain target attention information; The target attention information is decoded to obtain predicted text, where the predicted text is text used to describe the image content of the image to be predicted.
2. The method according to claim 1, characterized in that The decoding of the target attention information to obtain predicted text includes: Acquire a text decoding query vector according to the image feature; Performing attention calculation according to the text decoding query vector and the image feature to obtain a conditional visual feature; The target attention information is decoded according to the conditional visual features to obtain predicted text.
3. The method according to claim 2, characterized in that The obtaining of a text decoding query vector according to the image feature includes: Pooling the image features to obtain a plurality of pooled visual features, wherein the plurality of pooled visual features correspond one-to-one to a plurality of feature scales, and the pooled visual features are features obtained by pooling the image features according to the corresponding feature scales; Mapping each of the pooled visual features according to the self-attention mechanism to obtain detail features and semantic features of each of the pooled visual features; The text decoding query vector is obtained according to the detail features and semantic features of each of the pooled visual features.
4. The method according to claim 3, characterized in that The step of obtaining the text decoding query vector according to the detail features and semantic features of each of the pooled visual features includes: Performing attention calculations on the detail features and semantic features of each of the pooled visual features according to the hidden layer state of the target decoder to obtain detail attention data and semantic attention data of each of the visual features, wherein the target decoder is used to decode the target attention information; splicing the detail attention data and the semantic attention data of each of the pooled visual features to obtain spliced data; The hidden layer state and the concatenated data are fused to obtain the text decoding query vector.
5. The method according to claim 2, characterized in that: The decoding of the target attention information according to the conditional visual features to obtain predicted text includes: Iteratively process the conditional visual features and the target attention information to obtain predicted text; The tth iteration in the iterative process includes: Performing attention calculation on the hidden layer state of the target decoder at the t-th iteration, the conditional visual features, and the target attention information to obtain a context vector of the t-th iteration, wherein the target decoder is used to decode the target attention information; Performing full connection processing and normalization processing in sequence according to the hidden layer state of the t-th iteration and the context vector of the t-th iteration to obtain distribution data of the t-th iteration, wherein the distribution data includes multiple word units and probability distributions of the multiple word units; Wherein, t is a positive integer. When t is 1, the hidden layer state of the t-th iteration is determined according to the initial hidden layer state of the target decoder. When t is greater than 1, the hidden layer state of the t+1-th iteration is determined according to the word unit sampling data of the t-th iteration, the hidden layer state of the t-th iteration, and the context vector of the t-th iteration, wherein the word unit sampling data of the t-th iteration is obtained by sampling based on the distribution data of the t-th iteration; When the iterative process satisfies the iteration termination condition, the predicted text is obtained according to the distribution data of the last iteration.
6. The method according to any one of claims 1 to 5, characterized in that The image to be predicted is a comic image, and the non-image feature includes at least one of the following: a text feature and an audio feature.
7. A text generation device, characterized in that: The device comprises: An encoding module, used for encoding the image to be predicted to obtain image features; an attention calculation module, configured to calculate a non-image feature associated with the image to be predicted according to the image feature to obtain first attention information, and to calculate the image feature according to the non-image feature to obtain second attention information, wherein the non-image feature is used to represent the context of the image to be predicted; an attention fusion module, configured to fuse the first attention information and the second attention information to obtain target attention information; A decoding module is used to decode the target attention information to obtain a predicted text, where the predicted text is a text used to describe the image content of the image to be predicted.
8. An electronic device, characterized in that: The method comprises a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the computer program implements the steps of the method according to any one of claims 1 to 6 when executed by the processor.
9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.
10. A computer program product, characterized in that The method comprises computer instructions, which, when executed by a processor, implement the steps of the method according to any one of claims 1 to 6.
Citation Information
Cited By
Image description method and device, equipment, storage medium and program product
CN120510484A
