Text generation, model training method and apparatus

Abstract

The disclosure provides a text generation method and device and a model training method and device, and relates to the technical field of computer vision. The text generation method comprises the following steps: extracting visual features of a to-be-processed image; obtaining related text of the to-be-processed image; encoding the related text of the to-be-processed image to obtain related semantic features of the to-be-processed image; and generating a description text of the to-be-processed image according to the visual features of the to-be-processed image and the related semantic features of the to-be-processed image. Through the above steps, the accuracy of the generated image description text can be improved.

Description

Technical Field

[0001] The present disclosure relates to the field of computer vision technology, and in particular to a text generation and model training method and device. Background Art

[0002] Image captioning is a fundamental topic in the field of computer vision and language. Image captioning refers to automatically generating a description for an image that encompasses the image's semantic content and describes it in an appropriate order.

[0003] Image description mainly uses an encoding-decoding approach. Related technologies often use a pre-trained object detector or classifier as the encoder to extract image features, and a recurrent neural network (RNN) or an attention-based neural network model such as the Transformer as the decoder to decode the extracted image features and generate an image description. Summary of the Invention

[0004] A technical problem to be solved by the present disclosure is to provide a solution that can improve the accuracy of generated image description text.

[0005] According to a first aspect of the present disclosure, a text generation method is proposed, comprising: extracting visual features of an image to be processed; obtaining relevant text of the image to be processed; encoding the relevant text of the image to be processed to obtain relevant semantic features of the image to be processed; and generating a descriptive text of the image to be processed based on the visual features of the image to be processed and the relevant semantic features of the image to be processed.

[0006] In some embodiments, obtaining the relevant text of the image to be processed includes: determining the similarity between the image to be processed and existing text in a training text set; and selecting the relevant text of the image to be processed from the existing text based on the similarity.

[0007] In some embodiments, determining the similarity between the image to be processed and the existing text includes: extracting global features of the image to be processed and global features of the existing text; calculating the cosine similarity between the global features of the image to be processed and the global features of the existing text, and using the cosine similarity as the similarity between the image to be processed and the existing text.

[0008] In some embodiments, encoding the relevant text of the image to be processed to obtain relevant semantic features of the image to be processed includes: determining a relevant word sequence of the image to be processed based on the relevant text of the image to be processed; encoding the relevant word sequence based on a semantic understander to obtain relevant semantic features of the image to be processed, wherein the semantic understander is a trained neural network model based on an attention mechanism.

[0009] In some embodiments, encoding the relevant word sequence of the image to be processed based on the semantic understander to obtain the relevant semantic features of the image to be processed includes: splicing the relevant word sequence of the image to be processed with the additional memory parameters to obtain the input word sequence; context encoding the input word sequence based on the self-attention mechanism to obtain semantic features that integrate context information; with the assistance of the visual features of the image to be processed, semantically enhancing the semantic features that integrate context information based on the cross-attention mechanism to obtain the relevant semantic features of the image to be processed.

[0010] In some embodiments, it also includes: obtaining a relevant word sequence of the sample image; training a neural network model based on the attention mechanism according to the relevant word sequence of the sample image and a preset loss function to obtain a semantic understander, wherein the loss function is constructed with the goal of filtering out semantic words in the relevant word sequence of the sample image that are irrelevant to the sample image and reconstructing missing relevant semantic words.

[0011] In some embodiments, training a neural network model based on an attention mechanism according to a related word sequence of a sample image and a preset loss function includes: concatenating the related word sequence of the sample image with the initialized memory parameters to obtain an input word sequence; inputting the input word sequence into the neural network model based on the attention mechanism to obtain an output semantic feature, wherein the output semantic feature includes a plurality of semantic word features; performing a linear layer projection on the output semantic feature to determine the probability distribution of each semantic word feature in the output semantic feature on the semantic vocabulary; calculating the value of the loss function according to the probability distribution of each semantic word feature in the output semantic feature on the semantic vocabulary; and optimizing the neural network model based on the attention mechanism according to the value of the loss function to obtain a semantic understander.

[0012] In some embodiments, encoding the relevant text of the image to be processed to obtain the relevant semantic features of the image to be processed also includes: determining the position coding in which each semantic word feature participates in the semantic features output by the semantic understander; fusing the semantic word features and the position coding in which they participate to obtain fused semantic word features, and using the whole formed by all the fused semantic word features as the relevant semantic features of the image to be processed.

[0013] In some embodiments, determining the position coding in which each semantic word feature in the semantic features output by the semantic understander participates includes: for each semantic word feature, determining the attention distribution of all position codings of the semantic word feature in the position coding sequence; and aggregating all position codings in the position coding sequence according to the attention distribution to obtain the position coding in which the semantic word feature participates.

[0014] In some embodiments, generating a description text of the image to be processed based on the visual features of the image to be processed and the relevant semantic features of the image to be processed includes: processing the visual features of the image to be processed and the relevant semantic features of the image to be processed based on a text decoder to obtain the description text of the image to be processed, wherein the text decoder is a trained neural network model using an attention mechanism.

[0015] In some embodiments, the visual features of the image to be processed and the relevant semantic features of the image to be processed are processed based on a text decoder to obtain a description text of the image to be processed, including: based on a self-attention mechanism, feature fusion of the text features input at the current decoding moment and the predicted description words of the image to be processed to obtain a first semantic feature; with the assistance of the visual features of the image to be processed and the relevant semantic features of the image to be processed, semantic enhancement is performed on the text features input at the current decoding moment based on a cross-attention mechanism to obtain a second semantic feature; the first semantic feature and the second semantic feature are fused to obtain a fused semantic feature; based on the fused semantic feature, the probability distribution of each semantic word feature in the text feature input at the current decoding moment is determined; based on the probability distribution, the next description word of the image to be processed is determined; after obtaining all the description words of the image to be processed, the ordered sequence composed of all the description words is used as the description text of the image to be processed.

[0016] In some embodiments, extracting visual features of the image to be processed includes: extracting local features and global features of the image to be processed; and determining visual features of the image to be processed based on the local features and global features of the image to be processed.

[0017] In some embodiments, a text-image comparison pre-training model is used to extract local features and global features of the image to be processed.

[0018] In some embodiments, determining the visual features of the image to be processed based on the local features and global features of the image to be processed includes: mapping the local features and global features of the image to be processed to a new feature space, and splicing the mapped local features and global features; encoding the spliced ​​image features based on a visual encoder to obtain the visual features of the image to be processed, wherein the visual encoder is a trained neural network model with multiple layers of coding blocks stacked using a self-attention mechanism.

[0019] In some embodiments, encoding the spliced ​​image features based on a visual encoder to obtain visual features of the image to be processed includes: encoding the spliced ​​image features based on a coding block of a multi-layer self-attention mechanism to obtain local features after multi-layer encoding and global features after multi-layer encoding; splicing and fusing the global features output by the coding block of each layer of the self-attention mechanism to obtain overall global features; splicing the overall global features with the local features after multi-layer encoding to obtain visual features of the image to be processed.

[0020] According to a second aspect of the present disclosure, a model training method is proposed, including: extracting visual features of a sample image; obtaining relevant text of the sample image; encoding the relevant text of the sample image to obtain relevant semantic features of the sample image; and performing supervised training on a neural network model based on an attention mechanism based on the visual features of the sample image and the relevant semantic features of the sample image to obtain a text decoder, wherein the text decoder is used to generate image description text.

[0021] According to a third aspect of the present disclosure, a text generation device is proposed, comprising: a feature extraction module, configured to extract visual features of an image to be processed; a text acquisition module, configured to acquire relevant text of the image to be processed; a text encoding module, configured to encode the relevant text of the image to be processed to obtain relevant semantic features of the image to be processed; and a generation module, configured to generate a description text of the image to be processed based on the visual features of the image to be processed and the relevant semantic features of the image to be processed.

[0022] According to a fourth aspect of the present disclosure, a model training device is proposed, comprising: a feature extraction module configured to extract visual features of a sample image; a text acquisition module configured to acquire relevant text of the sample image; a text encoding module configured to encode the relevant text of the sample image to obtain relevant semantic features of the sample image; and a training module configured to perform supervised training on a neural network model based on an attention mechanism according to the visual features of the sample image and the relevant semantic features of the sample image to obtain a text decoder, wherein the text decoder is used to generate image description text.

[0023] According to a fifth aspect of the present disclosure, a text generation device is further proposed, comprising: a memory; and a processor coupled to the memory, wherein the processor is configured to execute the above-mentioned text generation method based on instructions stored in the memory.

[0024] According to the sixth aspect of the present disclosure, a model training device is also proposed, including: a memory; and a processor coupled to the memory, the processor being configured to execute the model training method as described above based on instructions stored in the memory.

[0025] According to the seventh aspect of the present disclosure, a computer-readable storage medium is also proposed, on which computer program instructions are stored, and when the instructions are executed by a processor, the above-mentioned text generation method or model training method is implemented.

[0026] Compared with the related art, in the embodiment of the present disclosure, by obtaining the relevant text of the image to be processed, the relevant text of the image to be processed is encoded to obtain the relevant semantic features of the image to be processed, and with the assistance of the visual features of the image to be processed and the relevant semantic features of the image to be processed, a description text of the image to be processed is generated, which can improve the accuracy and grammatical consistency of the generated image description text.

[0027] Other features and advantages of the present disclosure will become apparent from the following detailed description of exemplary embodiments of the present disclosure with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments of the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0029] The present disclosure can be more clearly understood from the following detailed description with reference to the accompanying drawings, in which:

[0030] Figure 1 Schematic diagram of a flow chart of a text generation method according to some embodiments of the present disclosure.

[0031] Figure 2a The figure is a flowchart of extracting visual features of an image according to some embodiments of the present disclosure.

[0032] Figure 2b The figure is a flowchart of obtaining relevant text of an image according to some embodiments of the present disclosure.

[0033] Figure 2c The figure is a flowchart of encoding text associated with an image according to some embodiments of the present disclosure.

[0034] Figure 3 The figure is a flowchart of obtaining a semantic understander through training according to some embodiments of the present disclosure.

[0035] Figure 4 The figure is a flowchart of encoding text associated with an image according to some other embodiments of the present disclosure.

[0036] Figure 5 Schematic diagram of a flow chart of a model training method according to some embodiments of the present disclosure.

[0037] Figure 6 Schematic diagram of the structure of a text generation device according to some embodiments of the present disclosure.

[0038] Figure 7 Schematic diagram of the structure of a model training device according to some embodiments of the present disclosure.

[0039] Figure 8 Schematic diagram of the structure of a text generation device or a model training device according to some other embodiments of the present disclosure.

[0040] Figure 9 Schematic diagram of the structure of a computer system according to some embodiments of the present disclosure. DETAILED DESCRIPTION

[0041] Various exemplary embodiments of the present disclosure will now be described in detail with reference to the accompanying drawings. It should be noted that unless otherwise specifically stated, the relative arrangement of components and steps, numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present disclosure.

[0042] At the same time, it should be understood that for the convenience of description, the sizes of the various parts shown in the drawings are not drawn according to the actual proportional relationship.

[0043] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the present disclosure, its application, or uses.

[0044] Technologies, methods and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, such technologies, methods and equipment should be considered part of the authorization specification.

[0045] In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not limiting. Therefore, other examples of the exemplary embodiments may have different values.

[0046] It should be noted that like reference numerals and letters refer to like items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0047] In order to make the objectives, technical solutions and advantages of the present disclosure more clearly understood, the present disclosure is further described in detail below in conjunction with specific embodiments and with reference to the accompanying drawings.

[0048] The text generation methods in related technologies rely too much on the prior knowledge of the language in the training data, resulting in the generated sentences being unable to emphasize the significant semantic information in the image. In addition, object hallucinations sometimes occur, that is, semantic words that do not appear in the image are generated, thus limiting the performance of the image description model.

[0049] Figure 1 FIG. 1 is a flow chart of a text generation method according to some embodiments of the present disclosure. Figure 1As shown, the method includes:

[0050] Step S110: extracting visual features of the image to be processed.

[0051] In step S110 , the image to be processed is encoded to obtain visual features of the image to be processed.

[0052] The visual features of an image can be represented by an ordered set of numerical values ​​with fixed dimensions, such as a vector. For example, the visual features of an image can be represented as in, is the visual feature vector of the image, v1, v2, v n are the different dimensions of the visual feature vector, v1, v2 and v n Can also be a vector.

[0053] Step S130: Obtain relevant text of the image to be processed.

[0054] In some embodiments, the text related to the image to be processed is one or more sentences related to the image to be processed.

[0055] Step S150: Encode the relevant text of the image to be processed to obtain relevant semantic features of the image to be processed.

[0056] In some embodiments, the relevant sentences of the image to be processed are encoded to obtain the features of the relevant sentences. The features of the relevant sentences can be represented by an ordered set of numerical values ​​with a fixed dimension, such as a vector. For example, the feature representation of a relevant sentence is in, is the sentence feature vector, s1, s2, s m is the word vector that constitutes the feature vector of the sentence.

[0057] Step S170: Generate a description text of the image to be processed based on the visual features of the image to be processed and the relevant semantic features of the image to be processed.

[0058] In some embodiments, a text decoder processes the visual features of the image being processed and its related semantic features to generate a descriptive text for the image being processed. The text decoder is a trained neural network model that uses an attention mechanism. For example, a trained neural network stacked with multiple masked multi-head attention-based decoding modules can be used as the text decoder.

[0059] The text decoder generates a description of the image by integrating its visual features and relevant semantic features. In the disclosed embodiments, the text decoder, aided by the relevant semantic and visual features of the image, can obtain accurate visual and semantic information, reducing reliance on prior linguistic knowledge in the training data and improving the accuracy and grammatical consistency of the generated image description.

[0060] In some embodiments, processing the visual features of the image to be processed and the relevant semantic features of the image to be processed based on the text decoder to obtain the description text of the image to be processed includes: steps S171 to S176.

[0061] Step S171: Based on the multi-head self-attention mechanism, the text features input at the current decoding moment and the predicted description words of the image to be processed are fused to obtain the first semantic features.

[0062] In some embodiments, the input text features are obtained by: representing the sentence corresponding to each training sample image I as S = {w0, w1, ..., w T-1}, where w0, w1, w T-1 For the words in the sentence, T represents the length of the sentence. Each word in the sentence S is encoded into a one-hot vector and further encoded to obtain a text feature vector in, is the feature vector of the word in the sentence (referred to as word vector). The text decoder takes the text features as input and predicts the description words of the image to be processed in turn based on the visual features and relevant semantic features of the image to be processed.

[0063] For example, at the t-th decoding moment, the masked multi-head attention layer in the i-th decoding module is based on the previous output hidden state vector Based on the self-attention mechanism, the text features input at the current decoding moment and the word vector of the predicted description word of the image to be processed are fused to obtain the first semantic feature h′ t i Specifically, multi-head self-attention can be performed according to the following formula:

[0064]

[0065] Where h′ t i represents the first semantic feature, represents the hidden state vector of the previous output, Represents the word vector of the predicted description word of the image to be processed, and MultiHead() indicates the execution of multi-head self-attention.

[0066] Step S172: With the assistance of the visual features of the image to be processed and the relevant semantic features of the image to be processed, the text features input at the current decoding moment are semantically enhanced based on the multi-head cross attention mechanism to obtain a second semantic feature.

[0067] In some embodiments, at the t-th decoding time, the multi-head cross attention layer in the i-th decoding module outputs the hidden state vector according to the previous Cross-attention is performed on the visual features of the image to be processed and the relevant semantic features of the image to be processed to obtain the second semantic features Specifically, multi-head cross attention can be performed according to the following formula:

[0068]

[0069] in, Represents the second semantic feature, represents the hidden state vector of the previous output, Represents the visual features of the image to be processed, Represents the relevant semantic features of the image to be processed, Indicates multi-head cross attention to the visual features of the image to be processed, Indicates multi-head cross attention to the relevant semantic features of the processed image.

[0070] Step S173: Fusing the first semantic feature and the second semantic feature to obtain a fused semantic feature.

[0071] In some embodiments, at the t-th decoding moment, the i-th decoding module uses the sigmoid gate function to fuse the first semantic feature and the second semantic feature, thereby obtaining the output of the i-th decoding module

[0072] The sigmoid function, also called the Logistic function, is used for the output of hidden layer neurons. Its value range is (0,1). It can map a real number to the interval (0,1).

[0073] In some embodiments, the output of the i-th decoding module is obtained according to the following formula

[0074]

[0075]

[0076] in, Represents the hidden state vector output by the i-th decoding module, norm() represents the normalization operation, Represents the second semantic feature, h′ t i represents the first semantic feature, Represents the hidden state of the previous output, sigmoid() represents the sigmoid gate function, W g Indicates network parameters.

[0077] At the tth decoding moment, each decoding module performs processing according to steps S171 to S173 in sequence, thereby obtaining the hidden state vector output by the last decoding module That is, the fused semantic features.

[0078] In the embodiment of the present disclosure, the text decoder can further improve the accuracy of the generated image description sentence by cross-attention and feature fusion of the visual features and semantic features of the processed image in the manner shown in steps S171 to S173.

[0079] Step S174: Determine the probability distribution of each semantic word feature in the text feature input at the current decoding moment based on the fused semantic features.

[0080] In some embodiments, the fused semantic features are processed based on a normalized exponential (softmax) function to obtain a probability distribution of each semantic word feature.

[0081] Step S175: Determine the next description word of the image to be processed according to the probability distribution.

[0082] In step S175, the semantic word with the highest probability is used as the next descriptor for the image to be processed. Next, the predicted descriptor for the image to be processed is appended to the end of the previously predicted preceding sentence to form the latest preceding sentence. Steps S171 to S175 are then executed repeatedly until the end marker is predicted.

[0083] Step S176: After obtaining all the descriptive words of the image to be processed, an ordered sequence consisting of all the descriptive words is used as a description text of the image to be processed.

[0084] In the embodiment of the present disclosure, by obtaining the relevant text of the image to be processed, encoding the relevant text of the image to be processed to obtain the relevant semantic features of the image to be processed, and generating a description text of the image to be processed with the assistance of the visual features of the image to be processed and the relevant semantic features of the image to be processed, it is possible to reduce the dependence on prior information of the text and improve the accuracy and grammatical consistency of the generated image description text.

[0085] Figure 2a The figure is a flowchart of extracting visual features of an image according to some embodiments of the present disclosure. Figure 2aThe process shown is an exemplary embodiment of step S110. Figure 2a As shown, the process of extracting visual features of an image in the embodiment of the present disclosure includes:

[0086] Step S111: extracting local features and global features of the image to be processed.

[0087] The global features of an image refer to the overall features of the image, and common global features include color features, texture features, and shape features. The local features of an image refer to features extracted from a local part of the image, such as edges, corners, lines, and regions.

[0088] In some embodiments, a Contrastive Language-Image Pre-Training (CLIP) model is used to extract local features (such as grid features) and global features of the image to be processed. Specifically, the image encoder in the CLIP model is used to extract local features and global features of the image to be processed.

[0089] The CLIP model is a self-supervised model trained on a large-scale dataset of image and text pairs scraped from the web. The features encoded by the image encoder in this model contain richer visual information. Furthermore, because the model doesn't require a predefined label range during pre-training, its ability to understand the semantics of visual features is unconstrained. This improves the visual feature extraction of processed images and, in turn, helps improve the accuracy of the resulting image descriptions.

[0090] In other embodiments, a pre-trained object detector or classifier is used to extract local features and global features of the image.

[0091] Step S112: Determine the visual features of the image to be processed based on the local features and global features of the image to be processed.

[0092] In some embodiments, step S112 includes: mapping the local features and global features of the image to be processed to a new feature space, and splicing the mapped local features and global features; encoding the spliced ​​image features based on a visual encoder to obtain the visual features of the image to be processed, wherein the visual encoder is a trained neural network model with multiple layers of coding blocks stacked using a self-attention mechanism.

[0093] In some embodiments, a fully connected layer is used to map the local features and global features of the processed image to a feature space that is more suitable for the image description text generation task, so as to improve the effect of the final generated image description text.

[0094] In some embodiments, encoding the spliced ​​image features based on the visual encoder includes: encoding the spliced ​​image features based on the encoding blocks of the multi-layer self-attention mechanism to obtain local features after multi-layer encoding and global features after multi-layer encoding; splicing and fusing the global features output by the encoding blocks of each layer of the self-attention mechanism to obtain the overall global features; splicing the overall global features with the local features after multi-layer encoding to obtain the visual features of the image to be processed.

[0095] For example, suppose the concatenated image features are represented as in, represents the global feature vector of the mapped image, N represents the image after mapping i local feature vectors, and the concatenated image features Input a stack of N v The visual encoder of the encoding block based on the self-attention mechanism of the layer performs mutual fusion operation between features to obtain the fused and improved visual features At the same time, the global features output by each layer of the visual encoder are spliced ​​and fused to obtain the overall global features. Finally, the final visual features are combined That is, the visual features of the image to be processed.

[0096] The attention mechanism can be described as a process of mapping a query vector and a series of key-value vectors to an output vector. This output vector is the sum of the weights calculated based on the semantic query vector and the key vector acting on the value vector. For example, the output of the attention layer can be calculated using the following matrix operation formula:

[0097]

[0098] Among them, Attention(Q,K,V) represents the output of the attention layer, Q represents the query matrix, K represents the key matrix, and V represents the value matrix. are predefined parameters.

[0099] In the visual encoder, the query vector, key vector, and value vector of the encoding block based on the self-attention mechanism are all concatenated image features.

[0100] In the embodiment of the present disclosure, by extracting the global features and local features of the image to be processed, and splicing and fusing the global features and local features of the image to obtain the visual features of the image to be processed, richer visual features can be extracted, which helps to improve the accuracy of the subsequent generation of image description text.

[0101] Figure 2b The figure is a flowchart of obtaining relevant text of an image according to some embodiments of the present disclosure. Figure 2b The process shown is an exemplary embodiment of step S130. Figure 2b As shown, the process of obtaining the relevant text of an image in the embodiment of the present disclosure includes:

[0102] Step S131: Determine the similarity between the image to be processed and the existing text in the training text set.

[0103] In some embodiments, the similarity between the image to be processed and the existing text is determined as follows: global features of the image to be processed and global features of the existing text in the training text set are extracted; cosine similarity between the global features of the image to be processed and the global features of the existing text in the training text set is calculated, and the cosine similarity is used as the similarity between the image to be processed and the existing text.

[0104] In some embodiments, sentences in a training set are used as existing texts, global features of the existing texts are extracted in advance based on the CLIP model, and the global features of all existing texts are stored. When processing an image to be processed, the global features of the image to be processed are extracted based on the CLIP model, and then the cosine similarity between the global features of the image to be processed and the global features of the existing texts is calculated and used as the similarity between the image to be processed and the existing texts.

[0105] Step S132: Select relevant text of the image to be processed from the existing texts based on the similarity.

[0106] In some embodiments, K sentences having the highest similarity to the image to be processed are used as relevant texts of the image to be processed, where K is an integer greater than or equal to 1.

[0107] In other embodiments, sentences whose similarity to the image to be processed is greater than or equal to a preset threshold are regarded as relevant texts of the image to be processed.

[0108] In the disclosed embodiment, cross-modal retrieval can efficiently and accurately obtain relevant text of the image to be processed, thereby helping to improve the accuracy of the image description text subsequently generated with the assistance of the relevant text of the image to be processed.

[0109] Figure 2c The figure is a flowchart of encoding text associated with an image according to some embodiments of the present disclosure. Figure 2c The process shown is an exemplary embodiment of step S150. Figure 2c As shown, the process of encoding the text related to the image in the embodiment of the present disclosure includes:

[0110] Step S151: determining a word sequence related to the image to be processed based on the related text of the image to be processed.

[0111] In some embodiments, the text related to the image to be processed is one or more sentences. The sentences are processed by removing stop words, etc., to obtain a word sequence related to the image to be processed.

[0112] Step S152: Encode the relevant word sequence based on the semantic understander to obtain relevant semantic features of the image to be processed.

[0113] The semantic understander is a trained neural network model based on the attention mechanism, for example, a trained neural network model with Ns layers of stacked attention-based transformer encoding blocks.

[0114] In some embodiments, a semantic understander is used to filter out semantic words in the related word sequence that are irrelevant to the image to be processed, while reconstructing more relevant but missing semantic words. In these embodiments, step S152 includes steps a1 to a3.

[0115] Step a1: Concatenate the relevant word sequence of the image to be processed with the additional memory parameters to obtain the input word sequence.

[0116] The word sequence associated with the image to be processed consists of multiple feature vectors of related words. The additional memory parameters are a set of learnable query parameters (i.e., a set of slots). These query parameters are randomly initialized before model training begins and updated as the model iterates. After model training, the final query parameters are saved and used as the memory parameters for concatenating the word sequence associated with the image to be processed.

[0117] Step a2: Contextually encode the input word sequence based on the multi-head self-attention mechanism to obtain semantic features that incorporate contextual information.

[0118] In some embodiments, when context encoding is performed on an input word sequence based on a multi-head self-attention mechanism, the query vector, key vector, and value vector are related word sequences of the image to be processed.

[0119] Step a3: With the assistance of the visual features of the image to be processed, the semantic features integrated with the contextual information are semantically enhanced based on the multi-head cross attention mechanism to obtain the relevant semantic features of the image to be processed.

[0120] In some embodiments, when semantic enhancement is performed on semantic features integrated with contextual information based on a multi-head cross-attention mechanism, the semantic features integrated with contextual information are used as query vectors, and the visual features of the image to be processed are used as key vectors and value vectors.

[0121] In the embodiment of the present disclosure, the semantic understander encodes the relevant word sequence of the image to be processed, which can filter out irrelevant semantic information in the relevant text of the image to be processed and infer the missing semantic information, thereby improving the accuracy of the generated image description text when the image description text is subsequently generated based on the relevant semantic features.

[0122] Figure 3 FIG. 1 is a flow chart of obtaining a semantic understander through training according to some embodiments of the present disclosure. Figure 3 As shown, the process of obtaining a semantic understander through training in the embodiment of the present disclosure includes:

[0123] Step S310: Obtain a related word sequence of a sample image.

[0124] In some embodiments, relevant text of the sample image is obtained through cross-modal retrieval, and a relevant word sequence of the sample image is determined based on the relevant text of the sample image.

[0125] In some embodiments, the text related to the sample image is one or more sentences, and the word sequence related to the sample image is obtained by removing stop words from these related sentences.

[0126] Step S320: According to the relevant word sequence of the sample image and the preset loss function, the neural network model based on the attention mechanism is trained to obtain a semantic understander.

[0127] The loss function is constructed with the goal of filtering out semantic words that are irrelevant to the sample image in the sequence of related words, and reconstructing the missing relevant semantic words. The process of optimizing the model based on this loss function can be formulated as a combination of single-label and multi-label classification problems.

[0128] In some embodiments, step S320 includes: steps b1 to b5.

[0129] Step b1: Concatenate the relevant word sequence of the sample image with the initialized memory parameters to obtain the input word sequence.

[0130] The relevant word sequence for a sample image is composed of feature vectors of multiple relevant words. The memory parameters are a set of learnable query parameters (i.e., a set of slots). These query parameters are randomly initialized before model training begins and updated as the model iterates. After model training is complete, the final query parameters are saved.

[0131] Step b2: Input the input word sequence into the neural network model based on the attention mechanism to obtain output semantic features. The output semantic features include multiple semantic word features.

[0132] Step b3: Perform linear layer projection on the output semantic features to determine the probability distribution of each semantic word feature in the output semantic features on the semantic vocabulary.

[0133] In some embodiments, the semantic vocabulary consists of all semantic words in the training text set plus a flag indicating irrelevant semantic words.

[0134] In this step, the semantic features finally output by the semantic understander are As a condition, a linear predictor is used to estimate the probability distribution of each semantic word feature in the semantic vocabulary to obtain semantic predictions. Specifically, a linear layer can be used to directly project each semantic word feature in the output semantic features into a D-dimensional vector, where D is the size of the predefined semantic vocabulary and each vector is the probability distribution of each semantic word feature in the entire semantic vocabulary.

[0135] Step b4: Calculate the value of the loss function based on the probability distribution of each semantic word feature in the output semantic feature on the semantic vocabulary.

[0136] In some embodiments, the process of filtering out irrelevant semantic words in a related word sequence is regarded as a single-label classification task, and the loss function corresponding to the single-label classification task may adopt a cross-entropy loss function.

[0137] The probability distribution of each semantic word in the related word sequence Afterwards, the first loss value is calculated based on the cross entropy loss function:

[0138]

[0139] Among them, L x represents the first loss value, and Represents y i and The cth element of , c represents the category, y i is the true label representation of the i-th semantic word.

[0140] In some embodiments, the process of inferring missing related semantic words is regarded as a multi-label classification task, and the loss function corresponding to the multi-label classification task may adopt an asymmetric loss function.

[0141] After getting the memory parameter eigenvector The corresponding probability distribution After that, the probability distribution is based on the sigmoid activation function Normalize them and then perform maximum pooling on them to get the overall probability distribution of the memory parameter feature vector on the semantic vocabulary Next, the second loss value is calculated based on the asymmetric loss function:

[0142]

[0143] Among them, L m Represents the second loss value, asym represents the asymmetric loss function, y m are the true labels of all missing relevant semantic words.

[0144] After obtaining the first loss value and the second loss value, the value of the total loss function is calculated based on the first loss value and the second loss value. For example, the total loss value is calculated according to the following formula:

[0145] L s =L x +L m

[0146] Among them, L s Indicates the total loss value.

[0147] Step b5: According to the value of the loss function, the neural network model based on the attention mechanism is optimized to obtain the semantic understander.

[0148] In the embodiment of the present disclosure, the performance of the semantic understander can be improved by training the semantic understander through the above steps.

[0149] Figure 4 The figure is a flowchart of encoding text associated with an image according to some other embodiments of the present disclosure. Figure 4 The process shown is another exemplary embodiment of step S150. Figure 4 As shown, the process of encoding the text related to the image in the embodiment of the present disclosure includes:

[0150] Step S151: Determine a word sequence related to the image to be processed based on the related text of the image to be processed.

[0151] In some embodiments, the text related to the image to be processed is one or more sentences. The sentences are processed by removing stop words, etc., to obtain a word sequence related to the image to be processed.

[0152] Step S152: Encode the relevant word sequence based on the semantic understander to output semantic features.

[0153] The semantic understander is a trained neural network model based on the attention mechanism, for example, a trained neural network model with Ns layers of stacked attention-based transformer encoding blocks.

[0154] Step S153: Based on the semantic sorter, the semantic features output by the semantic understander are fused with the position coding features to obtain relevant semantic features of the image to be processed.

[0155] In some embodiments, step S153 includes: step c1 and step c2.

[0156] Step c1: Determine the position encoding in which each semantic word feature participates in the semantic features output by the semantic understander based on the semantic sorter.

[0157] In some embodiments, for each semantic word feature, the attention distribution of all position codes of the semantic word feature in the position code sequence is determined; based on the attention distribution, all position codes in the position code sequence are aggregated to obtain the position codes involved in the semantic word feature. For example, the position codes involved in each semantic word are calculated using the following formula:

[0158]

[0159] Among them, P i Indicates the positional encoding of semantic word participation, p i It can be interpreted as the semantic features of each semantic word A "soft" estimate of the language order.

[0160] Step c2: Fuse the semantic word features and the position codes in which they participate to obtain fused semantic word features, and use the whole formed by all fused semantic word features as the relevant semantic features of the image to be processed.

[0161] For example, the fused semantic word features are obtained through the following formula:

[0162]

[0163] in, Represents the fused semantic word features, Represents semantic word features, p i Positional encoding that represents semantic word participation.

[0164] The whole composed of all the fused semantic word features is regarded as the relevant semantic features of the image to be processed. Used to present a sequence of ordered semantic words.

[0165] In the disclosed embodiments, the semantic understander and semantic sorter processes described above generate more relevant location-aware semantic features. Furthermore, using these more relevant location-aware semantic features as additional linguistic priors can improve the accuracy of generated image descriptions while encouraging the generation of relevant and coherent descriptions, thereby contributing to improved grammatical consistency in generated image descriptions.

[0166] Figure 5 FIG. 1 is a flow chart of a model training method according to some embodiments of the present disclosure. Figure 5 As shown, the model training method of the embodiment of the present disclosure includes:

[0167] Step S510: extracting visual features of the sample image.

[0168] The training dataset includes sample images and texts corresponding to the sample images. In some embodiments, the text corresponding to each sample image I is a sentence, which can be represented as S = {w0, w1, ..., w T-1}, T represents the length of the sentence.

[0169] In some embodiments, step S510 includes: extracting local features and global features of the sample image; and determining visual features of the sample image based on the local features and global features of the sample image.

[0170] In some embodiments, a Contrastive Language-Image Pre-Training (CLIP) model is used to extract local features (such as grid features) and global features of a sample image. Specifically, an image encoder in the CLIP model is used to extract local features and global features of the sample image.

[0171] In other embodiments, a pre-trained object detector or classifier is used to extract local features and global features of the sample image.

[0172] In some embodiments, determining the visual features of the sample image based on the local features and global features of the sample image includes: mapping the local features and global features of the sample image to a new feature space, and splicing the mapped local features and global features; encoding the spliced ​​image features based on a visual encoder to obtain the visual features of the sample image, wherein the visual encoder is a trained neural network model that stacks multiple layers of encoding blocks using a self-attention mechanism.

[0173] Step S530: Obtain relevant text of the sample image.

[0174] In some embodiments, the similarity between the sample image and the text in the training dataset is determined; and based on the similarity, relevant text of the sample image is selected from the text in the training dataset.

[0175] Step S550: Encode the relevant text of the sample image to obtain relevant semantic features of the sample image.

[0176] In some embodiments, a relevant word sequence of the sample image is determined based on the relevant text of the sample image; the relevant word sequence is encoded based on a semantic understander to obtain relevant semantic features of the sample image, wherein the semantic understander is a trained neural network model based on an attention mechanism.

[0177] In other embodiments, a relevant word sequence of the sample image is determined based on the relevant text of the sample image; the relevant word sequence is encoded based on a semantic understander, and the semantic features output by the semantic understander are fused with the position encoding features based on a semantic sorter to obtain the relevant semantic features of the sample image.

[0178] Step S570: Based on the visual features of the sample image and the relevant semantic features of the sample image, supervised training is performed on the neural network model based on the attention mechanism to obtain a text decoder. The text decoder is used to generate image description text.

[0179] In some embodiments, step S570 includes: based on the self-attention mechanism, performing feature fusion on the text features input at the current decoding moment and the description words of the predicted sample image to obtain a first semantic feature; with the assistance of the visual features of the sample image and the relevant semantic features of the sample image, based on the cross-attention mechanism, performing semantic enhancement on the text features input at the current decoding moment to obtain a second semantic feature; fusing the first semantic feature and the second semantic feature to obtain a fused semantic feature; determining the probability distribution of each semantic word feature in the text features input at the current decoding moment based on the fused semantic feature; determining the value of the loss function based on the probability distribution; and training the model based on the value of the loss function to obtain a text decoder.

[0180] In the embodiment of the present disclosure, the above steps can improve the performance of the generated text decoder and enhance the accuracy and grammatical consistency of the image description text generated based on the text decoder.

[0181] Figure 6 FIG. 1 is a schematic diagram of the structure of a text generation device according to some embodiments of the present disclosure. Figure 6 As shown, the text generation device of the embodiment of the present disclosure includes: a feature extraction module 610, a text acquisition module 620, a text encoding module 630, and a generation module 640.

[0182] The feature extraction module 610 is configured to extract visual features of the image to be processed.

[0183] The feature extraction module 610 encodes the image to be processed to obtain visual features of the image to be processed.

[0184] The visual features of an image can be represented by an ordered set of numerical values ​​with fixed dimensions, such as a vector. For example, the visual features of an image can be represented as in, is the visual feature vector of the image, v1, v2, v n are the different dimensions of the visual feature vector, v1, v2 and v n Can also be a vector.

[0185] The text acquisition module 620 is configured to acquire the relevant text of the image to be processed.

[0186] In some embodiments, the text related to the image to be processed is one or more sentences related to the image to be processed.

[0187] The text encoding module 630 is configured to encode the text related to the image to be processed to obtain the relevant semantic features of the image to be processed.

[0188] In some embodiments, the text encoding module 630 encodes the relevant sentences of the image to be processed to obtain the features of the relevant sentences. The features of the relevant sentences can be represented by an ordered set of numerical values ​​with a fixed dimension, such as a vector. For example, the feature representation of a relevant sentence is in, is the sentence feature vector, s1, s2, s m is the word vector that constitutes the feature vector of the sentence.

[0189] The generating module 640 is configured to generate a description text of the image to be processed according to the visual features of the image to be processed and the relevant semantic features of the image to be processed.

[0190] In some embodiments, a text decoder processes the visual features of the image being processed and its related semantic features to generate a descriptive text for the image being processed. The text decoder is a trained neural network model that uses an attention mechanism. For example, a trained neural network stacked with multiple masked multi-head attention-based decoding modules can be used as the text decoder.

[0191] In the embodiments of the present disclosure, with the assistance of relevant semantic features and visual features of the image to be processed, it is possible to reduce the dependence on prior language knowledge in the training data when generating image description text, thereby improving the accuracy and grammatical consistency of the generated image description text.

[0192] Figure 7 Schematic diagram of the structure of a model training device according to some embodiments of the present disclosure. Figure 7 FIG. 1 is a schematic diagram of the structure of a model training device according to some embodiments of the present disclosure. Figure 7 As shown, the model training device of the embodiment of the present disclosure includes: a feature extraction module 710, a text acquisition module 720, a text encoding module 730, and a training module 740.

[0193] The feature extraction module 710 is configured to extract visual features of the sample image.

[0194] The training dataset includes sample images and texts corresponding to the sample images. In some embodiments, the text corresponding to each sample image I is a sentence, which can be represented as S = {w0, w1, ..., w T-1}, T represents the length of the sentence.

[0195] In some embodiments, the feature extraction module 710 extracts local features and global features of the sample image; the feature extraction module 710 determines the visual features of the sample image based on the local features and global features of the sample image.

[0196] In some embodiments, the feature extraction module 710 utilizes a Contrastive Language-Image Pre-Training (CLIP) model to extract local features (e.g., grid features) and global features of the sample image. Specifically, the image encoder in the CLIP model is used to extract local features and global features of the sample image.

[0197] In other embodiments, the feature extraction module 710 uses a pre-trained object detector or classifier to extract local features and global features of the sample image.

[0198] In some embodiments, the feature extraction module 710 determines the visual features of the sample image based on the local features and global features of the sample image, including: mapping the local features and global features of the sample image to a new feature space, and splicing the mapped local features and global features; encoding the spliced ​​image features based on a visual encoder to obtain the visual features of the sample image, wherein the visual encoder is a trained neural network model that stacks multiple layers of encoding blocks using a self-attention mechanism.

[0199] The text acquisition module 720 is configured to acquire the relevant text of the sample image.

[0200] In some embodiments, the text acquisition module 720 determines the similarity between the sample image and the text in the training dataset; the text acquisition module 720 selects relevant text of the sample image from the text in the training dataset based on the similarity.

[0201] The text encoding module 730 is configured to encode the relevant text of the sample image to obtain relevant semantic features of the sample image.

[0202] In some embodiments, the text encoding module 730 determines a relevant word sequence of the sample image based on the relevant text of the sample image; the text encoding module 730 encodes the relevant word sequence based on a semantic understander to obtain relevant semantic features of the sample image, wherein the semantic understander is a trained neural network model based on an attention mechanism.

[0203] In other embodiments, the text encoding module 730 determines the relevant word sequence of the sample image based on the relevant text of the sample image; the text encoding module 730 encodes the relevant word sequence based on the semantic understander; the text encoding module 730 fuses the semantic features output by the semantic understander with the position encoding features based on the semantic sorter to obtain the relevant semantic features of the sample image.

[0204] The training module 740 is configured to perform supervised training on a neural network model based on an attention mechanism based on the visual features of the sample image and the relevant semantic features of the sample image to obtain a text decoder. The text decoder is used to generate image description text.

[0205] In the embodiment of the present disclosure, the above apparatus can improve the performance of the generated text decoder, and enhance the accuracy and grammatical consistency of the image description text generated based on the text decoder.

[0206] Figure 8 is a block diagram illustrating a text generation device or a model training device according to some other embodiments of the present disclosure.

[0207] like Figure 8 As shown, the text generation device 800 or model training device 800 includes a memory 810 and a processor 820 coupled to the memory 810. The memory 810 is used to store instructions for executing the corresponding embodiments of the text generation method. The processor 820 is configured to execute the text generation method or model training method in any of the embodiments of the present disclosure based on the instructions stored in the memory 810.

[0208] Figure 9is a block diagram illustrating a computer system for implementing some embodiments of the present disclosure.

[0209] like Figure 9 As shown, computer system 900 may be implemented as a general-purpose computing device. Computer system 900 includes memory 910, processor 920, and bus 930 that connects various system components.

[0210] The memory 910 may include, for example, a system memory, a non-volatile storage medium, and the like. The system memory may store, for example, an operating system, application programs, a boot loader, and other programs. The system memory may include volatile storage media, such as random access memory (RAM) and / or cache memory. The non-volatile storage medium may store, for example, instructions for executing at least one corresponding embodiment of the text generation method. Non-volatile storage media include, but are not limited to, disk storage, optical storage, flash memory, and the like.

[0211] Processor 920 can be implemented using a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, or discrete hardware components such as discrete gates or transistors. Accordingly, each module, such as the building block and the deviation compensation module, can be implemented by a central processing unit (CPU) executing instructions in memory that execute corresponding steps, or by dedicated circuits that execute corresponding steps.

[0212] The bus 930 may use any of a variety of bus architectures, including, but not limited to, an Industry Standard Architecture (ISA) bus, a Micro Channel Architecture (MCA) bus, and a Peripheral Component Interconnect (PCI) bus.

[0213] The computer system 900 interfaces 940, 950, and 960, as well as the memory 910 and the processor 920, can be connected via a bus 930. The input / output interface 940 provides a connection interface for input / output devices such as a display, mouse, and keyboard. The network interface 950 provides a connection interface for various networked devices. The storage interface 960 provides a connection interface for external storage devices such as floppy disks, USB flash drives, and SD cards.

[0214] Here, various aspects of the present disclosure are described with reference to flowcharts and / or block diagrams of methods, devices, and computer program products according to embodiments of the present disclosure. It should be understood that each block of the flowcharts and / or block diagrams, and combinations of blocks, can be implemented by computer-readable program instructions.

[0215] These computer-readable program instructions may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable device to produce a machine, so that the processor executes the instructions to produce means for implementing the functions specified in one or more blocks in the flowcharts and / or block diagrams.

[0216] These computer-readable program instructions may also be stored in a computer-readable memory, which cause the computer to operate in a specific manner to produce an article of manufacture, including instructions for implementing the functions specified in one or more blocks in the flowcharts and / or block diagrams.

[0217] The present disclosure can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects.

[0218] The text generation and model training methods and devices in the above embodiments can improve the accuracy of the generated image description text.

[0219] The text generation and model training methods and apparatuses disclosed herein have been described in detail. To avoid obscuring the concepts of the present disclosure, some details known in the art have been omitted. Based on the above description, those skilled in the art will fully understand how to implement the technical solutions disclosed herein.

Claims

1. A text generation method, comprising: Extract visual features of the image to be processed; Determining the similarity between the image to be processed and the existing text in the training text set, including: extracting global features of the image to be processed and global features of the existing text; calculating the cosine similarity between the global features of the image to be processed and the global features of the existing text, and using the cosine similarity as the similarity between the image to be processed and the existing text; Selecting relevant text of the image to be processed from the existing texts according to the similarity; Encoding the relevant text of the image to be processed to obtain relevant semantic features of the image to be processed; Based on the text decoder, the visual features of the image to be processed and the relevant semantic features of the image to be processed are processed to obtain a description text of the image to be processed, including: based on the self-attention mechanism, the text features input at the current decoding moment and the predicted description words of the image to be processed are subjected to feature fusion to obtain a first semantic feature; with the assistance of the visual features of the image to be processed and the relevant semantic features of the image to be processed, the text features input at the current decoding moment are subjected to semantic enhancement based on the cross-attention mechanism to obtain a second semantic feature; the first semantic feature and the second semantic feature are fused to obtain a fused semantic feature; according to the fused semantic feature, the probability distribution of each semantic word feature in the text feature input at the current decoding moment is determined; according to the probability distribution, the next description word of the image to be processed is determined; after obtaining all the description words of the image to be processed, the ordered sequence composed of all the description words is used as the description text of the image to be processed, wherein the text decoder is a trained neural network model adopting the attention mechanism.

2. The text generation method according to claim 1, wherein: The step of encoding the relevant text of the image to be processed to obtain relevant semantic features of the image to be processed includes: Determining a related word sequence of the image to be processed according to the related text of the image to be processed; The relevant word sequence is encoded based on a semantic understander to obtain relevant semantic features of the image to be processed, wherein the semantic understander is a trained neural network model based on an attention mechanism.

3. The text generation method according to claim 2, wherein: The encoding of the related word sequence of the image to be processed based on the semantic understander to obtain the related semantic features of the image to be processed includes: concatenating the related word sequence of the image to be processed with the additional memory parameters to obtain an input word sequence; Performing context encoding on the input word sequence based on a self-attention mechanism to obtain semantic features that incorporate context information; With the assistance of the visual features of the image to be processed, the semantic features integrated with the contextual information are semantically enhanced based on the cross-attention mechanism to obtain the relevant semantic features of the image to be processed.

4. The text generation method according to claim 2, further comprising: Obtain relevant word sequences for sample images; According to the relevant word sequence of the sample image and a preset loss function, a neural network model based on the attention mechanism is trained to obtain the semantic understander, wherein the loss function is constructed with the goal of filtering out semantic words that are irrelevant to the sample image in the relevant word sequence of the sample image and reconstructing the missing relevant semantic words.

5. The text generation method according to claim 4, wherein: According to the relevant word sequence of the sample image and the preset loss function, training the neural network model based on the attention mechanism includes: Concatenating the relevant word sequence of the sample image with the initialized memory parameters to obtain an input word sequence; Inputting the input word sequence into a neural network model based on an attention mechanism to obtain output semantic features, wherein the output semantic features include multiple semantic word features; Performing linear layer projection on the output semantic features to determine the probability distribution of each semantic word feature in the output semantic features on a semantic vocabulary; Calculating a value of a loss function according to a probability distribution of each semantic word feature in the output semantic feature on a semantic vocabulary; According to the value of the loss function, the neural network model based on the attention mechanism is optimized to obtain the semantic understander.

6. The text generation method according to claim 2, wherein: The encoding of the relevant text of the image to be processed to obtain relevant semantic features of the image to be processed further comprises: Determine the position encoding in which each semantic word feature participates in the semantic features output by the semantic understander; The semantic word features and the position codes involved are fused to obtain fused semantic word features, and the whole formed by all fused semantic word features is used as the relevant semantic features of the image to be processed.

7. The text generation method according to claim 6, wherein: Determining the position encoding in which each semantic word feature in the semantic features output by the semantic understander participates includes: For each semantic word feature, determining the attention distribution of all position codes of the semantic word feature in the position code sequence; According to the attention distribution, all position codes in the position code sequence are aggregated to obtain position codes in which the semantic word features participate.

8. The text generation method according to claim 6, wherein: The step of extracting visual features of the image to be processed comprises: Extract local features and global features of the image to be processed; Determine the visual features of the image to be processed based on the local features and global features of the image to be processed.

9. The text generation method according to claim 8, wherein: The text-image comparison pre-training model is used to extract local and global features of the image to be processed.

10. The text generation method according to claim 8, wherein determining the visual features of the image to be processed based on the local features and global features of the image to be processed comprises: Mapping the local features and global features of the image to be processed to a new feature space, and splicing the mapped local features and global features; The spliced ​​image features are encoded based on a visual encoder to obtain visual features of the image to be processed, wherein the visual encoder is a trained neural network model that stacks multiple layers of encoding blocks using a self-attention mechanism.

11. The text generation method according to claim 9, wherein: Encoding the spliced ​​image features based on the visual encoder to obtain the visual features of the image to be processed includes: The encoding block based on the multi-layer self-attention mechanism encodes the spliced ​​image features to obtain multi-layer encoded local features and multi-layer encoded global features; The global features output by the encoding blocks of each layer of the self-attention mechanism are spliced ​​and fused to obtain the overall global features; The overall global features are spliced ​​with the local features after the multi-layer encoding to obtain the visual features of the image to be processed.

12. A model training method for use in the text generation method according to any one of claims 1 to 11, comprising: Extract visual features of sample images; Determining the similarity between the sample image and text in the training dataset; Selecting relevant text of the sample image from the texts in the training dataset based on the similarity; Encoding the relevant text of the sample image to obtain relevant semantic features of the sample image; According to the visual features of the sample image and the relevant semantic features of the sample image, a neural network model based on the attention mechanism is supervisedly trained to obtain a text decoder, wherein the text decoder is used to generate image description text.

13. A text generation device, configured to execute the text generation method according to any one of claims 1 to 11, comprising: A feature extraction module is configured to extract visual features of the image to be processed; A text acquisition module is configured to acquire relevant text of the image to be processed; a text encoding module configured to encode the relevant text of the image to be processed to obtain relevant semantic features of the image to be processed; The generating module is configured to generate a description text of the image to be processed according to the visual features of the image to be processed and the relevant semantic features of the image to be processed.

14. A model training device for executing the model training method according to claim 12, comprising: a feature extraction module configured to extract visual features of a sample image; The text acquisition module is configured as Determining the similarity between the sample image and text in the training dataset; Selecting relevant text of the sample image from the texts in the training dataset based on the similarity; a text encoding module configured to encode the relevant text of the sample image to obtain relevant semantic features of the sample image; The training module is configured to perform supervised training on a neural network model based on an attention mechanism according to the visual features of the sample image and the relevant semantic features of the sample image to obtain a text decoder, wherein the text decoder is used to generate image description text.

15. A text generation device comprising: Memory; as well as A processor coupled to the memory, wherein the processor is configured to execute the text generation method according to any one of claims 1 to 11 based on instructions stored in the memory.

16. A model training device comprising: Memory; as well as A processor coupled to the memory, the processor being configured to execute the model training method as claimed in claim 12 based on instructions stored in the memory.

17. A computer-readable storage medium having computer program instructions stored thereon, which, when executed by a processor, implement the text generation method described in any one of claims 1 to 11, or the model training method described in claim 12.

Images