Image description method, device, electronic device and storage medium

Abstract

The present invention provides an image description method, apparatus, electronic device, and storage medium. The image description model employed is constructed based on a Transformer model, replacing the traditional recurrent neural network model. This model can improve the readability of the description text of the target image. Furthermore, the image description model is obtained by first-stage training with a general Chinese corpus and second-stage training with an image description corpus. This not only makes the description text generated by the image description model more fluent and coherent, but also reduces the number of training samples required during the training process, thereby reducing the training cost and labor costs of the image description model.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to an image description method, device, electronic device and storage medium. Background Art

[0002] With the development of automated image processing, how to describe a given target image (Image captioning) and automatically determine the descriptive text of the target image has gradually become the focus of the image processing field.

[0003] Image description usually refers to inputting a target image into a computer, and the computer performs calculations and outputs a descriptive text about the target image. In the prior art, computers usually use an image description model to describe the target image, and the image description model is a recurrent neural network model. First, the image information of the target image is extracted through a convolutional neural network (CNN) model, and then the image information is input into the recurrent neural network model. The image information is processed by the attention layer of the recurrent neural network model, and the image information processed by the attention layer is described by the image description layer of the recurrent neural network model, and the descriptive text corresponding to the target image is output.

[0004] Because in the existing technology, when training a recurrent neural network model, a large number of "image-text" pairs need to be provided as training samples, which will lead to great difficulties in obtaining training samples and increase the training cost of the recurrent neural network model. Summary of the Invention

[0005] The present invention provides an image description method, device, electronic device and storage medium to solve the defects in the prior art.

[0006] In a first aspect, the present invention provides an image description method, comprising:

[0007] Acquire the target image;

[0008] Determining a description text of the target image based on an image description model;

[0009] The image description model is constructed based on the Transformer model and is used to perform text description on the target image; the image description model is obtained by performing first-stage training based on a general Chinese corpus and second-stage training based on an image description corpus.

[0010] In one embodiment, there are multiple description texts; accordingly, the image description method further includes:

[0011] Determining, based on a text error correction model, error correction text embedding vectors corresponding to a plurality of the description texts; the text error correction model is trained based on the general Chinese corpus;

[0012] Based on a clustering algorithm, determining a plurality of groups formed by a plurality of the error correction text embedding vectors, and selecting a target group containing the largest number of the error correction text embedding vectors from the plurality of groups;

[0013] A target error correction text embedding vector closest to the cluster center of the target group is selected from the target group, and the error correction text corresponding to the target error correction text embedding vector is used as the final description text of the target image.

[0014] In one embodiment, determining the error correction text embedding vectors corresponding to the plurality of description texts based on the text error correction model specifically includes:

[0015] Inputting the plurality of description texts into the text error correction model respectively, and obtaining error correction texts corresponding to the plurality of description texts output by the text error correction model;

[0016] The plurality of error-correcting texts are respectively input into the text error-correcting model to obtain error-correcting text embedding vectors corresponding to the plurality of error-correcting texts output by the text error-correcting model.

[0017] In one embodiment, the first stage of training specifically includes:

[0018] performing vectorization processing on the general Chinese corpus in the general Chinese corpus to obtain a vectorized corpus;

[0019] splicing the vectorized corpus in the vectorized corpus and an all-0 matrix having the same size as the image vector of the target image to obtain a spliced ​​corpus;

[0020] Based on the concatenated corpus, the Transformer model is trained unsupervised to obtain a general language model.

[0021] In one embodiment, the first stage training further includes:

[0022] Based on the vectorized corpus in the vectorized corpus, generating a masked corpus corresponding to the vectorized corpus;

[0023] Accordingly, the unsupervised training of the Transformer model based on the concatenated corpus to obtain a general language model specifically includes:

[0024] Based on the concatenated corpus and the masked corpus, the Transformer model is trained unsupervised to obtain a general language model.

[0025] In one embodiment, the image description corpus includes image samples and description corpus samples corresponding to the image samples; accordingly,

[0026] The second phase of training specifically includes:

[0027] Performing feature extraction on the image sample to obtain an image feature vector;

[0028] Determining an image attention vector based on the image feature vector;

[0029] Concatenating the image attention vector with a vectorized result obtained by vectorizing the description corpus sample to obtain a text-image hybrid vector;

[0030] The universal language model is trained based on the text-image mixed vector to obtain the image description model.

[0031] In one embodiment, extracting features from the image sample to obtain an image feature vector specifically includes:

[0032] Performing feature extraction on the image sample to obtain a first candidate image feature vector;

[0033] Performing dimension transformation on the candidate image feature vector to obtain a second candidate image feature vector;

[0034] Perform dimensionality reduction processing on the second candidate image feature vector to obtain the image feature vector.

[0035] In a second aspect, the present invention provides an image description device, comprising:

[0036] An acquisition module, used to acquire a target image;

[0037] A description module, configured to determine a description text of the target image based on an image description model;

[0038] The image description model is constructed based on the Transformer model and is used to perform text description on the target image; the image description model is obtained by performing first-stage training based on a general Chinese corpus and second-stage training based on an image description corpus.

[0039] In a third aspect, the present invention provides an electronic device comprising a memory and a memory storing a computer program, wherein the processor implements the steps of the image description method according to the first aspect when executing the program.

[0040] In a fourth aspect, the present invention provides a processor-readable storage medium, wherein the processor-readable storage medium stores a computer program, and the computer program is used to enable the processor to execute the steps of the image description method described in the first aspect.

[0041] The image description method, device, electronic device, and storage medium provided by the present invention utilize an image description model based on the Transformer model, replacing the traditional recurrent neural network model. This model can improve the readability of the description text of the target image. Furthermore, the image description model is obtained by first-stage training with a general Chinese corpus and second-stage training with an image description corpus. This not only makes the description text generated by the image description model more fluent and coherent, but also reduces the number of training samples required during the training process, thereby reducing the training cost and labor costs of the image description model. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the technical solutions in the present invention or the prior art, a brief introduction is given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0043] Figure 1 1 is a flow chart of the image description method provided by the present invention;

[0044] Figure 2 It is a complete flow chart of the image description method provided by the present invention;

[0045] Figure 3 1 is a flow chart of the first stage of training in the image description method provided by the present invention;

[0046] Figure 4 1 is a flow chart of the second stage training in the image description method provided by the present invention;

[0047] Figure 5 1 is a schematic structural diagram of an image description device provided by the present invention;

[0048] Figure 6 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0049] To make the objectives, technical solutions, and advantages of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the embodiments described are only some of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0050] In the prior art, computers usually use image description models to describe the target image. These image description models are recurrent neural network models such as Long Short-Term Memory (LSTM) and Gated Recurrent Unit (GRU). First, the image information of the target image is extracted through a Convolutional Neural Network (CNN) model. This image information is then input into the recurrent neural network model. The image information is processed by the attention layer of the recurrent neural network model. The image information processed by the attention layer is then described by the image description layer of the recurrent neural network model, and the description text corresponding to the target image is output.

[0051] The current technical bottlenecks can be divided into two aspects: first, there is a certain degree of key information loss when the CNN model extracts image information; second, the text generated by the recurrent neural network model has poor readability and lacks logic; third, when training the image description model, a large number of "image-text" pairs need to be provided as training samples, which will lead to great difficulties in obtaining training samples and increase the training cost of the recurrent neural network model.

[0052] To this end, an embodiment of the present invention provides an image description method, including:

[0053] S1, acquire target image;

[0054] S2, determining a description text of the target image based on an image description model;

[0055] The image description model is constructed based on the Transformer model and is used to perform text description on the target image; the image description model is obtained by performing first-stage training based on a general Chinese corpus and second-stage training based on an image description corpus.

[0056] Specifically, the image description method provided in the embodiment of the present invention is executed by an image description device, which can be configured in a server. The server can be a local server or a cloud server. The local server can specifically be a computer, a tablet computer, a smart phone, etc., and this is not specifically limited in the embodiment of the present invention.

[0057] First, step S1 is executed to obtain a target image. The target image refers to an image for which a corresponding description text needs to be determined, and can be any image selected by a user.

[0058] Next, step S2 is performed to determine the description text of the target image based on the image description model. This image description model can be constructed using a Transformer model. Specifically, the Transformer model is the underlying model of the image description model and can be obtained by training the Transformer model. Because the Transformer model uses an attention mechanism, it can better model time series, thereby making the description text of the target image more accurate, improving the readability of the description text of the target image, and making the description text more logical.

[0059] The Transformer model in this embodiment of the present invention can be implemented using six Transformer layers. The parameters for each Transformer layer include: output dimension, attention heads, point-wise feedforward network dimension, and maximum positional encoding length. The output dimension can be 512, the number of attention heads can be 6, the point-wise feedforward network dimension can be 2048, and the maximum positional encoding length can be 100.

[0060] When training the Transformer model to obtain an image description model, the first phase of training can be performed using a general Chinese corpus, followed by a second phase of training using an image description corpus. The general Chinese corpus contains multiple general Chinese corpora, which can be obtained through various search engines or pre-stored in a database. In this embodiment of the present invention, the general Chinese corpora are described in Chinese. The image description corpus contains multiple image samples and a description corpus sample corresponding to each image sample. The description corpus sample refers to the description text of the corresponding image sample.

[0061] The first stage of training refers to the process of pre-training the Transformer model through a general Chinese corpus. Through the first stage of training, a general language model M1 can be obtained. The general language model can obtain prior language knowledge and output the given corpus word by word, so that the description text obtained by the subsequent image description model is more fluent and smooth. Moreover, since the general Chinese corpus is used for pre-training, the demand for the number of "image-text" pairs can be reduced, and labor costs can be reduced. The second stage of training refers to the training of the general language model obtained by the first stage of training through the image description corpus. Through the second stage of training, the image description model M2 can be obtained. The function of the image description model is to provide a textual description of the target image. The target image can be input into the image description model, and the description text of the target image can be output through the image description model.

[0062] The image description method provided in an embodiment of the present invention first obtains a target image; then, based on an image description model, determines a description text for the target image. The image description model used is constructed based on a Transformer model, replacing a traditional recurrent neural network model, and can improve the readability of the description text for the target image. Moreover, the image description model is obtained by performing the first-stage training using a general Chinese corpus and the second-stage training using an image description corpus. This not only makes the description text obtained by the image description model more fluent and smooth, but also reduces the number of training samples required by the image description model during the training process, thereby reducing the training cost and labor cost of the image description model.

[0063] Based on the above embodiments, in the image description method provided in the embodiments of the present invention, the image description model can combine greedy search, beam search, top k sample, and other statement generation algorithms to determine the description text for the target image, so that at least one description text is obtained. For example, the image description model can obtain one description text, or it can obtain ten description texts from the image description text, and these ten description texts can constitute a text list. The number of description texts obtained by the image description model can be set as needed and is not specifically limited in the embodiments of the present invention.

[0064] Based on the above embodiment, the image description method provided in the embodiment of the present invention further includes:

[0065] Based on a text error correction model, an error correction text embedding vector corresponding to the description text is determined; the text error correction model is trained based on the general Chinese corpus.

[0066] Specifically, in an embodiment of the present invention, when the number of description texts obtained by the image description model is one, the image description method can also determine the correction text embedding vector corresponding to the description text based on the text correction model M3. The text correction model can be constructed based on the Bert model and trained based on the general Chinese corpus, so that the text correction model can have correction capabilities. When training the Bert model, the general Chinese corpus can be used as the training library X1, and each word in each general Chinese corpus in X1 can be replaced with a random Chinese word with a random probability of 10%-30% to obtain the training library X2. Using X2 as training data and X1 as target data, the Bert model is trained to obtain the text correction model M3.

[0067] Through the text error correction model M3, the description text can be corrected and vectorized to obtain the error correction text corresponding to the description text.

[0068] In the embodiment of the present invention, a text error correction model is introduced to correct the description text, so that the obtained error-corrected text is more accurate than the description text, and the readability of the error-corrected text is further improved.

[0069] Based on the above embodiment, in the image description method provided in the embodiment of the present invention, there are multiple description texts; accordingly, the method further includes:

[0070] Determining, based on a text error correction model, error correction text embedding vectors corresponding to a plurality of the description texts; the text error correction model is trained based on the general Chinese corpus;

[0071] Based on a clustering algorithm, determining a plurality of groups formed by a plurality of the error correction text embedding vectors, and selecting a target group containing the largest number of the error correction text embedding vectors from the plurality of groups;

[0072] A target error correction text embedding vector closest to the cluster center of the target group is selected from the target group, and the error correction text corresponding to the target error correction text embedding vector is used as the final description text of the target image.

[0073] Specifically, in an embodiment of the present invention, when the number of description texts obtained by the image description model is multiple, the image description method can also determine the error correction text embedding vectors corresponding to the multiple description texts based on the text error correction model M3. The text error correction model can be constructed based on the Bert model and trained based on the general Chinese corpus, so that the text error correction model can have error correction capabilities. When training the Bert model, the general Chinese corpus can be used as the training library X1, and each word in each general Chinese corpus in X1 can be replaced with a random Chinese word with a random probability of 10%-30% to obtain the training library X2. Using X2 as training data and X1 as target data, the Bert model is trained to obtain the text error correction model M3.

[0074] Through the text error correction model M3, the text list Y1 composed of multiple description texts can be corrected and vectorized to obtain the error correction text embedding vector corresponding to each description text. All the error correction text embedding vectors can form a vector list Y2.

[0075] Then, based on a clustering algorithm, multiple groups formed by the multiple error-correcting text embedding vectors can be determined. The clustering algorithm can use a K-means algorithm to group the multiple error-correcting text embedding vectors, and each resulting group can include at least one error-correcting text embedding vector. When grouping the multiple error-correcting text embedding vectors using the K-means algorithm, a K value used for grouping can be first determined, and then grouping can be performed based on the determined K value, with each group corresponding to a cluster center.

[0076] When determining the K value, assume that the number of groups ranges from 2 to 5, that is, K∈range(2,5). Then, the K-means algorithm is used to calculate the total scores of each group for different K values, and the K value with the highest total score is selected as the K value used for grouping.

[0077] Finally, the target group with the largest number of error correction text embedding vectors is selected from all groups. The target error correction text embedding vector closest to the cluster center of the target group is selected from the target group. The error correction text corresponding to the target error correction text embedding vector is used as the final description text of the target image.

[0078] In an embodiment of the present invention, a text error correction model is introduced to correct the description text, and the final description text is determined in combination with the K-means algorithm. The method of generating multiple description texts is adopted to reduce the risk of the image description model being off-topic, and further optimization is performed through the K-means algorithm to generate the final description text. Compared with the image description model that generates a single description text, the final description text will be closer to the topic and less likely to go off track, thereby improving the accuracy of the image description model.

[0079] Based on the above embodiment, the image description method provided in the embodiment of the present invention, wherein the determining of the error correction text embedding vectors corresponding to the plurality of description texts based on the text error correction model specifically includes:

[0080] Inputting the plurality of description texts into the text error correction model respectively, and obtaining error correction texts corresponding to the plurality of description texts output by the text error correction model;

[0081] The plurality of error-correcting texts are respectively input into the text error-correcting model to obtain error-correcting text embedding vectors corresponding to the plurality of error-correcting texts output by the text error-correcting model.

[0082] Specifically, in an embodiment of the present invention, when determining the correction text embedding vector corresponding to the description text, the text list Y1 composed of multiple description texts can be first input into the text correction model to obtain a text list Y3 composed of correction texts corresponding to multiple description texts output by the text correction model, and then the text list Y3 is input into the text correction model again, and the text correction model performs vectorization processing, and its [CLS] corresponding output is taken as the correction text embedding vector, and then a vector list Y2 composed of correction text embedding vectors corresponding to multiple correction texts can be obtained.

[0083] In the embodiment of the present invention, the text correction model can be used to obtain not only the correction text but also the correction text embedding vector corresponding to the correction text, which facilitates the subsequent determination of the final description text.

[0084] Figure 2 FIG. 1 is a complete flow chart of the image description method provided in an embodiment of the present invention. Figure 2 As shown, the Transformer model is first trained using image and text input to create an image description model. In the first training phase, the image input is a matrix of all zeros, and the text input is a general Chinese corpus from a general Chinese corpus. In the second training phase, the image input is an image sample from the image description corpus, and the text input is a description corpus sample corresponding to the image sample. The image description model is then used to describe the target image, generating a text description of the target image. The text description of the target image is then corrected and vectorized using a text correction model, generating a corrected text embedding vector. Finally, a clustering algorithm is used to select and output the final description of the target image.

[0085] Based on the above embodiment, in the image description method provided in the embodiment of the present invention, the first stage of training specifically includes:

[0086] performing vectorization processing on the general Chinese corpus in the general Chinese corpus to obtain a vectorized corpus;

[0087] splicing the vectorized corpus in the vectorized corpus and an all-0 matrix having the same size as the image vector of the target image to obtain a spliced ​​corpus;

[0088] Based on the concatenated corpus, the Transformer model is trained unsupervised to obtain a general language model.

[0089] Specifically, in an embodiment of the present invention, during the first stage of training, the general Chinese corpus in the general Chinese corpus can be vectorized to obtain a vectorized corpus X3. The vectorized corpus in the vectorized corpus X3 and an all-zero matrix of the same size as the image vector of the target image are then concatenated to obtain a concatenated corpus X4, which contains multiple concatenated corpora.

[0090] Based on the concatenated corpus X4, the Transformer model is trained unsupervised, that is, each concatenated corpus in the concatenated corpus is used as the input of the Transformer model for unsupervised training to obtain a universal language model M1.

[0091] Before concatenation, the vectorized corpus can be input into the trained embedding layer (EmbeddingLayer) to obtain a feature matrix. This feature matrix and the all-zero matrix are then input into the trained concatenation layer (ConcatLayer) for concatenation to obtain the concatenated corpus. The embedding layer can be expressed as EmbeddingLayer[batch_size,100,512], where batch_size represents the number of vectorized corpora that can be processed.

[0092] In the embodiment of the present invention, unsupervised training of the Transformer model is adopted, which can avoid labeling of training samples and save training costs.

[0093] Based on the above embodiment, in the image description method provided in the embodiment of the present invention, the first stage training further includes:

[0094] Based on the vectorized corpus in the vectorized corpus, generating a masked corpus corresponding to the vectorized corpus;

[0095] Accordingly, the unsupervised training of the Transformer model based on the concatenated corpus to obtain a general language model specifically includes:

[0096] Based on the concatenated corpus and the masked corpus, the Transformer model is trained unsupervised to obtain a general language model.

[0097] Specifically, in an embodiment of the present invention, during the first stage of training, a mask corpus corresponding to the vectorized corpus can also be generated based on the vectorized corpus in the vectorized corpus. The mask corpus corresponding to each vectorized corpus can be obtained by inputting each vectorized corpus into a pre-trained mask layer (MASK Layer), and all masked corpora constitute a masked corpus. The mask layer is used to generate a masked corpus (MASK), and the generation rule of the masked corpus is: if the length of the data input to the mask layer is L, the mask layer generates an L×L lower triangular matrix, and adds 100-L integers 0 after each row of the matrix, and then adds 100-L rows with a length of 100, and finally the mask layer outputs a 100×100 matrix to represent the masked corpus.

[0098] Then, the Transformer model is trained unsupervisedly based on the concatenated corpus and the masked corpus. That is, the concatenated corpus and the masked corpus are used as the input of the Transformer model, and unsupervised training is performed on it to obtain a general language model.

[0099] In the embodiment of the present invention, the introduction of the mask corpus can enable the Transformer model to achieve unidirectional attention, thereby ensuring the accuracy of the description text obtained by the image description model.

[0100] Figure 3 FIG. 1 is a flow chart of the first stage training in the image description method provided in an embodiment of the present invention. Figure 3As shown in the figure, a vectorized corpus is first determined. The vectorized corpus contains multiple vectorized corpora (TEXT Vector). The vectorized corpora are passed through the embedding layer (Embedding Layer[batch_size,100,512]) to obtain a feature matrix. This feature matrix and an all-zero matrix (IMAGE Vector) of the same size as the image vector of the target image are input into the trained concatenation layer (CONCATE[batch_size,100,768]) for concatenation to obtain the concatenated corpus. The vectorized corpus is passed through the masking layer (MASK Layer[batch_size,100,100]) to obtain a masked corpus corresponding to each vectorized corpus. The masked corpus and the concatenated corpus are input into the Transformer model to obtain Transformer features. The Transformer model contains six Transformer layers (Transformer Layers[batch_size,100,512]). Finally, the output is obtained through the trained fully connected layer (Dense Layer[batch_size,100,vocab_size]) and output through the output layer (OUTPUT). Among them, vocab_size is the total number of words allowed by the fully connected layer to be included in a corpus.

[0101] On the basis of the above embodiment, in the image description method provided in the embodiment of the present invention, the image description corpus includes image samples and description corpus samples corresponding to the image samples; accordingly,

[0102] The second phase of training specifically includes:

[0103] Performing feature extraction on the image sample to obtain an image feature vector;

[0104] Determining an image attention vector based on the image feature vector;

[0105] Concatenating the image attention vector with a vectorized result obtained by vectorizing the description corpus sample to obtain a text-image hybrid vector;

[0106] The universal language model is trained based on the text-image mixed vector to obtain the image description model.

[0107] Specifically, in an embodiment of the present invention, during the second stage of training, an image description corpus is used. The image description corpus contains image samples and description corpus samples corresponding to the image samples. The resolution of the image samples can be limited to 229×229×3.

[0108] First, perform feature extraction on the image sample to obtain the image feature vector W1(64,256). Then, based on the image feature vector W1(64,256), determine the image attention vector W2(100,256). The image feature vector W1(64,256) can be input into the trained attention layer for processing to obtain the image attention vector W3(256). Repeat W3(256) 100 times according to the character length to obtain W2(100,256).

[0109] The image attention vector W2 (100, 256) is concatenated with the vectorized result W0 (100, 512) obtained by vectorizing the description corpus sample to obtain the text-image mixed vector W4 (100, 768).

[0110] The text-image mixed vector W4 (100, 768) is input into the general language model M1 to train the general language model M1 and obtain the image description model output vector W5 (100, vocab_size), which is the output of the image description model.

[0111] Before extracting features from image samples, the image samples can be scaled to a resolution of 224×224, and the description corpus samples corresponding to the image samples can be vectorized to transform the image description corpus into an image corpus training library. Subsequently, the general language model M1 can be trained based on the image corpus training library.

[0112] In the embodiment of the present invention, supervised training of the general language model is performed in combination with image samples and description corpus samples corresponding to the image samples during the second stage of training, so that the trained image description model can be more accurate.

[0113] On the basis of the above embodiment, the image description method provided in the embodiment of the present invention, wherein the feature extraction of the image sample to obtain the image feature vector specifically includes:

[0114] Performing feature extraction on the image sample to obtain a first candidate image feature vector;

[0115] Performing dimension transformation on the candidate image feature vector to obtain a second candidate image feature vector;

[0116] Perform dimensionality reduction processing on the second candidate image feature vector to obtain the image feature vector.

[0117] Specifically, in an embodiment of the present invention, when obtaining the image feature vector W1 (64, 256), the image sample can be first subjected to feature extraction by the trained InceptionV3 to obtain a first candidate image feature vector W6 (8, 8, 2048), and W6 (8, 8, 2048) is subjected to the trained Reshape to obtain a second candidate image feature vector W7 (64, 2048), and W7 (64, 2048) is subjected to dimensionality reduction processing by the trained fully connected layer to obtain the image feature vector W1 (64, 256).

[0118] Figure 4 FIG. 1 is a flow chart of the second stage training in the image description method provided in an embodiment of the present invention. Figure 4 As shown in the figure, we first determine the image sample (Image Input[batch_size,299,299,3]), and then pass it through the trained InceptionV3[batch_size,8,8,2048], the trained Reshape[batch_size,64,2048], and the trained fully connected layer (Dense Layer[batch_size,64,256]) to obtain the image feature vector W1(64,256). W1(64,256) passes through the trained Attention Layer[batch_size,256] and the trained Repeat Vector Layer[batch_size,100,256] to obtain W2(100,256). Next, the description corpus sample (Text Input[batch_size, 100]) corresponding to the image sample is determined. After passing through the embedding layer (Embedding Layer[batch_size, 100, 512) and the mask layer (MASK Layer[batch_size, 100, 100]), a feature matrix and masked corpus are obtained, respectively. W2(100, 256) is concatenated with the feature matrix through the trained concatenation layer (Concat Layer[batch_size, 100, 768]) to obtain the concatenated corpus. By inputting the masked corpus and concatenated corpus into the general language model trained in the first stage, the general language model can be trained to ultimately obtain the image description model.

[0119] Figure 5 A schematic diagram of the structure of an image description device provided by an embodiment of the present invention is shown in FIG. Figure 5 As shown, the device includes:

[0120] An acquisition module 51 is used to acquire a target image;

[0121] A description module 52 is configured to determine a description text of the target image based on an image description model;

[0122] The image description model is constructed based on the Transformer model and is used to perform text description on the target image; the image description model is obtained by performing first-stage training based on a general Chinese corpus and second-stage training based on an image description corpus.

[0123] Based on the above embodiment, there are multiple description texts; accordingly, the image description device provided in the embodiment of the present invention further includes:

[0124] A text error correction module, configured to determine, based on a text error correction model, error correction text embedding vectors corresponding to a plurality of the description texts; the text error correction model is trained based on the general Chinese corpus;

[0125] A clustering module, configured to determine, based on a clustering algorithm, a plurality of groups formed by the plurality of error-correcting text embedding vectors, and select a target group containing the largest number of error-correcting text embedding vectors from the plurality of groups;

[0126] A selection module is used to select a target error correction text embedding vector closest to the cluster center of the target group from the target group, and use the error correction text corresponding to the target error correction text embedding vector as the final description text of the target image.

[0127] On the basis of the above embodiment, in the image description device provided in the embodiment of the present invention, the text error correction module is specifically configured to:

[0128] Inputting the plurality of description texts into the text error correction model respectively, and obtaining error correction texts corresponding to the plurality of description texts output by the text error correction model;

[0129] The plurality of error-correcting texts are respectively input into the text error-correcting model to obtain error-correcting text embedding vectors corresponding to the plurality of error-correcting texts output by the text error-correcting model.

[0130] Based on the above embodiment, the image description device provided in the embodiment of the present invention further includes a first-stage training module for:

[0131] performing vectorization processing on the general Chinese corpus in the general Chinese corpus to obtain a vectorized corpus;

[0132] splicing the vectorized corpus in the vectorized corpus and an all-0 matrix having the same size as the image vector of the target image to obtain a spliced ​​corpus;

[0133] Based on the concatenated corpus, the Transformer model is trained unsupervised to obtain a general language model.

[0134] Based on the above embodiment, in the image description device provided in the embodiment of the present invention, the first-stage training module is specifically used to:

[0135] Based on the vectorized corpus in the vectorized corpus, generating a masked corpus corresponding to the vectorized corpus;

[0136] Based on the concatenated corpus and the masked corpus, the Transformer model is trained unsupervised to obtain a general language model.

[0137] Based on the above embodiment, the image description corpus includes image samples and description corpus samples corresponding to the image samples; accordingly, the image description device provided in the embodiment of the present invention further includes a second stage of training, specifically for:

[0138] Performing feature extraction on the image sample to obtain an image feature vector;

[0139] Determining an image attention vector based on the image feature vector;

[0140] Concatenating the image attention vector with a vectorized result obtained by vectorizing the description corpus sample to obtain a text-image hybrid vector;

[0141] The universal language model is trained based on the text-image mixed vector to obtain the image description model.

[0142] Based on the above embodiment, the image description device provided in the embodiment of the present invention, the second-stage training module is specifically used to:

[0143] Performing feature extraction on the image sample to obtain a first candidate image feature vector;

[0144] Performing dimension transformation on the candidate image feature vector to obtain a second candidate image feature vector;

[0145] Perform dimensionality reduction processing on the second candidate image feature vector to obtain the image feature vector.

[0146] It should be noted that the image description device provided in the embodiment of the present invention can implement all the method steps implemented in the above method embodiment and can achieve the same technical effects. The parts and beneficial effects of this embodiment that are the same as those in the method embodiment will not be described in detail here.

[0147] Figure 6An example of a physical structure diagram of an electronic device is shown below. Figure 6 As shown, the electronic device may include: a processor 610, a communication interface 620, a memory 630, and a communication bus 640, wherein the processor 610, the communication interface 620, and the memory 630 communicate with each other via the communication bus 640. The processor 610 may call the computer program in the memory 630 to execute the steps of the image description method provided in the above-mentioned method embodiments, for example, including: obtaining a target image; determining a description text of the target image based on an image description model; the image description model is constructed based on a Transformer model and is used to provide a text description of the target image; the image description model is obtained by performing a first-stage training based on a general Chinese corpus and a second-stage training based on an image description corpus.

[0148] In addition, the logic instructions in the above-mentioned memory 630 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0149] On the other hand, the present invention also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer can execute the steps of the image description method provided in the above-mentioned method embodiments, for example, including: obtaining a target image; determining a description text of the target image based on an image description model; the image description model is constructed based on a Transformer model and is used to perform textual description of the target image; the image description model is obtained by performing a first-stage training based on a general Chinese corpus and a second-stage training based on an image description corpus.

[0150] On the other hand, an embodiment of the present application also provides a processor-readable storage medium, which stores a computer program, and the computer program is used to enable the processor to execute the steps of the image description method provided in the above-mentioned method class embodiments, for example, including: obtaining a target image; determining a description text of the target image based on an image description model; the image description model is constructed based on a Transformer model and is used to perform text description of the target image; the image description model is obtained by performing a first-stage training based on a general Chinese corpus and a second-stage training based on an image description corpus.

[0151] The processor-readable storage medium can be any available medium or data storage device that can be accessed by the processor, including but not limited to magnetic storage (such as floppy disks, hard disks, magnetic tapes, magneto-optical disks (MO)), optical storage (such as CDs, DVDs, BDs, HVDs, etc.), and semiconductor storage (such as ROMs, EPROMs, EEPROMs, non-volatile memories (NANDFLASH), solid-state drives (SSDs)), etc.

[0152] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one location or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0153] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, or of course, by hardware. Based on this understanding, the essence of the above technical solution or the part that contributes to the existing technology can be embodied in the form of a software product. The computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or certain parts of the embodiments.

[0154] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. An image description method, characterized in that: include: Acquire the target image; Determining a description text of the target image based on an image description model; The image description model is constructed based on the Transformer model and is used to perform text description on the target image; the image description model is obtained by performing a first-stage training based on a general Chinese corpus and a second-stage training based on an image description corpus; The first phase of training specifically includes: performing vectorization processing on the general Chinese corpus in the general Chinese corpus to obtain a vectorized corpus; splicing the vectorized corpus in the vectorized corpus and an all-0 matrix having the same size as the image vector of the target image to obtain a spliced ​​corpus; Based on the concatenated corpus, the Transformer model is trained unsupervised to obtain a general language model.

2. The image description method according to claim 1, characterized in that There are multiple description texts; accordingly, the image description method further includes: Determining, based on a text error correction model, error correction text embedding vectors corresponding to a plurality of the description texts; the text error correction model is trained based on the general Chinese corpus; Based on a clustering algorithm, determining a plurality of groups formed by a plurality of the error correction text embedding vectors, and selecting a target group containing the largest number of the error correction text embedding vectors from the plurality of groups; A target error correction text embedding vector closest to the cluster center of the target group is selected from the target group, and the error correction text corresponding to the target error correction text embedding vector is used as the final description text of the target image.

3. The image description method according to claim 2, characterized in that The determining, based on the text error correction model, error correction text embedding vectors corresponding to the plurality of description texts specifically includes: Inputting the plurality of description texts into the text error correction model respectively, and obtaining error correction texts corresponding to the plurality of description texts output by the text error correction model; The plurality of error-correcting texts are respectively input into the text error-correcting model to obtain error-correcting text embedding vectors corresponding to the plurality of error-correcting texts output by the text error-correcting model.

4. The image description method according to claim 1, wherein: The first phase of training also includes: Based on the vectorized corpus in the vectorized corpus, generating a masked corpus corresponding to the vectorized corpus; Accordingly, the unsupervised training of the Transformer model based on the concatenated corpus to obtain a general language model specifically includes: Based on the concatenated corpus and the masked corpus, the Transformer model is trained unsupervised to obtain a general language model.

5. The image description method according to claim 1, wherein: The image description corpus contains image samples and description corpus samples corresponding to the image samples; accordingly, The second phase of training specifically includes: Performing feature extraction on the image sample to obtain an image feature vector; Determining an image attention vector based on the image feature vector; Concatenating the image attention vector with a vectorized result obtained by vectorizing the description corpus sample to obtain a text-image hybrid vector; The universal language model is trained based on the text-image mixed vector to obtain the image description model.

6. The image description method according to claim 5, characterized in that Extracting features from the image sample to obtain an image feature vector specifically includes: Performing feature extraction on the image sample to obtain a first candidate image feature vector; Performing dimension transformation on the candidate image feature vector to obtain a second candidate image feature vector; Perform dimensionality reduction processing on the second candidate image feature vector to obtain the image feature vector.

7. An image description device, characterized in that: include: An acquisition module, used to acquire a target image; A description module, configured to determine a description text of the target image based on an image description model; The image description model is constructed based on the Transformer model and is used to perform text description on the target image; the image description model is obtained by performing a first-stage training based on a general Chinese corpus and a second-stage training based on an image description corpus; The first phase of training specifically includes: performing vectorization processing on the general Chinese corpus in the general Chinese corpus to obtain a vectorized corpus; splicing the vectorized corpus in the vectorized corpus and an all-0 matrix having the same size as the image vector of the target image to obtain a spliced ​​corpus; Based on the concatenated corpus, the Transformer model is trained unsupervised to obtain a general language model.

8. An electronic device comprising a processor and a memory storing a computer program, characterized in that: When the processor executes the computer program, the steps of the image description method according to any one of claims 1 to 6 are implemented.

9. A processor-readable storage medium, characterized in that: The processor-readable storage medium stores a computer program, and the computer program is used to enable the processor to execute the steps of the image description method according to any one of claims 1 to 6.

Images