Description text generation method and device

Abstract

The invention provides a description text generation method and device, and relates to the technical field of artificial intelligence, and the description text generation method comprises the steps: carrying out the recognition of a plurality of different image texts of a to-be-described target image, and obtaining a plurality of initial description texts corresponding to the target image; performing duplicate removal processing and splicing processing on each initial description text to obtain a standby description text corresponding to the target image; and generating a target description text corresponding to the target image according to at least one pre-acquired initial style identifier and the standby description text. According to the method, high-quality description texts corresponding to images can be generated massively.

Description

Technical Field

[0001] The present invention relates to the fields of artificial intelligence technology, image processing technology and text processing technology, and in particular to a description text generation method and device. Background Art

[0002] With the progress of artificial intelligence in various fields, the combination of computer vision and natural language processing has also received widespread attention. For example, the text-based image model, which generates images based on text information, has become a research hotspot due to its advantages such as image, vividness and fluency.

[0003] Training a text-to-image model requires a large number of image-text samples, i.e., images and corresponding image description texts. However, most of the text descriptions in image-text samples are obtained through online crawlers and are of low quality, i.e., the accuracy of describing the image is low or the sentences are poor. Therefore, an effective solution is urgently needed to solve the above problems. Summary of the invention

[0004] In view of the problems existing in the prior art, an embodiment of the present invention provides a description text generation method and device.

[0005] The present invention provides a description text generation method, comprising:

[0006] Performing multiple different image text recognitions on the target image to be described to obtain multiple initial description texts corresponding to the target image;

[0007] Deduplication processing and splicing processing are performed on each of the initial description texts to obtain a backup description text corresponding to the target image;

[0008] A target description text corresponding to the target image is generated according to at least one pre-acquired initial style identifier and the backup description text.

[0009] According to a description text generation method provided by the present invention, the deduplication processing and splicing processing of each of the initial description texts are performed to obtain the backup description text corresponding to the target image, including:

[0010] Calculating initial text similarities between the initial description texts, and determining target text similarities among the initial text similarities that are greater than a text similarity threshold;

[0011] For each of the two initial description texts corresponding to the target text similarity, the initial description text with the lower first image-text similarity is deleted, where the first image-text similarity is the similarity between the initial text and the target image;

[0012] The remaining initial description texts are spliced ​​to obtain backup description texts corresponding to the target image.

[0013] According to a description text generation method provided by the present invention, before generating the target description text corresponding to the target image according to the at least one initial style identifier and the backup description text acquired in advance, the method further includes:

[0014] Obtain at least one initial style identifier;

[0015] Calculating the second graphic similarity between each of the initial style identifiers and the target image;

[0016] According to each of the second image-text similarities, determining whether there is a target style identifier corresponding to the target image in each of the initial style identifiers;

[0017] Accordingly, generating the target description text corresponding to the target image according to the at least one pre-acquired initial style identifier and the backup description text includes:

[0018] If there is a target style identifier corresponding to the target image, concatenate the target style identifier corresponding to the target image with the backup description text to obtain the target description text corresponding to the target image;

[0019] If there is no target style identifier corresponding to the target image, the backup description text is determined as the target description text corresponding to the target image.

[0020] According to a description text generation method provided by the present invention, before performing deduplication processing and splicing processing on each of the initial description texts to obtain a spare description text corresponding to the target image, the method further includes:

[0021] Calculating the similarity between each of the initial description texts and the first image-text of the target image;

[0022] The initial description text whose first image-text similarity is lower than a first image-text similarity threshold is deleted.

[0023] According to a description text generation method provided by the present invention, the method performs a plurality of different image text recognitions on a target image to be described to obtain a plurality of initial description texts corresponding to the target image, including:

[0024] Inputting a target image to be described into a pre-trained first image-to-text model to obtain an initial description text corresponding to the target image;

[0025] Inputting the target image to be described into the pre-trained second image-to-text model to obtain an initial description text corresponding to the target image;

[0026] Inputting the target image to be described into the pre-trained third image-to-text model to obtain an initial description text corresponding to the target image;

[0027] The first image-to-text model, the second image-to-text model and the third image-to-text model are trained based on different image-to-text object sets.

[0028] According to a description text generation method provided by the present invention, the target image to be described is input into a pre-trained first image-to-text model to obtain an initial description text corresponding to the target image, including:

[0029] Inputting the target image into the image encoder of the first image-to-text model for image encoding to obtain a first image embedding feature, and inputting empty text into the text encoder of the first image-to-text model for text encoding to obtain a first text embedding feature;

[0030] The first image embedding feature is input into the multimodal text encoder of the first image-to-text model through the attention mechanism of the first image-to-text model, and the first text embedding feature is input into the multimodal text encoder of the first image-to-text model to perform feature fusion to obtain an initial description text corresponding to the target image.

[0031] According to a description text generation method provided by the present invention, the target image to be described is input into a pre-trained second image-to-text model to obtain an initial description text corresponding to the target image, including:

[0032] Step 1: Input the target image into the image encoder of the second image-to-text model for image encoding to obtain a second image embedding feature, and input the specified text into the text encoder of the second image-to-text model for text encoding to obtain a second text embedding feature, wherein the specified text is empty text;

[0033] Step 2: Analyze and process each character in the character library according to the second image embedding feature and the second text embedding feature to determine the character feature of each character;

[0034] Step 3: Determine the probability of each character according to the characteristics of each character, and randomly sample each character based on the probability to obtain a sampling result;

[0035] Step 4: concatenate the sampling result and the designated text to obtain an updated designated text;

[0036] Continue to execute step 1 to step 4 until the sampling result is a stop symbol or a blank, and determine the designated text as the initial description text corresponding to the target image.

[0037] According to a description text generation method provided by the present invention, the target image to be described is input into a pre-trained third image-to-text model to obtain an initial description text corresponding to the target image, including:

[0038] Inputting the target image into the image encoder of the third image-to-text model for image encoding to obtain a third image embedding feature, and inputting the empty text into the text encoder of the third image-to-text model for text encoding to obtain a third text embedding feature;

[0039] Concatenate the third image embedding feature and the third text embedding feature through a separator to obtain a total embedding feature;

[0040] The total embedded features are input into the feature fusion device of the third image-to-text model for feature fusion to obtain the initial description text corresponding to the target image.

[0041] According to a description text generation method provided by the present invention, before performing image text recognition on a target image to be described to obtain a plurality of initial description texts corresponding to the target image, the method further comprises:

[0042] Acquire a plurality of sample image and text objects, wherein the sample image and text objects include sample images and descriptive text labels corresponding to the sample images;

[0043] Randomly extracting a first specified number of sample graphic objects from each of the sample graphic objects to obtain a first graphic object set, randomly extracting a second specified number of sample graphic objects from each of the sample graphic objects to obtain a second graphic object set, and randomly extracting a third specified number of sample graphic objects from each of the sample graphic objects to obtain a third graphic object set;

[0044] The first prediction model is trained according to the first image-text object set to obtain the first image-text model, the second prediction model is trained according to the second image-text object set to obtain the second image-text model, and the third prediction model is trained according to the third image-text object set to obtain the third image-text model.

[0045] The present invention also provides a description text generating device, comprising:

[0046] A recognition module is configured to perform a plurality of different image text recognitions on a target image to be described, and obtain a plurality of initial description texts corresponding to the target image;

[0047] A deduplication module is configured to perform deduplication processing and splicing processing on each of the initial description texts to obtain a backup description text corresponding to the target image;

[0048] The generating module is configured to generate a target description text corresponding to the target image according to at least one pre-acquired initial style identifier and the backup description text.

[0049] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, any of the above-mentioned description text generation methods is implemented.

[0050] The present invention also provides a non-transitory computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the description text generation method as described in any one of the above is implemented.

[0051] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements any of the above-mentioned description text generation methods.

[0052] The description text generation method and device provided by the present invention obtain multiple initial description texts corresponding to the target image by performing multiple different image text recognition on the target image to be described; perform deduplication processing and splicing processing on each of the initial description texts to obtain a backup description text corresponding to the target image; generate a target description text corresponding to the target image according to at least one initial style identifier and the backup description text obtained in advance. By deduplication and splicing processing on the generated multiple initial description texts, the target description text can more accurately and comprehensively describe the target image; generating the target description text according to the initial style identifier and the backup description text can make the target description text contain not only text describing the content of the target image, but also text describing the style of the target image, thereby improving the comprehensiveness and accuracy of the target description text in describing the target image; thereby forming a higher quality target image-text object to train a better text-image model. Corresponding high-quality description texts can also be generated for a large number of images. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0054] Figure 1 It is one of the flow charts of the description text generation method provided by the present invention;

[0055] Figure 2 It is a schematic diagram of the process of training the image-text similarity prediction model provided by the present invention;

[0056] Figure 3 is a structural schematic diagram of the first image-to-text model provided by the present invention;

[0057] Figure 4 is a structural schematic diagram of a second image-to-text model provided by the present invention;

[0058] Figure 5 is a schematic diagram of the structure of the third image-generated text model provided by the present invention;

[0059] Figure 6 This is the second flow chart of the description text generation method provided by the present invention;

[0060] Figure 7 It is a schematic diagram of the process of generating text description by multiple models provided by the present invention;

[0061] Figure 8 It is a flowchart of text deduplication provided by the present invention;

[0062] Fig. 9 It is a schematic diagram of the process of adding a style identifier provided by the present invention;

[0063] Fig.10 It is a structural schematic diagram of the description text generation device provided by the present invention;

[0064] Fig.11 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0065] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0066] In order to facilitate a clearer understanding of the embodiments of the present invention, some relevant background knowledge is first introduced as follows.

[0067] With the progress of artificial intelligence in various fields, the combination of computer vision and natural language processing has also received widespread attention. For example, the text-based image model, which generates images based on text information, has become a research hotspot due to its advantages such as image, vividness and fluency.

[0068] Training a text-to-image model requires a large number of image-text samples, i.e., images and corresponding image description texts. However, most of the text descriptions in the image-text samples are obtained through online crawlers and are of low quality, i.e., the accuracy of describing the image is low or the sentences are poor.

[0069] In addition, there are no ready-made description texts for most images, and description texts need to be manually written one by one based on the images. This is inefficient and makes it impossible to obtain a large number of image and text samples.

[0070] In order to solve the problem of low efficiency in manually writing description texts, the prior art generates description texts based on images through image-to-text generation technology, but the quality of the generated description texts is poor.

[0071] Therefore, the present invention provides a description text generation method and device to solve the above problems.

[0072] Combine the following Figure 1-Figure 9 The description text generation method and device of the present invention are described.

[0073] Figure 1 This is one of the flow charts of the description text generation method provided by the present invention, see Figure 1 As shown, it includes steps 101 to 102, wherein:

[0074] Step 101: performing a plurality of different image text recognitions on a target image to be described, and obtaining a plurality of initial description texts corresponding to the target image.

[0075] First of all, it should be noted that the execution subject of the present invention can be any electronic device that generates description text, for example, it can be any one of a smart phone, a smart watch, a desktop computer, a laptop computer, etc.

[0076] Specifically, the target image is the image to be described in text. Image text recognition refers to the process of describing an image and generating a description text. The description text refers to the text describing the image. The initial description text refers to the initially generated description text.

[0077] In practical applications, before performing image text recognition on the target image to be described, it is necessary to first obtain the target image to be described. There are many ways to obtain the target image: for example, the user uploads the target image to be described through the upload page provided by the description text generation platform, and accordingly, the execution subject obtains the target image to be described. For another example, the execution subject receives a description text generation instruction or a target image acquisition instruction, and accordingly, the execution subject obtains the target image to be described from the storage area pointed to by the description text generation instruction or the target image acquisition instruction. The present invention is not limited to this.

[0078] Furthermore, according to a preset image text recognition strategy, a plurality of different image text recognitions are performed on the target image to be described to obtain a plurality of initial description texts corresponding to the target image.

[0079] Exemplarily, a target image to be described is input into different image-to-text models for image-to-text recognition to obtain a plurality of initial description texts corresponding to the target image.

[0080] Exemplarily, an image recognition model is used to identify elements in a target image and output words corresponding to each element, and then each word is input into a different text generation model to compile the words, or each word is input into the same text generation model multiple times to compile the words to form multiple initial description texts.

[0081] Step 102: performing deduplication processing and splicing processing on each of the initial description texts to obtain a backup description text corresponding to the target image.

[0082] Specifically, the backup description text refers to the description text after deduplication and splicing processing.

[0083] In practical applications, after obtaining multiple initial description texts, repeated and redundant initial description texts are removed from each initial description text: the semantic similarity between the initial description texts can be calculated, and the two initial description texts with semantic similarity greater than a set value can be deduplicated; all initial description texts can be input into a preset deduplication model for deduplication, and the deduplicated initial description texts can be obtained.

[0084] Furthermore, the remaining initial description texts, ie, the initial description texts after deduplication, are spliced ​​using punctuation marks, such as at least one of a comma, a colon, and a semicolon, to obtain a spliced ​​standby description text.

[0085] Step 103: Generate a target description text corresponding to the target image according to at least one pre-acquired initial style identifier and the backup description text.

[0086] Specifically, style tags refer to text that describes the style of an image, such as art style, cyberpunk, steampunk, science fiction, sketch, and watercolor.

[0087] In practical applications, on the basis of obtaining the backup description text, further, at least one initial style identifier is obtained, and then based on the obtained initial style identifier and the backup description text, the target description text is generated.

[0088] Exemplarily, the initial style identifier is obtained by performing image style recognition on the target image in advance, such as using a style recognition model to perform image style recognition on the target image, thereby obtaining at least one initial style identifier corresponding to the target image. That is, firstly identify at least one initial style identifier corresponding to the target image, and then concatenate each initial style identifier and the backup description text to obtain the target description text.

[0089] Exemplarily, the initial style identifier is an existing initial style identifier collected in advance, and then each initial style identifier is matched with the target image respectively, and the successfully matched initial style identifier is used as the target style identifier of the target image, and then the target style identifier is spliced ​​with the backup description text to obtain the target description text. If there is no successfully matched target style identifier, the backup description text can be used as the target description text.

[0090] The description text generation method provided by the present invention obtains multiple initial description texts corresponding to the target image by performing multiple different image text recognition on the target image to be described; performs deduplication processing and splicing processing on each of the initial description texts to obtain a backup description text corresponding to the target image; generates a target description text corresponding to the target image according to at least one initial style identifier and the backup description text obtained in advance. By deduplication and splicing processing on the generated multiple initial description texts, the target description text can more accurately and comprehensively describe the target image; generating the target description text according to the initial style identifier and the backup description text can make the target description text contain not only text describing the content of the target image, but also text describing the style of the target image, thereby improving the comprehensiveness and accuracy of the target description text in describing the target image; thereby forming a higher quality target image-text object to train a better text-image model. Corresponding high-quality description texts can also be generated for a large number of images.

[0091] In one or more optional embodiments of the present invention, the deduplication processing and splicing processing are performed on each of the initial description texts to obtain the backup description text corresponding to the target image. The specific implementation process may be as follows:

[0092] Calculating initial text similarities between the initial description texts, and determining target text similarities from the initial text similarities;

[0093] For each of the two initial description texts corresponding to the target text similarity, the initial description text with the lower first image-text similarity is deleted, where the first image-text similarity is the similarity between the initial text and the target image;

[0094] The remaining initial description texts are spliced ​​to obtain backup description texts corresponding to the target image.

[0095] Specifically, text similarity refers to the similarity between two initial description texts. Image-text similarity refers to the similarity between text and an image. The first image-text similarity refers to the similarity between the initial description text and the target image.

[0096] In practical applications, a pre-trained text similarity prediction model can be used to calculate the initial text similarity between each initial description text. For example, a transformer-based bidirectional encoder model can be used to calculate the text similarity between the initial text descriptions to obtain the initial text similarity between each initial description text. A preset semantic similarity algorithm can also be used to calculate the initial text similarity between each initial description text.

[0097] Furthermore, the target text similarity is determined from each initial text similarity. For each target text similarity, one of the two initial description texts can be deleted or discarded, thereby improving the efficiency of deduplication and simplifying the process; the initial description text with a lower text-image similarity to the target image can also be deleted or discarded, and the initial description text with a higher text-image similarity can be retained, further improving the accuracy of the target description text in describing the target image.

[0098] After deduplication is completed, the deduplicated initial description texts can be spliced ​​together to obtain candidate description texts, thereby improving the description accuracy of the target description text.

[0099] In an optional embodiment of the present invention, the target text similarity is determined from the initial text similarities, and the specific implementation process may be as follows:

[0100] Determine a target text similarity among the initial text similarities that is greater than a text similarity threshold.

[0101] Specifically, the text similarity threshold refers to a preset threshold used to measure whether the text similarity meets the requirements.

[0102] In practical applications, each initial text similarity can be compared with a text similarity threshold, and the initial text similarity greater than the text similarity threshold is used as the target text similarity. In this way, the efficiency and accuracy of deduplication can be improved.

[0103] In another optional embodiment of the present invention, the target text similarity is determined from the initial text similarities, and the specific implementation process may be as follows:

[0104] A maximum preset number of the initial text similarities are selected as the target text similarities.

[0105] Specifically, the preset number refers to a pre-set value, such as a natural number such as 2, 3, 4, etc.

[0106] In practical applications, the initial text similarities can be arranged in order from large to small, and the first preset number of initial text similarities can be used as the target text similarity, or the initial text similarities can be arranged in order from small to large, and the last preset number of initial text similarities can be used as the target text similarity. In this way, the efficiency of determining the similarity of the target text can be improved.

[0107] It should be noted that, before calculating the initial text similarity between each of the initial description texts, each of the initial description texts can be arranged in descending order according to the first image-text similarity. Further, the text similarity between the first initial description text in the queue and each of the other initial description texts thereafter is calculated, and other initial description texts whose text similarity with the first initial description text is greater than the text similarity threshold are deleted from the queue to obtain an updated queue; then the text similarity between the second initial description text in the queue and each of the other initial description texts thereafter is calculated, and other initial description texts whose text similarity with the second initial description text is greater than the text similarity threshold are deleted from the queue to obtain an updated queue, and so on, until the last initial description text in the queue. In this way, the calculation of the text similarity between all the initial description texts can be avoided, thereby reducing the amount of data processing and improving the generation efficiency of the target description text.

[0108] In one or more optional embodiments of the present invention, before generating the target description text corresponding to the target image according to the at least one initial style identifier obtained in advance and the backup description text, the method further includes:

[0109] Obtain at least one initial style identifier;

[0110] Calculating the second graphic similarity between each of the initial style identifiers and the target image;

[0111] According to each of the second image-text similarities, determining whether there is a target style identifier corresponding to the target image in each of the initial style identifiers;

[0112] Accordingly, generating the target description text corresponding to the target image according to the at least one pre-acquired initial style identifier and the backup description text includes:

[0113] If there is a target style identifier corresponding to the target image, concatenate the target style identifier corresponding to the target image with the backup description text to obtain the target description text corresponding to the target image;

[0114] If there is no target style identifier corresponding to the target image, the backup description text is determined as the target description text corresponding to the target image.

[0115] Specifically, the second image-text similarity refers to the similarity between the initial style identifier and the target image.

[0116] In practical applications, a style identifier library is obtained, which includes at least one collected initial style identifier. Then, a second image-text similarity between each initial style identifier and the target image is calculated using a pre-trained image-text similarity prediction model, such as using a comparative language image pre-training model to calculate the similarity between each initial style identifier and the target image, and obtain the second image-text similarity between each initial style identifier and the target image.

[0117] Furthermore, based on each second image-text similarity, it is determined whether there is a target style identifier corresponding to the target image in each initial style identifier: for example, the second image-text similarity corresponding to each initial style identifier is compared with the second image-text similarity threshold, to determine whether there is a target style identifier in each initial style identifier whose second image-text similarity is greater than the second image-text similarity threshold.

[0118] If the target style identifier exists, the target style identifier and the backup description text are concatenated to obtain the target description text; if the target style identifier does not exist, the backup description text is used as the target description text.

[0119] In this way, the accuracy of the target style identifier in representing the target image style can be improved, thereby improving the description accuracy of the target description text.

[0120] In addition, a preset number of initial style identifiers with the largest second image-text similarity can be used as target style identifiers. In this way, the existence of a target style identifier can be guaranteed, and then the target style identifier and the alternative description text can be spliced ​​together to obtain a target description text, so that the target description text can describe both the content of the target image and the style of the target image, thereby improving the completeness of the description.

[0121] In one or more optional embodiments of the present invention, before performing deduplication processing and splicing processing on the initial description texts to obtain the backup description text corresponding to the target image, the method further includes:

[0122] Calculating the similarity between each of the initial description texts and the first image-text of the target image;

[0123] The initial description texts are filtered according to the similarities between the first images and texts.

[0124] In practical applications, a pre-trained image-text similarity prediction model can be used to calculate the first image-text similarity between each initial description text and the target image. For example, the similarity between each initial description text and the target image is calculated using the image-text similarity prediction model to obtain the first image-text similarity between each initial description text and the target image.

[0125] Then, each initial description text is filtered according to the first image-text similarity corresponding to each initial description text: for example, the initial description text whose first image-text similarity is lower than the first image-text similarity threshold is deleted; for another example, a preset number of initial description texts with the lowest first image-text similarity are deleted. In this way, the initial description texts with lower first image-text similarity are filtered out, and the initial description texts with higher first image-text similarity are retained for generating the target description text, thereby further improving the description accuracy of the target description text.

[0126] It should be noted that the image-text similarity prediction models for calculating the first image-text similarity and the second image-text similarity may be different or the same. The process of using the image-text similarity prediction model to calculate the specified image-text similarity may be as follows:

[0127] For each target text, the target text is input into the text encoding layer of the pre-trained image-text similarity prediction model for text encoding processing to obtain a text embedding vector, and the target image is input into the image encoding layer of the image-text similarity prediction model for image encoding processing to obtain an image embedding vector, wherein the target text is an initial description text or an initial style identifier;

[0128] The text embedding vector and the image embedding vector are input into the similarity prediction layer of the image-text similarity prediction model to perform similarity calculation to obtain a specified image-text similarity between the target text and the initial image, wherein the specified image-text similarity is a first image-text similarity or a second image-text similarity.

[0129] Specifically, the image-text similarity prediction model refers to a model used to calculate the similarity between an image and a text, such as the image-text similarity prediction model. The text encoding layer, i.e., the text encoder, refers to the unit that encodes the text in the image-text similarity prediction model, and can be a bidirectional encoding model of a transformer; the image encoding layer, i.e., the image encoder, refers to the unit that encodes the image in the image-text similarity prediction model, and can be a visual transformer. The similarity prediction layer refers to the unit that calculates the similarity of the image in the similarity prediction model.

[0130] In practical applications, the target text can be encoded by the text encoding layer of the image-text similarity prediction model to obtain a text embedding vector, and the target image can be encoded by the image encoding layer of the image-text similarity prediction model to obtain an image embedding vector. Then, the similarity prediction layer of the image-text similarity prediction model can be used to calculate the similarity between the text embedding vector and the image embedding vector to obtain the specified image-text similarity.

[0131] In addition, there are multiple algorithms for calculating similarity. The similarity prediction layer can perform dot product calculation on the text embedding vector and the image embedding vector to obtain the specified image-text similarity, or perform cosine distance calculation on the text embedding vector and the image embedding vector to obtain the specified image-text similarity, or other algorithms, which are not limited by the present invention.

[0132] It should be noted that, when the target text is the initial description text, the designated image-text similarity is the first image-text similarity; when the target text is the initial style identifier, the designated image-text similarity is the second image-text similarity.

[0133] In one or more optional embodiments of the present specification, the training process of the image-text similarity prediction model is as follows: obtaining an untrained image-text similarity prediction model, and obtaining a sample image-text object carrying an image-text similarity label, the image-text similarity prediction model comprising a text encoding layer, an image encoding layer and a similarity prediction layer, the sample image-text object comprising a first sample image and a sample text of the first sample image, the sample text being a sample style identifier or a sample description text; inputting the sample text into the text encoding layer to obtain a predicted text embedding vector, and inputting the first sample image into the image encoding layer to obtain a predicted image embedding vector; inputting the predicted text embedding vector and the predicted image embedding vector into the similarity prediction layer to obtain a predicted similarity; calculating a loss value according to the image-text similarity label and the predicted similarity; performing gradient optimization on the image-text similarity prediction model according to the loss value, and continuing to execute the step of obtaining a sample image-text object carrying an image-text similarity label until a preset stop condition is reached to obtain a trained image-text similarity prediction model.

[0134] The preset stop condition may be that the loss value is less than the loss threshold, the number of iterations reaches the number threshold, the rate of change of the loss value is less than the rate of change threshold, or a combination of the above. The loss value may be obtained by binary cross entropy loss calculation or mean square error loss calculation.

[0135] For example, see Figure 2 , Figure 2It is a flow chart of the training image-text similarity prediction model provided by the present invention: the batch size of the untrained image-text similarity prediction model is 1, the sample text is encoded into a text embedding (predicted text embedding vector) through a text encoder (text encoding layer), and the dimension is 1x256; the first sample image is encoded into an image embedding (predicted image embedding vector) through a visual transformer (image encoder, i.e., image encoding layer), and the dimension is 1x256; the text embedding and the image embedding are dot-producted to obtain a 1x1 Logit, i.e., predicted similarity, and the Logit and the image-text similarity label are binary cross entropy loss calculated to obtain Loss (loss value), and the image-text similarity prediction model is gradient optimized. Two methods are used during training. When the image and text match, the Label (similarity label) is 1, and when the image and text do not match, the Label is 0. In this way, after a large amount of training, the image-text similarity prediction model can calculate the specified image-text similarity between the target image and the target text.

[0136] In one or more optional embodiments of the present invention, the step of performing multiple different image text recognition on the target image to be described to obtain multiple initial description texts corresponding to the target image includes: inputting the target image to be described into multiple pre-trained image-to-text models for image text recognition to obtain multiple initial description texts corresponding to the target image, wherein each image-to-text model is trained based on a different image-to-text object set. The image-to-text model refers to a model that generates description text based on an image.

[0137] In this way, generating the initial description text based on multiple pre-trained image-text models can improve the acquisition efficiency and description accuracy of the initial description text. In addition, the image-text models trained using different sets of image-text objects can make the initial description text have different focuses, thereby improving the richness of the initial description text.

[0138] In one or more optional embodiments of the present invention, the multiple image-generated text models include a first image-generated text model, a second image-generated text model, and a third image-generated text model; accordingly, the target image to be described is respectively input into the multiple pre-trained image-generated text models for image text recognition to obtain multiple initial description texts corresponding to the target image, including:

[0139] Inputting a target image to be described into a pre-trained first image-to-text model to obtain an initial description text corresponding to the target image;

[0140] Inputting the target image to be described into the pre-trained second image-to-text model to obtain an initial description text corresponding to the target image;

[0141] Inputting the target image to be described into the pre-trained third image-to-text model to obtain an initial description text corresponding to the target image;

[0142] The first image-to-text model, the second image-to-text model and the third image-to-text model are trained based on different image-to-text object sets.

[0143] In practical applications, the target image to be described can be input into the first image-generated text model, the second image-generated text model and the third image-generated text model respectively, so as to obtain the initial description text output by the first image-generated text model, the second image-generated text model and the third image-generated text model respectively. In this way, the acquisition efficiency, description accuracy and richness of the initial description text can be further improved.

[0144] Optionally, the target image to be described is input into a pre-trained first image-to-text model to obtain an initial description text corresponding to the target image. The specific implementation process may be as follows:

[0145] Inputting the target image into the image encoder of the first image-to-text model for image encoding to obtain a first image embedding feature, and inputting empty text into the text encoder of the first image-to-text model for text encoding to obtain a first text embedding feature;

[0146] The first image embedding feature is input into the multimodal text encoder of the first image-to-text model through the attention mechanism of the first image-to-text model, and the first text embedding feature is input into the multimodal text encoder of the first image-to-text model to perform feature fusion to obtain an initial description text corresponding to the target image.

[0147] Specifically, the first image-to-text model can be a contrast subtitle model. The text encoder refers to the unit that encodes the text in the image-to-text model, which can be a bidirectional encoding model based on a transformer; the attention mechanism adopts CrossAttention, which is a multimodal feature fusion method. The image encoder refers to the unit that encodes the image in the image-to-text model, which can be a visual transformer. The multimodal text encoder refers to the unit that performs feature fusion in the image-to-text model, which can be composed of 6 Tansformer Blocks.

[0148] In practical applications, the image encoder of the first image-to-text model extracts features from the target image to obtain a first image embedding feature (Embedding) with a dimension of 1x768, and the text encoder of the first image-to-text model extracts features from empty text to obtain a first text embedding feature (Embedding) with a dimension of 1x768.

[0149] Furthermore, the first text embedding feature is input into the multimodal text encoder, and the first image embedding feature is input into the multimodal text encoder through CrossAttention, and the multimodal text encoder fuses the first text embedding feature and the first image embedding feature. After that, the multimodal text encoder outputs the initial description text corresponding to the target image: the multimodal text encoder directly outputs a matrix of M x N dimensions, where M represents the output length, that is, the number of words contained in the initial description text, which can be set as needed, for example, set to 100; N is the number of vocabularies, and the maximum value in the dimension N is taken to obtain the output initial text description. In this way, the efficiency of obtaining the initial description text can be improved.

[0150] See also Figure 3 , Figure 3 It is a structural schematic diagram of the first image-to-text model provided by the present invention. The target image is encoded using a visual transformer (image encoder) to obtain a first image embedding feature with a dimension of 1x768. Next, the transformer's bidirectional encoding model (text encoder) is used to extract features from the empty text (sample description text) to obtain a first text embedding feature with a dimension of 1x768. Then, the first text embedding feature is directly input into the multimodal text encoder. At the same time, the first image embedding feature is embedded into the multimodal text encoder through Cross Attention to achieve the fusion of image features and text features. 4) Finally, the multimodal text encoder directly outputs a dimension of M x N. Taking the maximum value in the dimension N, the output initial text description can be obtained.

[0151] Optionally, the target image to be described is input into a pre-trained second image-to-text model to obtain an initial description text corresponding to the target image. The specific implementation process may be as follows:

[0152] Step 1: Input the target image into the image encoder of the second image-to-text model for image encoding to obtain a second image embedding feature, and input the specified text into the text encoder of the second image-to-text model for text encoding to obtain a second text embedding feature, wherein the specified text is empty text;

[0153] Step 2: Analyze and process each character in the character library according to the second image embedding feature and the second text embedding feature to determine the character feature of each character;

[0154] Step 3: Determine the probability of each character according to the characteristics of each character, and randomly sample each character based on the probability to obtain a sampling result;

[0155] Step 4: concatenate the sampling result and the designated text to obtain an updated designated text;

[0156] Continue to execute step 1 to step 4 until the sampling result is a stop symbol or a blank, and determine the designated text as the initial description text corresponding to the target image.

[0157] Specifically, the second image-to-text model can be a guided language image pre-training model. The character library is a character table, which stores multiple characters for use in generating text. Probability refers to the sampling probability of the characters. The sampling result refers to the result of sampling, which can be the sampled characters, or it can be empty, or it can be a stop symbol, where the stop symbol is such as the [STOP] symbol.

[0158] In practical applications, the image encoder of the second image-to-text model extracts features from the target image to obtain second image embedding features, and the text encoder of the second image-to-text model extracts features from the specified text to obtain second text embedding features.

[0159] Further, the second image embedding feature and the second text embedding feature are processed according to the autoregressive text generation strategy to obtain the initial description text corresponding to the target image, including: analyzing and processing each character in the character library according to the second image embedding feature and the second text embedding feature to determine the text features of each second character, and each text feature can be output as a 1xM-dimensional Logit, where M is the number of characters in the character library or character table, that is, M text features are output in the form of a matrix; obtaining the probability of each character through a probability function for each text feature, and randomly sampling each character according to the probability of each character, randomly selecting a character from the character library to obtain a sampling result; splicing the sampling result and the designated text to obtain an updated designated text; when the sampling result is a character, splicing the character and the designated text to obtain an updated designated text, and then extracting features from the target image and the updated designated text respectively to obtain the first image embedding feature and the first text embedding feature, and then analyzing and processing each character in the character library based on the first image embedding feature and the first text embedding feature, and so on, until the sampling result is a stop symbol or a blank, and the updated designated text is used as the initial description text corresponding to the target image.

[0160] By determining the probability of each character based on the character features of each character and sampling based on the probability, the description accuracy of the initial description text can be improved. And through random sampling, each generated initial description text can be as different as possible while describing the target image, which is conducive to determining the efficiency and accuracy of the target description text in each initial description text.

[0161] It should be noted that, since this embodiment generates the initial description text based on the joint processing of image embedding features and text embedding features, in order to avoid the initial text embedding features affecting the accuracy of the initial description text, it is necessary to set the initial designated text to empty text. In this way, the empty text does not contain any text, so that the text embedding features corresponding to the empty text do not contain features of the text content, thereby avoiding the text embedding features from guiding the generation of the initial description text, thereby ensuring the accuracy of the initial description text.

[0162] Exemplarily, a multimodal model based on language image pre-training is used as the second image-to-text model, wherein the multimodal model is an algorithm or model for generating text descriptions based on images.

[0163] See also Figure 4 , Figure 4 It is a structural schematic diagram of the second image-to-text model provided by the present invention: the second image-to-text model is a model framework that unifies the three tasks of image-text similarity, image-text matching, and image-generated text. In the process of training the second image-to-text model, these three tasks are performed simultaneously, thereby improving the feature quality of the second image-to-text model, that is, improving the quality of the initial description text generated by the second image-to-text model. In order to distinguish the text inputs of the three tasks, [CLS] is added before the text input of the image-text similarity task to summarize the sentence, and [Encode] is added before the text input of the image-text matching task as a joint representation of the image and text, and [Decode] is added before the text input of the image-generated text task as the starting point for generating the result.

[0164] For the image-text similarity task, during training, a visual transformer is used to encode the image (sample image) to obtain an image embedding feature (Embedding) with a dimension of 1x768. The visual transformer is a model or module that introduces the Transformer into the image field. Six Transformer Blocks are used to extract features from the text (sample description text) to obtain a text embedding feature (Embedding) with a dimension of 1x768. The text embedding and the image embedding are dot-producted to obtain a 1x1 Logit, which is the predicted similarity. The Logit and the image-text similarity label are calculated using a binary cross entropy loss to obtain the Loss (loss value), and the second image-text model is gradient optimized. Two methods are used during training. When the image and text match, the Label (similarity label) is 1, and when the image and text do not match, the Label is 0.

[0165] For the image-text matching task, in actual application, the visual transformer is used to encode the image to obtain an image embedding feature (Embedding) with a dimension of 1x768. Six Transformer Blocks are used to extract text features to obtain a text embedding feature (Embedding) with a dimension of 1x768. Then the Cross Attention (a multimodal feature fusion method) attention mechanism is used for feature fusion to fuse the image Embedding and the text Embedding, and the initial fused Embedding with a dimension of 1x768 is output. After a fully connected layer, the target fused Embedding with a dimension of 1x2 is obtained. The target fused Embedding is activated by the softmax function to obtain a probability value of 0 or 1, which is used to determine whether the image and text match. A probability value of 0 indicates a mismatch, and a probability value of 1 indicates a match.

[0166] For the task of generating text from image, in actual application, the image (target image) is subjected to the visual transformer (image encoder) for image feature extraction, and an image embedding feature (Embedding) with a dimension of 1x768 is obtained. The text (specified text) is subjected to 6 Transformer Blocks for text feature extraction, and a text embedding feature (Embedding) with a dimension of 1x768 is obtained. The image embedding and text embedding are fused through the Cross Attention mechanism to obtain the fused embedding feature (Embedding). For the fused embedding, the autoregressive method is used to generate text features, and each time the output dimension is 1xM (M is the number of characters in the character library or character table). The logit is activated by the softmax function to obtain the probability of each character. Then random sampling is performed according to the probability, and a character is randomly selected from the character library as the output result, that is, the sampling result. Due to the randomness, the output result may be different each time. Then this character is used as the specified text and is input into the image-text matching task of the second image-to-text model together with the target text, and a character is output again. The newly generated text is concatenated with the previous text to form a new specified text, and the target text is continuously input into the image-text matching task until the output result is the [STOP] symbol, and the previously generated text is used as the final generated initial description text. When generating the initial text description, only the image-generated text capability of the second image-to-text model is used.

[0167] Optionally, the target image to be described is input into a pre-trained third image-to-text model to obtain an initial description text corresponding to the target image. The specific implementation process may be as follows:

[0168] Inputting the target image into the image encoder of the third image-to-text model for image encoding to obtain a third image embedding feature, and inputting the empty text into the text encoder of the third image-to-text model for text encoding to obtain a third text embedding feature;

[0169] Concatenate the third image embedding feature and the third text embedding feature through a separator to obtain a total embedding feature;

[0170] The total embedded features are input into the feature fusion device of the third image-to-text model for feature fusion to obtain the initial description text corresponding to the target image.

[0171] Specifically, the third image-to-text model can be a generative vision and language transformer. The separator is a [SEP] symbol. The feature fuser refers to a unit that performs feature fusion in the image-to-text model, such as composed of 6 Transformer Blocks.

[0172] In practical applications, the image encoder of the third image-to-text model extracts features from the target image and obtains a third image embedding feature (Embedding) with a dimension of Nx768, where N is the number of Tokens (sub-images of the target image); the text encoder of the third image-to-text model extracts features from empty text and obtains a third text embedding feature (Embedding) with a dimension of Mx768, where M is the number of Tokens (characters).

[0173] Furthermore, the third image embedding feature and the third text embedding feature are concatenated through a separator to obtain a total embedding feature. The total embedding feature is then input into the feature fusion device for feature fusion. After that, the feature fusion device outputs the initial description text corresponding to the target image: the feature fusion device directly outputs a matrix of dimensions C x F, where C represents the length of the output text, that is, the number of words contained in the initial description text, which can be set as needed; F is the number of vocabularies. Then, the probability distribution is obtained through the Softmax function, and the word with the highest probability in the dimension F is taken to obtain the output initial text description. In this way, the efficiency of obtaining the initial description text can be improved.

[0174] See also Figure 5 , Figure 5It is a structural schematic diagram of the third image-to-text model provided by the present invention. The target image is subjected to feature extraction by a visual transformer (image encoder) to obtain a third image embedding feature with a dimension of Nx768. Next, the empty text is subjected to feature extraction by a bidirectional encoding model to obtain a third text embedding feature with a dimension of Mx768. Then, the third text embedding feature and the third image embedding feature are concatenated, separated by the [SEP] symbol in the middle, to obtain the total embedding feature. The total embedding feature is subjected to feature fusion by 6 Transformer Blocks, and a matrix of C x F dimensions is directly output. Then, the probability distribution is obtained by the Softmax function, and the word with the highest probability in the dimension F is taken to obtain the output initial text description.

[0175] In one or more optional embodiments of the present invention, before performing image text recognition on the target image to be described to obtain a plurality of initial description texts corresponding to the target image, the method further includes:

[0176] Acquire a plurality of sample image and text objects, wherein the sample image and text objects include sample images and descriptive text labels corresponding to the sample images;

[0177] Randomly extracting a first specified number of sample graphic objects from each of the sample graphic objects to obtain a first graphic object set, randomly extracting a second specified number of sample graphic objects from each of the sample graphic objects to obtain a second graphic object set, and randomly extracting a third specified number of sample graphic objects from each of the sample graphic objects to obtain a third graphic object set;

[0178] The first prediction model is trained according to the first image-text object set to obtain the first image-text model, the second prediction model is trained according to the second image-text object set to obtain the second image-text model, and the third prediction model is trained according to the third image-text object set to obtain the third image-text model.

[0179] Specifically, the sample image-text object refers to the image-text data used for training. The sample image refers to the image used for training the image-text model. The descriptive text label refers to the text describing the sample image. The specified quantity refers to a preset value. The first specified quantity, the second specified quantity, and the third specified quantity may be the same or different. The first prediction model refers to the first image-text model before training. The second prediction model refers to the second image-text model before training. The third prediction model refers to the third image-text model before training.

[0180] In practical applications, a plurality of sample graphic objects are first obtained, each of which includes a sample image and a descriptive text label corresponding to the sample image. Then, a first specified number of sample graphic objects are randomly selected from all sample graphic objects to form a first graphic object set; a second specified number of sample graphic objects are randomly selected from each sample graphic object to form a second graphic object set; and a third specified number of sample graphic objects are randomly selected from each sample graphic object to form a third graphic object set. In this way, while minimizing the number of sample graphic objects obtained, random sampling is used to ensure that the first graphic object set, the second graphic object set, and the third graphic object set are different.

[0181] Exemplarily, 10,000 sample graphic objects are obtained, 9,000 (first specified number) sample graphic objects are randomly selected from the 10,000 sample graphic objects to train the first prediction model, 7,000 (second specified number) sample graphic objects are randomly selected from the 10,000 sample graphic objects to train the second prediction model, and 8,000 (third specified number) sample graphic objects are randomly selected from the 10,000 sample graphic objects to train the third prediction model.

[0182] In addition, the first designated quantity may be determined by the first percentage, the second designated quantity may be determined by the second percentage, and the third designated quantity may be determined by the third percentage. For example, if the first to third percentages are all 80%, then 80% of the 10,000 sample graphic objects are randomly sampled, that is, 8,000 sample graphic objects are randomly selected to train the first prediction model, and 80% of the 10,000 sample graphic objects are randomly sampled, that is, 8,000 sample graphic objects are randomly selected to train the second prediction model, and then 80% of the 10,000 sample graphic objects are randomly sampled, that is, 8,000 sample graphic objects are randomly selected to train the third prediction model.

[0183] Combine the following Figure 6-Figure 9 The description text generation method provided by the present invention is further described. Figure 6 This is the second flow chart of the description text generation method provided by the present invention. Figure 6 As shown, it mainly includes three steps: generating text description of the target image through multiple models, removing text duplication, and adding style identification (style tag) to obtain the target description text.

[0184] In the same batch of image and text data (sample image and text objects), 80% is randomly sampled three or more times to obtain different image and text data sets, i.e., multiple image and text object sets, and three or more different image-generated text models (image-generated text models) are trained. The trained image-generated text models are used to generate different initial text descriptions for the target image, i.e., multiple models generate text descriptions. Since the training data (image and text object sets) of each image-generated text model are different and the model structure is different, the initial text descriptions generated for the same target image are also slightly different.

[0185] like Figure 7 The flowchart of the multi-model text description generation provided by the present invention is shown as follows: the target image is input into three different image-generated text models: the first image-generated text model, the second image-generated text model and the third image-generated text model, and the initial text description output by each image-generated text model is obtained. Then, the image-text similarity prediction model is used to calculate the similarity between the target image and each initial text description, and the first image-text similarity between each initial text description and the target image is obtained. For each first image-text similarity, it is determined whether it is greater than 0.5 (the first image-text similarity threshold). If the first image-text similarity is greater than 0.5, the initial description text corresponding to the first image-text similarity is retained, otherwise the initial description text corresponding to the first image-text similarity is discarded. Finally, the filtered initial text description text is obtained, and then the text is deduplicated.

[0186] like Figure 8 The flowchart of text deduplication provided by the present invention is shown as follows: Text deduplication adopts a transformer-based bidirectional encoding model. First, sort the filtered multiple initial text descriptions in descending order according to the first image-text similarity, and use the bidirectional encoding model to calculate the text similarity of all initial text descriptions in pairs. For each text similarity, determine whether it is 0.9 (text similarity threshold). If the text similarity is greater than 0.9, then for the two initial description texts corresponding to the text similarity, remove the initial description text with a low first image-text similarity, and only retain the initial description text with a high first image-text similarity; if the text similarity is not greater than 0.9, then retain the two initial description texts corresponding to the text similarity, that is, retain both. After deduplication, splice the remaining initial description texts with commas as intervals to obtain a backup description text.

[0187] like Fig. 9The flow chart of adding style identifiers provided by the present invention is shown as follows: prepare an identifier library in advance, which contains some collected style identifiers, such as art style, cyberpunk, steampunk and science fiction style identifiers (style tags). Then input the style identifiers and the target image in the identifier library into the image-text similarity prediction model to calculate the similarity, and obtain the second image-text similarity between each style identifier and the target image, and determine whether each second image-text similarity is greater than 0.5 (second image-text similarity threshold): If the second image-text similarity is greater than 0.5, it is considered that the image and text match, and the style identifier and the backup description text corresponding to the second image-text similarity are spliced ​​together, separated by commas, to obtain the final output target description text, otherwise the style identifier corresponding to the second image-text similarity is discarded, and the backup description text is directly output.

[0188] The description text generating device provided by the present invention is described below. The description text generating device described below and the description text generating method described above can be referred to each other.

[0189] Fig.10 is a schematic diagram of the structure of the description text generation device provided by the present invention, such as Fig.10 As shown, the description text generating device 1000 includes: an identification module 1001, a deduplication module 1002 and a generation module 1003, wherein:

[0190] The recognition module 1001 is configured to perform a plurality of different image text recognitions on a target image to be described, and obtain a plurality of initial description texts corresponding to the target image;

[0191] A deduplication module 1002 is configured to perform deduplication processing and splicing processing on each of the initial description texts to obtain a backup description text corresponding to the target image;

[0192] The generating module 1003 is configured to generate a target description text corresponding to the target image according to at least one pre-acquired initial style identifier and the backup description text.

[0193] The description text generation device provided by the present invention performs multiple different image text recognitions on the target image to be described to obtain multiple initial description texts corresponding to the target image; performs deduplication processing and splicing processing on each of the initial description texts to obtain a backup description text corresponding to the target image; generates a target description text corresponding to the target image according to at least one pre-acquired initial style identifier and the backup description text. By deduplication and splicing the generated multiple initial description texts, the target description text can more accurately and comprehensively describe the target image; generating the target description text according to the initial style identifier and the backup description text can make the target description text contain not only text describing the content of the target image, but also text describing the style of the target image, thereby improving the comprehensiveness and accuracy of the target description text in describing the target image; thereby forming a higher quality target image-text object to train a better text-image model. Corresponding high-quality description texts can also be generated for a large number of images.

[0194] In one or more optional embodiments of the present invention, the deduplication module 1002 is further configured to:

[0195] Calculating initial text similarities between the initial description texts, and determining target text similarities among the initial text similarities that are greater than a text similarity threshold;

[0196] For each of the two initial description texts corresponding to the target text similarity, the initial description text with the lower first image-text similarity is deleted, where the first image-text similarity is the similarity between the initial text and the target image;

[0197] The remaining initial description texts are spliced ​​to obtain backup description texts corresponding to the target image.

[0198] In one or more optional embodiments of the present invention, the generating module 1003 is further configured to:

[0199] Obtain at least one initial style identifier;

[0200] Calculating the second graphic similarity between each of the initial style identifiers and the target image;

[0201] According to each of the second image-text similarities, determining whether there is a target style identifier corresponding to the target image in each of the initial style identifiers;

[0202] If there is a target style identifier corresponding to the target image, concatenate the target style identifier corresponding to the target image with the backup description text to obtain the target description text corresponding to the target image;

[0203] If there is no target style identifier corresponding to the target image, the backup description text is determined as the target description text corresponding to the target image.

[0204] In one or more optional embodiments of the present invention, the description text generating device 1000 further includes a deletion module configured to:

[0205] Calculating the similarity between each of the initial description texts and the first image-text of the target image;

[0206] The initial description text whose first image-text similarity is lower than a first image-text similarity threshold is deleted.

[0207] In one or more optional embodiments of the present invention, the identification module 1001 is further configured to:

[0208] Inputting a target image to be described into a pre-trained first image-to-text model to obtain an initial description text corresponding to the target image;

[0209] Inputting the target image to be described into the pre-trained second image-to-text model to obtain an initial description text corresponding to the target image;

[0210] Inputting the target image to be described into the pre-trained third image-to-text model to obtain an initial description text corresponding to the target image;

[0211] The first image-to-text model, the second image-to-text model and the third image-to-text model are trained based on different image-to-text object sets.

[0212] In one or more optional embodiments of the present invention, the identification module 1001 is further configured to:

[0213] Inputting the target image into the image encoder of the first image-to-text model for image encoding to obtain a first image embedding feature, and inputting empty text into the text encoder of the first image-to-text model for text encoding to obtain a first text embedding feature;

[0214] The first image embedding feature is input into the multimodal text encoder of the first image-to-text model through the attention mechanism of the first image-to-text model, and the first text embedding feature is input into the multimodal text encoder of the first image-to-text model to perform feature fusion to obtain an initial description text corresponding to the target image.

[0215] In one or more optional embodiments of the present invention, the identification module 1001 is further configured to:

[0216] Step 1: Input the target image into the image encoder of the second image-to-text model for image encoding to obtain a second image embedding feature, and input the specified text into the text encoder of the second image-to-text model for text encoding to obtain a second text embedding feature, wherein the specified text is empty text;

[0217] Step 2: Analyze and process each character in the character library according to the second image embedding feature and the second text embedding feature to determine the character feature of each character;

[0218] Step 3: Determine the probability of each character according to the characteristics of each character, and randomly sample each character based on the probability to obtain a sampling result;

[0219] Step 4: concatenate the sampling result and the designated text to obtain an updated designated text;

[0220] Continue to execute step 1 to step 4 until the sampling result is a stop symbol or a blank, and determine the designated text as the initial description text corresponding to the target image.

[0221] In one or more optional embodiments of the present invention, the identification module 1001 is further configured to:

[0222] Inputting the target image into the image encoder of the third image-to-text model for image encoding to obtain a third image embedding feature, and inputting the empty text into the text encoder of the third image-to-text model for text encoding to obtain a third text embedding feature;

[0223] Concatenate the third image embedding feature and the third text embedding feature through a separator to obtain a total embedding feature;

[0224] The total embedded features are input into the feature fusion device of the third image-to-text model for feature fusion to obtain the initial description text corresponding to the target image.

[0225] In one or more optional embodiments of the present invention, the description text generating device 1000 further includes a training module configured to:

[0226] Acquire a plurality of sample image and text objects, wherein the sample image and text objects include sample images and descriptive text labels corresponding to the sample images;

[0227] Randomly extracting a first specified number of sample graphic objects from each of the sample graphic objects to obtain a first graphic object set, randomly extracting a second specified number of sample graphic objects from each of the sample graphic objects to obtain a second graphic object set, and randomly extracting a third specified number of sample graphic objects from each of the sample graphic objects to obtain a third graphic object set;

[0228] The first prediction model is trained according to the first image-text object set to obtain the first image-text model, the second prediction model is trained according to the second image-text object set to obtain the second image-text model, and the third prediction model is trained according to the third image-text object set to obtain the third image-text model.

[0229] Fig.11 An example of a physical structure diagram of an electronic device is shown in FIG. Fig.11 As shown, the electronic device may include: a processor 1110, a communication interface 1120, a memory 1130 and a communication bus 1140, wherein the processor 1110, the communication interface 1120 and the memory 1130 communicate with each other through the communication bus 1140. The processor 1110 may call the logic instructions in the memory 1130 to execute the description text generation method, which includes: performing a plurality of different image text recognitions on the target image to be described to obtain a plurality of initial description texts corresponding to the target image; performing deduplication processing and splicing processing on each of the initial description texts to obtain a backup description text corresponding to the target image; and generating a target description text corresponding to the target image according to at least one initial style identifier obtained in advance and the backup description text.

[0230] In addition, the logic instructions in the above-mentioned memory 1130 can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0231] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the description text generation method provided by the above-mentioned methods, which includes: performing a variety of image text recognition on a target image to be described to obtain a plurality of initial description texts corresponding to the target image; performing deduplication processing and splicing processing on each of the initial description texts to obtain a backup description text corresponding to the target image; and generating a target description text corresponding to the target image based on at least one initial style identifier obtained in advance and the backup description text.

[0232] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, it is implemented to execute the description text generation method provided by the above-mentioned methods, the method comprising: performing a plurality of different image text recognitions on a target image to be described to obtain a plurality of initial description texts corresponding to the target image; performing deduplication processing and splicing processing on each of the initial description texts to obtain a backup description text corresponding to the target image; and generating a target description text corresponding to the target image based on at least one initial style identifier obtained in advance and the backup description text.

[0233] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Those of ordinary skill in the art may understand and implement it without creative work.

[0234] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0235] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A description text generation method, characterized in that: include: Performing multiple different image text recognitions on the target image to be described to obtain multiple initial description texts corresponding to the target image; Deduplication processing and splicing processing are performed on each of the initial description texts to obtain a backup description text corresponding to the target image; A target description text corresponding to the target image is generated according to at least one pre-acquired initial style identifier and the backup description text.

2. The description text generation method according to claim 1, characterized in that: The performing of deduplication processing and splicing processing on each of the initial description texts to obtain a backup description text corresponding to the target image includes: Calculating initial text similarities between the initial description texts, and determining target text similarities among the initial text similarities that are greater than a text similarity threshold; For each of the two initial description texts corresponding to the target text similarity, the initial description text with the lower first image-text similarity is deleted, where the first image-text similarity is the similarity between the initial text and the target image; The remaining initial description texts are spliced ​​to obtain backup description texts corresponding to the target image.

3. The description text generation method according to claim 1, characterized in that: Before generating the target description text corresponding to the target image according to the at least one initial style identifier and the backup description text acquired in advance, the method further includes: Obtain at least one initial style identifier; Calculating the second graphic similarity between each of the initial style identifiers and the target image; According to each of the second image-text similarities, determining whether there is a target style identifier corresponding to the target image in each of the initial style identifiers; Accordingly, generating the target description text corresponding to the target image according to the at least one pre-acquired initial style identifier and the backup description text includes: If there is a target style identifier corresponding to the target image, concatenate the target style identifier corresponding to the target image with the backup description text to obtain the target description text corresponding to the target image; If there is no target style identifier corresponding to the target image, the backup description text is determined as the target description text corresponding to the target image.

4. The description text generation method according to claim 1, characterized in that: Before performing deduplication processing and splicing processing on the initial description texts to obtain the backup description text corresponding to the target image, the method further includes: Calculating the similarity between each of the initial description texts and the first image-text of the target image; The initial description text whose first image-text similarity is lower than a first image-text similarity threshold is deleted.

5. The description text generation method according to any one of claims 1 to 4, characterized in that: The method performs a plurality of different image text recognitions on the target image to be described to obtain a plurality of initial description texts corresponding to the target image, including: Inputting a target image to be described into a pre-trained first image-to-text model to obtain an initial description text corresponding to the target image; Inputting the target image to be described into the pre-trained second image-to-text model to obtain an initial description text corresponding to the target image; Inputting the target image to be described into the pre-trained third image-to-text model to obtain an initial description text corresponding to the target image; The first image-to-text model, the second image-to-text model and the third image-to-text model are trained based on different image-to-text object sets.

6. The description text generation method according to claim 5, characterized in that: The step of inputting the target image to be described into the pre-trained first image-to-text model to obtain the initial description text corresponding to the target image includes: Inputting the target image into the image encoder of the first image-to-text model for image encoding to obtain a first image embedding feature, and inputting empty text into the text encoder of the first image-to-text model for text encoding to obtain a first text embedding feature; The first image embedding feature is input into the multimodal text encoder of the first image-to-text model through the attention mechanism of the first image-to-text model, and the first text embedding feature is input into the multimodal text encoder of the first image-to-text model to perform feature fusion to obtain an initial description text corresponding to the target image.

7. The description text generation method according to claim 5, characterized in that: The step of inputting the target image to be described into the pre-trained second image-to-text model to obtain the initial description text corresponding to the target image includes: Step 1: Input the target image into the image encoder of the second image-to-text model for image encoding to obtain a second image embedding feature, and input the specified text into the text encoder of the second image-to-text model for text encoding to obtain a second text embedding feature, wherein the specified text is empty text; Step 2: Analyze and process each character in the character library according to the second image embedding feature and the second text embedding feature to determine the character feature of each character; Step 3: Determine the probability of each character according to the characteristics of each character, and randomly sample each character based on the probability to obtain a sampling result; Step 4: concatenate the sampling result and the designated text to obtain an updated designated text; Continue to execute step 1 to step 4 until the sampling result is a stop symbol or a blank, and determine the designated text as the initial description text corresponding to the target image.

8. The description text generation method according to claim 5, characterized in that: The step of inputting the target image to be described into the pre-trained third image-to-text model to obtain the initial description text corresponding to the target image includes: Inputting the target image into the image encoder of the third image-to-text model for image encoding to obtain a third image embedding feature, and inputting the empty text into the text encoder of the third image-to-text model for text encoding to obtain a third text embedding feature; Concatenate the third image embedding feature and the third text embedding feature through a separator to obtain a total embedding feature; The total embedded features are input into the feature fusion device of the third image-to-text model for feature fusion to obtain the initial description text corresponding to the target image.

9. The description text generation method according to claim 5, characterized in that: Before performing image text recognition on the target image to be described to obtain a plurality of initial description texts corresponding to the target image, the method further includes: Acquire a plurality of sample image and text objects, wherein the sample image and text objects include sample images and descriptive text labels corresponding to the sample images; Randomly extracting a first specified number of sample graphic objects from each of the sample graphic objects to obtain a first graphic object set, randomly extracting a second specified number of sample graphic objects from each of the sample graphic objects to obtain a second graphic object set, and randomly extracting a third specified number of sample graphic objects from each of the sample graphic objects to obtain a third graphic object set; The first prediction model is trained according to the first image-text object set to obtain the first image-text model, the second prediction model is trained according to the second image-text object set to obtain the second image-text model, and the third prediction model is trained according to the third image-text object set to obtain the third image-text model.

10. A description text generation device, characterized in that: include: A recognition module is configured to perform a plurality of different image text recognitions on a target image to be described, and obtain a plurality of initial description texts corresponding to the target image; A deduplication module is configured to perform deduplication processing and splicing processing on each of the initial description texts to obtain a backup description text corresponding to the target image; The generating module is configured to generate a target description text corresponding to the target image according to at least one pre-acquired initial style identifier and the backup description text.

Images