Description generation model training method and device, description generation method and device and electronic equipment

Abstract

The embodiment of the invention provides a description generation model training method, a description generation method, a description generation device and electronic equipment, and relates to the technical field of computer vision, and the description generation model training method comprises the following steps: processing each first sample image by utilizing a first text prompt and a description generation model, obtaining a description text of a specified image area in the first sample image as a first sample description text; determining a second sample description text matched with a specified image area in the corresponding first sample image from each first sample description text; inputting the first sample image corresponding to each second sample description text and the first text prompt into a description generation model to obtain a first prediction description text of a specified image area in the first sample image; based on the difference between the first prediction description text and the second sample description text, parameters of the description generation model are adjusted, and a new description generation model is obtained. The description text of the image area can be accurately generated.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to a description generation model training method, a description generation method, a device and an electronic device. Background Art

[0002] With the development of image processing technology, in some application scenarios (such as text search or image retrieval, etc.), it is often necessary to generate description text for a specified image area in an image. The accuracy of the generated description text of the image area has a significant impact on the application effect. Therefore, how to accurately generate description text of the image area has become an urgent problem to be solved. Summary of the invention

[0003] The purpose of the embodiments of the present application is to provide a description generation model training method, description generation method, device and electronic device to accurately generate description text of an image area. The specific technical solution is as follows:

[0004] In a first aspect of an embodiment of the present application, a description generation model training method is provided, the method comprising:

[0005] Using the first text prompt and the current description generation model, each current first sample image is processed to obtain a description text for describing a specified image area in the first sample image as the current first sample description text; wherein the first text prompt is used to instruct the generation of a description text for the specified image area in the input image;

[0006] Determine, from each of the current first sample description texts, a description text that matches the designated image area in the corresponding first sample image as the current second sample description text;

[0007] Inputting the first sample image and the first text prompt corresponding to each current second sample description text into the current description generation model to obtain the description text of the specified image area in the first sample image as the current first predicted description text;

[0008] Based on the difference between the current first predicted description text and the current second sample description text, the parameters of the current description generation model are adjusted to obtain a new description generation model.

[0009] Optionally, determining, from the current first sample description texts, a description text that matches a specified image area in the corresponding first sample image as the current second sample description text includes:

[0010] Input each current first sample image, the first sample description text corresponding to the first sample image, and the second text prompt into a pre-trained reward model to obtain a matching result between a specified image area in the first sample image and the corresponding first sample description text; wherein the second text prompt is used to indicate whether the specified image area in the input image matches the input description text; the reward model is pre-trained based on positive sample pairs and negative sample pairs; a positive sample pair includes: a second sample image, a specified image area in the second sample image, and a description text that matches the specified image area in the second sample image; a negative sample pair includes: a third sample image, a specified image area in the third sample image, and a description text that does not match the specified image area in the third sample image;

[0011] When the obtained matching result indicates that the designated image area in the first sample image matches the corresponding first sample description text, the first sample description text corresponding to the first sample image is determined as the corresponding second sample description text.

[0012] Optionally, the reward model includes: a first large language model, a first adapter, a first image encoder and a first tag parser; the first image encoder is used to extract features of an input image; the first adapter is used to convert image features output by the first image encoder into a tag form; the first tag parser is used to convert input text into a tag form.

[0013] Optionally, the description text in each sample pair is obtained by processing the sample image in the sample pair using the first text prompt and the current description generation model; and the reward model is obtained by training through the following steps:

[0014] For each sample pair, the sample image and the specified image region in the sample pair are input into the first image encoder in the reward model of the initial structure to obtain the features of the sample image and the features of the specified image region in the sample pair, and the obtained features are spliced;

[0015] Input the obtained splicing result into the first adapter in the reward model of the initial structure to obtain a marked form of the splicing result;

[0016] Inputting the second text prompt and the description text in the sample pair into the first token parser in the reward model of the initial structure to obtain the token form of the second text prompt and the token form of the description text in the sample pair;

[0017] Inputting the marked form of the splicing result, the marked form of the second text prompt, and the marked form of the description text in the sample pair into the first language model in the reward model of the initial structure, and obtaining a predicted matching result between the specified image area in the sample pair and the description text;

[0018] Based on the difference between the predicted matching result and the actual matching result of the sample pair, the parameters of the reward model of the initial structure are adjusted until the preset convergence condition is reached to obtain a trained reward model.

[0019] Optionally, the using the first text prompt and the current description generation model to process each current first sample image to obtain a description text for describing a specified image area in the first sample image as the current first sample description text includes:

[0020] For each current first sample image, when the proportion of the designated image area in the first sample image is less than a preset threshold, the designated image area in the first sample image is expanded to obtain an image to be processed;

[0021] Inputting the obtained image to be processed and the first text prompt into the current description generation model to obtain a description text for describing the specified image area in the first sample image as the current first sample description text;

[0022] When the proportion of the specified image area in the first sample image is not less than the preset threshold, the first sample image and the first text prompt are input into the current description generation model to obtain a description text for describing the specified image area in the first sample image as the current first sample description text.

[0023] Optionally, the using the first text prompt and the current description generation model to process each current first sample image to obtain a description text for describing a specified image area in the first sample image as the current first sample description text includes:

[0024] The first sample image is processed according to different sampling parameters using the first text prompt and the current description generation model to obtain a plurality of description texts of a designated image area in the first sample image as the current first sample description text.

[0025] Optionally, after adjusting the parameters of the current description generation model based on the difference between the current first predicted description text and the current second sample description text to obtain a new description generation model, the method further includes:

[0026] Acquire a fourth sample image; wherein, among the description texts generated by the current description generation model for each fourth sample image, there are description texts that do not match the specified image area in the corresponding fourth sample image;

[0027] Using the first text prompt and the current description generation model, each fourth sample image obtained is processed according to different sampling parameters to obtain a plurality of description texts for a specified image area in the fourth sample image;

[0028] Determine, from the obtained description texts, at least one positive sample description text that matches the specified image region in the fourth sample image, and at least one negative sample description text that does not match the specified image region in the fourth sample image;

[0029] Inputting the fourth sample image and the first text prompt into a current description generation model to obtain a description text of a specified image area in the fourth sample image as a current second predicted description text;

[0030] Based on the difference between the current second predicted description text and the determined positive sample description text, and the difference between the current second predicted description text and the determined negative sample description text, the parameters of the current description generation model are adjusted to obtain a new description generation model.

[0031] Optionally, the first text prompt is further used to indicate that: the current description generation model generates a description text that conforms to a first specified description style; a description style is used to indicate the language of the description text and / or the number of sentences included;

[0032] After adjusting the parameters of the current description generation model based on the difference between the current first predicted description text and the current second sample description text to obtain a new description generation model, the method further includes:

[0033] Using the first text prompt and the current description generation model, each current first sample image is processed to obtain a description text of the first specified description style for a specified image area in the first sample image as the current third sample description text;

[0034] Adjusting the description style of the current third sample description text from the first specified description style to the second specified description style to obtain a current fourth sample description text;

[0035] Determine, from the current fourth sample description text, a description text that matches the designated image area in the corresponding first sample image, to obtain the current fifth sample description text;

[0036] Inputting the first sample image and the third text prompt corresponding to each fifth sample description text into the current description generation model, obtaining the description text of the second specified description style of the specified image area in the first sample image as the current third predicted description text;

[0037] Based on the difference between the current third predicted description text and the current fifth sample description text, the parameters of the current description generation model are adjusted to obtain a new description generation model.

[0038] Optionally, after adjusting the parameters of the current description generation model based on the difference between the current third predicted description text and the current fifth sample description text to obtain a new description generation model, the method further includes:

[0039] Input each current first sample image and the third text prompt into the current description generation model to obtain a description text of the second specified description style for a specified image area in the first sample image as the current fourth sample description text;

[0040] Return to the step of determining, from the current fourth sample description text, a description text that matches the designated image area in the corresponding first sample image to obtain the current fifth sample description text, until the first preset condition is met to obtain a new description generation model.

[0041] Optionally, after adjusting the parameters of the current description generation model based on the difference between the current first predicted description text and the current second sample description text to obtain a new description generation model, the method further includes:

[0042] Return to the step of using the first text prompt and the current description generation model to process each current first sample image to obtain a description text for describing the specified image area in the first sample image as the current first sample description text, until the second preset condition is met and a new description generation model is obtained.

[0043] Optionally, the description generation model includes: a second large language model, a second adapter, a second image encoder and a second token parser; the second image encoder is used to extract features of an input image; the second adapter is used to convert image features output by the second image encoder into a token form; the second token parser is used to convert input text into a token form.

[0044] A second aspect of the embodiments of the present application further provides a description generation method, the method comprising:

[0045] Obtaining images to be used and text prompts to be used;

[0046] The image to be used and the text prompt to be used are input into a pre-trained description generation model to obtain a description text of the image area indicated by the text prompt to be used in the image to be used; wherein the description generation model is trained based on any of the above-mentioned description generation model training methods.

[0047] Optionally, the position of the image area indicated by the to-be-used text prompt in the to-be-used image is obtained by the following steps:

[0048] Displaying the image to be used;

[0049] In response to a user's frame selection operation on the displayed image to be used, a position indicated by the frame selection operation in the image to be used is determined, and a position of an image area indicated by the text prompt to be used in the image to be used is obtained.

[0050] Optionally, the description style of the obtained description text is the description style indicated by the text prompt to be used; a description style is used to indicate the language of the description text and / or the number of sentences included.

[0051] According to a third aspect of the embodiments of the present application, a description generation model training device is provided, the device comprising:

[0052] A first sample description text generation module is used to process each current first sample image using the first text prompt and the current description generation model to obtain a description text for describing a specified image area in the first sample image as the current first sample description text; wherein the first text prompt is used to indicate the generation of a description text for the specified image area in the input image;

[0053] A second sample description text determination module is used to determine, from the current first sample description texts, a description text that matches the designated image area in the corresponding first sample image as the current second sample description text;

[0054] A first predicted description text generation module, used for inputting the first sample image and the first text prompt corresponding to each current second sample description text into the current description generation model, and obtaining the description text of the specified image area in the first sample image as the current first predicted description text;

[0055] The first parameter adjustment module is used to adjust the parameters of the current description generation model based on the difference between the current first predicted description text and the current second sample description text to obtain a new description generation model.

[0056] Optionally, the second sample description text determination module is specifically used to input each current first sample image, the first sample description text corresponding to the first sample image, and the second text prompt into a pre-trained reward model to obtain a matching result between the specified image area in the first sample image and the corresponding first sample description text; wherein the second text prompt is used to indicate whether the specified image area in the input image matches the input description text; the reward model is pre-trained based on positive sample pairs and negative sample pairs; a positive sample pair includes: a second sample image, a specified image area in the second sample image, and a description text that matches the specified image area in the second sample image; a negative sample pair includes: a third sample image, a specified image area in the third sample image, and a description text that does not match the specified image area in the third sample image; when the matching result obtained indicates that the specified image area in the first sample image matches the corresponding first sample description text, the first sample description text corresponding to the first sample image is determined as the corresponding second sample description text.

[0057] Optionally, the reward model includes: a first large language model, a first adapter, a first image encoder and a first tag parser; the first image encoder is used to extract features of an input image; the first adapter is used to convert image features output by the first image encoder into a tag form; the first tag parser is used to convert input text into a tag form.

[0058] Optionally, the description text in each sample pair is obtained by processing the sample image in the sample pair using the first text prompt and the current description generation model; the device further includes:

[0059] The reward model training module is used to input the sample image and the designated image area in the sample pair into the first image encoder in the reward model of the initial structure for each sample pair, obtain the features of the sample image and the designated image area in the sample pair, and splice the obtained features; input the obtained splicing result into the first adapter in the reward model of the initial structure to obtain the marked form of the splicing result; input the second text prompt and the description text in the sample pair into the first mark parser in the reward model of the initial structure to obtain the marked form of the second text prompt and the marked form of the description text in the sample pair; input the marked form of the splicing result, the marked form of the second text prompt and the marked form of the description text in the sample pair into the first large language model in the reward model of the initial structure to obtain the predicted matching result between the designated image area in the sample pair and the description text; based on the difference between the obtained predicted matching result and the actual matching result of the sample pair, adjust the parameters of the reward model of the initial structure until the preset convergence condition is reached to obtain a trained reward model.

[0060] Optionally, the first sample description text generation module is specifically used to, for each current first sample image, when the proportion of the specified image area in the first sample image is less than a preset threshold, expand the specified image area in the first sample image to obtain an image to be processed; input the obtained image to be processed and the first text prompt into the current description generation model to obtain a description text for describing the specified image area in the first sample image as the current first sample description text; when the proportion of the specified image area in the first sample image is not less than the preset threshold, input the first sample image and the first text prompt into the current description generation model to obtain a description text for describing the specified image area in the first sample image as the current first sample description text.

[0061] Optionally, the first sample description text generation module is specifically used to utilize the first text prompt and the current description generation model to process the first sample image according to different sampling parameters to obtain multiple description texts of the specified image area in the first sample image as the current first sample description text.

[0062] Optionally, the device further comprises:

[0063] A fourth sample image acquisition module, configured to acquire a fourth sample image after adjusting the parameters of the current description generation model based on the difference between the current first predicted description text and the current second sample description text to obtain a new description generation model; wherein, among the description texts generated by the current description generation model for each fourth sample image, there are description texts that do not match the specified image area in the corresponding fourth sample image;

[0064] A description text inference module, used to process each fourth sample image acquired according to different sampling parameters by using the first text prompt and the current description generation model, to obtain a plurality of description texts of a specified image area in the fourth sample image;

[0065] a positive and negative sample description text determination module, used to determine, from the obtained description texts, at least one positive sample description text that matches the specified image area in the fourth sample image, and at least one negative sample description text that does not match the specified image area in the fourth sample image;

[0066] A second predicted description text generation module, used for inputting the fourth sample image and the first text prompt into a current description generation model to obtain a description text of a specified image area in the fourth sample image as a current second predicted description text;

[0067] The second parameter adjustment module is used to adjust the parameters of the current description generation model based on the difference between the current second predicted description text and the determined positive sample description text, and the difference between the current second predicted description text and the determined negative sample description text to obtain a new description generation model.

[0068] Optionally, the first text prompt is further used to indicate that: the current description generation model generates a description text that conforms to a first specified description style; a description style is used to indicate the language of the description text and / or the number of sentences included;

[0069] The device also includes:

[0070] A third sample description text generation module is used for adjusting the parameters of the current description generation model based on the difference between the current first predicted description text and the current second sample description text to obtain a new description generation model, and then using the first text prompt and the current description generation model to process each current first sample image to obtain a description text of the first specified description style for a specified image area in the first sample image as the current third sample description text;

[0071] a fourth sample description text generating module, configured to adjust the description style of the current third sample description text from the first specified description style to the second specified description style, to obtain a current fourth sample description text;

[0072] A fifth sample description text determination module determines, from the current fourth sample description text, a description text that matches the designated image area in the corresponding first sample image, to obtain the current fifth sample description text;

[0073] A third predicted description text generation module is used to input the first sample image and the third text prompt corresponding to each fifth sample description text into the current description generation model, and obtain the description text of the second specified description style of the specified image area in the first sample image as the current third predicted description text;

[0074] The third parameter adjustment model is used to adjust the parameters of the current description generation model based on the difference between the current third predicted description text and the current fifth sample description text to obtain a new description generation model.

[0075] Optionally, the device further comprises:

[0076] The fourth sample description text updating module is used to adjust the parameters of the current description generation model based on the difference between the current third predicted description text and the current fifth sample description text, and after obtaining the new description generation model, each current first sample image and the third text prompt are input into the current description generation model to obtain the description text of the second specified description style of the specified image area in the first sample image as the current fourth sample description text; trigger the fifth sample description text determination module until the first preset condition is met to obtain the new description generation model.

[0077] Optionally, the device further comprises:

[0078] A trigger module adjusts the parameters of the current description generation model based on the difference between the current first predicted description text and the current second sample description text to obtain a new description generation model, and then triggers the first sample description text generation module until a second preset condition is met to obtain a new description generation model.

[0079] Optionally, the description generation model includes: a second large language model, a second adapter, a second image encoder and a second token parser; the second image encoder is used to extract features of an input image; the second adapter is used to convert image features output by the second image encoder into a token form; the second token parser is used to convert input text into a token form.

[0080] According to a fourth aspect of the embodiments of the present application, a description generating device is provided, the device comprising:

[0081] A data acquisition module, used to acquire images to be used and text prompts to be used;

[0082] A description text generation module is used to input the image to be used and the text prompt to be used into a pre-trained description generation model to obtain a description text of the image area indicated by the text prompt to be used in the image to be used; wherein the description generation model is trained based on any of the description generation model training methods described above.

[0083] Optionally, the position of the image area indicated by the to-be-used text prompt in the to-be-used image is obtained by the following steps:

[0084] Displaying the image to be used;

[0085] In response to a user's frame selection operation on the displayed image to be used, a position indicated by the frame selection operation in the image to be used is determined, and a position of an image area indicated by the text prompt to be used in the image to be used is obtained.

[0086] Optionally, the description style of the obtained description text is the description style indicated by the text prompt to be used; a description style is used to indicate the language of the description text and / or the number of sentences included.

[0087] The present application also provides an electronic device, including:

[0088] Memory, used to store computer programs;

[0089] The processor is used to implement any of the above-mentioned description generation model training methods or description generation methods when executing the program stored in the memory.

[0090] An embodiment of the present application also provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, it implements any of the above-mentioned description generation model training methods or description generation methods.

[0091] An embodiment of the present application also provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute any of the description generation model training methods or description generation methods described above.

[0092] Beneficial effects of the embodiments of the present application:

[0093] A description generation model training method provided in an embodiment of the present application can use a first text prompt and a current description generation model to process each current first sample image to obtain a description text for describing a specified image area in the first sample image as the current first sample description text; wherein the first text prompt is used to indicate the generation of a description text for a specified image area in an input image; from each current first sample description text, a description text matching the specified image area in the corresponding first sample image is determined as the current second sample description text; the first sample image and the first text prompt corresponding to each current second sample description text are input into the current description generation model to obtain a description text for the specified image area in the first sample image as the current first predicted description text; based on the difference between the current first predicted description text and the current second sample description text, the parameters of the current description generation model are adjusted to obtain a new description generation model.

[0094] Based on the above processing, the latest description generation model can be used to generate a description text (i.e., a first sample description text) of a specified image area in a sample image. Due to the influence of the accuracy of the description generation model, the first sample description text inferred by the description generation model may not match the specified image area in the corresponding first sample image. Therefore, the description text (i.e., the second sample description text) that matches the specified image area in the corresponding first sample image can be further screened from the obtained first sample description text. Furthermore, each second sample description text can be used as a true value label, and the first sample image corresponding to each second sample description text can be used as a training sample to train the current description generation model to obtain a new description generation model. In this way, the description generation capability of the existing description generation model can be used to generate training samples, and the description generation capability of the existing description generation model can be further optimized to obtain a new description generation model. That is, the probability of the description generation model generating a description text that matches the specified image area in the input image is further improved. The accuracy of the description text of the image area generated by the description generation model can also be improved. Subsequently, the description text of the image area can be accurately generated using the trained description generation model.

[0095] Of course, implementing any product or method of the present application does not necessarily require achieving all of the advantages described above at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0096] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application, and for ordinary technicians in this field, other embodiments can also be obtained based on these drawings.

[0097] Figure 1 A first flow chart of a method for generating model training provided in an embodiment of the present application;

[0098] Figure 2 A second flow chart of a method for generating model training is provided in accordance with an embodiment of the present application;

[0099] Figure 3 A schematic diagram of a process for training a description generation model based on a reward model provided in an embodiment of the present application;

[0100] Figure 4a A schematic diagram of an image of an input reward model provided in an embodiment of the present application;

[0101] Figure 4b A schematic diagram of an image of another input reward model provided in an embodiment of the present application;

[0102] Figure 5 A flowchart of a training reward model provided in an embodiment of the present application;

[0103] Figure 6 A schematic diagram of a first process for directly optimizing the preference of the description generation model provided in an embodiment of the present application;

[0104] Figure 7 A second flow chart of directly optimizing the preference of the description generation model provided in the embodiment of the present application;

[0105] Figure 8 A schematic diagram of a process for iterating a description generation model provided in an embodiment of the present application;

[0106] Fig. 9 A flowchart of the ability of the training description generation model provided in an embodiment of the present application to generate description text of a specified description style;

[0107] Fig.10 A schematic diagram of an image including multiple image regions for which description texts of different description styles need to be generated provided in an embodiment of the present application;

[0108] Fig.11 A schematic diagram of capability reuse with different description styles provided in an embodiment of the present application;

[0109] Fig.12 A flowchart of a description generation method provided in an embodiment of the present application;

[0110] Fig.13 A flowchart of another description generation method provided in an embodiment of the present application;

[0111] Fig.14 A schematic diagram of the structure of a description generation model training device provided in an embodiment of the present application;

[0112] Fig.15 A schematic diagram of the structure of a description generation device provided in an embodiment of the present application;

[0113] Fig.16 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0114] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field based on the present application belong to the scope of protection of the present application.

[0115] With the development of image processing technology, in some application scenarios (such as text-based image search or image retrieval, etc.), it is often necessary to generate description text for a specified image area in an image. The accuracy of the generated description text for the image area has a significant impact on the application effect.

[0116] In order to accurately generate description text for an image region, the present application embodiment provides a description generation model training method, see Figure 1 , Figure 1 A first flow chart of a description generation model training method provided in an embodiment of the present application. The description generation model training method includes:

[0117] Step S101: using the first text prompt and the current description generation model, processing each current first sample image to obtain a description text for describing a specified image area in the first sample image as the current first sample description text.

[0118] The first text prompt is used to instruct the generation of a description text for a specified image area in the input image.

[0119] Step S102: Determine, from the current first sample description texts, a description text that matches the designated image area in the corresponding first sample image as the current second sample description text.

[0120] Step S103: inputting the first sample image and the first text prompt corresponding to each current second sample description text into the current description generation model to obtain the description text of the specified image area in the first sample image as the current first predicted description text.

[0121] Step S104: Based on the difference between the current first predicted description text and the current second sample description text, the parameters of the current description generation model are adjusted to obtain a new description generation model.

[0122] Based on the above processing, the latest description generation model can be used to generate a description text (i.e., a first sample description text) of a specified image area in a sample image. Due to the influence of the accuracy of the description generation model, the first sample description text inferred by the description generation model may not match the specified image area in the corresponding first sample image. Therefore, the description text (i.e., the second sample description text) that matches the specified image area in the corresponding first sample image can be further screened from the obtained first sample description text. Furthermore, each second sample description text can be used as a true value label, and the first sample image corresponding to each second sample description text can be used as a training sample to train the current description generation model to obtain a new description generation model. In this way, the description generation capability of the existing description generation model can be used to generate training samples, and the description generation capability of the existing description generation model can be further optimized to obtain a new description generation model. That is, the probability of the description generation model generating a description text that matches the specified image area in the input image is further improved. The accuracy of the description text of the image area generated by the description generation model can also be improved. Subsequently, the description text of the image area can be accurately generated using the trained description generation model.

[0123] For step S101, the description generation model may be a multimodal large language model, and may include an existing pre-trained large language model. For example, the description generation model may include: a large language model (i.e., the second large language model in the present application), an adapter (i.e., the second adapter in the present application), an image encoder (i.e., the second image encoder in the present application), and a tag parser (i.e., the second tag parser in the present application). The second image encoder is used to extract features of the input image; the second adapter is used to convert the image features output by the second image encoder into a tag (i.e., Token) form; the second tag parser is used to convert the input text into a tag form. For example, the adapter in the embodiment of the present application may be a neural network, and the image encoder may be a convolutional neural network. The tag parser may also be called a Tokenizer (a word segmenter), for example, the tag parser may be a WordPiece Tokenizer (a letter-based word segmenter), or, it may also be a BPETokenizer (Byte-Pair Encoding Tokenizer, a word segmenter based on byte-level encoding).

[0124] In this way, the description generation model can also process the input image and text prompt to ensure that the description generation model can be used to process the input image in combination with the first text prompt to generate a description text of the specified image area in the input image. The description text of the specified image area in an image can be used to describe the image content of the specified image area in the image. For example, the image can be a photo of a forest, which includes trees and a yellow squirrel holding a pine cone, and the specified image area in the image can be the image area where the squirrel is located. Correspondingly, the description text of the specified image area in the image can be "a yellow squirrel holding a pine cone in its hand". The positions of the specified image areas in different images can be the same, or they can be different.

[0125] After determining the structure of the description generation model, the description generation model of the initial structure can be trained by using a plurality of image-text pairs acquired in advance to obtain a first version of the description generation model. An image-text pair may include: a sample image and a description text of a specified image area in the sample image. For example, the image-text pair may come from a public data set, or the sample image may be acquired by an image acquisition device, and the description text of the specified image area in the sample image may be manually determined to obtain the image-text pair. The current description generation model may be the latest version of the description generation model. When the version of the description generation model needs to be updated, the latest description generation model may be trained by step S101 and subsequent steps S102-S104 to obtain a new version of the description generation model.

[0126] The first sample image may be acquired in advance. For example, the first sample image may be from a public data set, or the first sample image may be acquired by an image acquisition device and manually annotating a designated image area in the acquired image. The designated image area in each first sample image may be any image area in the first sample image. For example, the designated image area in an image may be a foreground portion of the image and may include at least one subject object, such as an animal or a vehicle.

[0127] The first text prompt may indicate that the description generation model generates descriptive text for a specified image area in the input image. In one implementation, the first text prompt may include information about the location of the specified image area in the input image (e.g., bounding box coordinates of the specified image area). For example, the first text prompt may be "Please generate descriptive text for the image area in a rectangular box with the upper left vertex coordinates (a, b) and the lower right vertex coordinates (c, d) in the input image." In this implementation, for different images, when the location of the specified image area in the image is different, the information about the location of the specified image area included in the first text prompt is also different. That is, the first text prompt may be different for different images.

[0128] In another implementation, the bounding box of the designated image area in the input image has been annotated, and the first text prompt may indicate that the description generation model generates description text of the image area (i.e., the designated image area) in the annotated bounding box in the input image. For example, the first text prompt may be "Please generate description text of the image area in the annotated bounding box in the input image".

[0129] By processing each first sample image using the first text prompt and the current latest version of the description generation model, a description text for describing a specified image area in the first sample image can be obtained as the current first sample description text. The description text for describing a specified image area in an image can be called the description text corresponding to the image. For each first sample image, one or more first sample description texts corresponding to the first sample image can be obtained using the current latest version of the description generation model.

[0130] In one embodiment, step S101 includes: using the first text prompt and the current description generation model, processing the first sample image according to different sampling parameters to obtain multiple description texts of the specified image area in the first sample image as the current first sample description text.

[0131] In an embodiment of the present application, in the process of processing each first sample image to obtain the first sample description text, the sampling parameters of the description generation model can be adjusted. For example, the sampling parameters may include at least one of the Top-k, Top-p, and Temperature parameters. According to Top-k sampling, in the process of generating each word, the description generation model can select a word from the top k words with the highest probability according to the probability of all possible words as the generated word according to the probability of all possible words. According to Top-p sampling, in the process of generating each word, the description generation model can select words in sequence according to the probability of all possible words in order from high to low until the sum of the probabilities of the selected words is not less than p, and can randomly select one word from the selected words as the generated word. Accordingly, the first text prompt and the description generation model can be used to process the first sample image according to different sampling parameters to obtain multiple description texts of the specified image area in the first sample image as the first sample description text.

[0132] Based on the above processing, for each first sample image, multiple first sample description texts corresponding to the first sample image can be obtained. In this way, the diversity of the first sample description texts can be improved, and it is further ensured that the description text (i.e., the second sample description text) that matches the specified image area in the corresponding first sample image can be determined from the obtained first sample description texts in the future, so as to obtain sample data for optimizing the description generation capability of the description generation model. It is ensured that the description generation capability of the description generation model can be further optimized (also called supervised fine-tuning) to obtain a new description generation model. It is also possible to improve the accuracy of the description text of the image area generated by the description generation model. Subsequently, the description text of the image area can be accurately generated using the new description generation model.

[0133] For step S102, a description text that matches the specified image area in the corresponding first sample image can be determined from the current first sample descriptions as the current second sample description text. The matching of a specified image area in an image with a description text can indicate that the content described by the description text conforms to the image content of the specified image area in the image. A first sample image may not have a corresponding second sample description text, or may have one or more corresponding second sample description texts. In this way, the description text (i.e., the second sample description text) that matches the specified image area in the corresponding first sample image can be screened out from the current first sample descriptions, and this screening process can be called rejection sampling. Subsequently, the current description generation model can be trained based on the screened second sample description text and the corresponding first sample image to optimize the description generation capability of the description generation model.

[0134] For example, for each first sample description, it can be determined manually based on preset preferences whether the first sample description text matches the specified image area in the corresponding first sample image. For example, the preset preferences may include dimensions such as description text accuracy, information richness, description style, etc., which can be determined as needed without specific limitation. Alternatively, for each first sample description, a pre-trained reward model that has the ability to determine whether the specified image area in the input image and the description text match can be used to determine whether the first sample description text matches the specified image area in the corresponding first sample image. The specific method of determining the second sample description text based on the reward model can refer to the relevant description of steps S1021-step S1022 in the subsequent embodiments.

[0135] For step S103 and step S104, the first sample image and the first text prompt corresponding to each second sample description text can be input into the current description generation model. Accordingly, the current description generation model can generate the description text of the specified image area in the input first sample image according to the instruction of the first text prompt, that is, obtain the first predicted description text. Furthermore, the difference between the current first predicted description text and the current second sample description text can be calculated. For example, the similarity between the current first predicted description text and the current second sample description text can be calculated to indicate the difference between the two. According to the difference between the current first predicted description text and the current second sample description text, the parameters of the current description generation model can be adjusted to obtain a new description generation model. For example, the parameters of the current description generation model can be adjusted until the first convergence condition is reached to obtain a new description generation model. For example, the first convergence condition can be that the number of times the parameters of the description generation model are adjusted reaches the first specified number, or it can also be that the difference between the current first predicted description text and the current second sample description text is less than the first preset difference.

[0136] Through the above steps S101-S104, the description generation capability of the latest version of the description generation model can be used to generate training samples, and the description generation capability of the latest version of the description generation model can be further optimized using the obtained training samples to obtain a new description generation model. This process can be called a round of iteration. In the present application, the number of rounds for iterating the description generation model can be set as needed, without specific limitation. In other words, in the present application, the description generation model can be iterated for one or more rounds as needed.

[0137] In one embodiment, after step S104, the description generation model training method further includes: returning to execute step S101 until a second preset condition is met to obtain a new description generation model.

[0138] In an embodiment of the present application, after completing a round of iterations and updating the description generation model to obtain the latest description generation model, a new round of iterations can be continued. That is, the latest description generation model can be continued to be used to process each current first sample image to generate a new first sample description text. The current first sample image can be the same as the first sample image used in the previous round of iterations, or a new first sample image can be obtained. And the new second sample description text can be screened from the obtained new first sample description text. The first sample image and the first text prompt corresponding to each new second sample description text are input into the new description generation model to obtain a new first predicted description text. Correspondingly, the parameters of the new description generation model can be adjusted based on the difference between the new first predicted description text and the new second sample description text until the specified convergence condition is reached to obtain the latest description generation model. Furthermore, the latest description generation model can be continued to be used to process each current first sample image to generate a new first sample description text. Repeat the above process until the second preset condition is reached to obtain a new description generation model.

[0139] The second preset condition may be that the number of times the parameters describing the generated model are adjusted reaches a second preset number. The second preset number can be set as needed and is not specifically limited.

[0140] Alternatively, after completing a round of iterations, the current latest description generation model can be used to process a second preset number of first sample images to obtain description texts of the specified image area in each first sample image. The second preset number can be set as needed and is not specifically limited. And it can be determined whether the proportion of description texts that match the specified image area in the corresponding first sample image in each description text obtained is greater than the second ratio. If the proportion of description texts that match the specified image area in the corresponding first sample image in each description text obtained is greater than the second ratio, it is determined that the second preset condition is met. The second ratio can be set as needed and is not specifically limited. For each description text obtained, it can be manually determined according to preset preferences whether the description text matches the specified image area in the corresponding image. Alternatively, a pre-trained model that has the ability to determine whether the specified image area in the input image and the description text match can be used to determine whether the description text matches the specified image area in the corresponding image.

[0141] Based on the above processing, the description generation model can be iterated for multiple rounds, which can also be called self-loop iterative optimization of the description generation model. In this way, the ability of the description generation model to generate description text that matches the specified image area in the input image can be iteratively optimized, and the probability of the description generation model generating description text that matches the specified image area in the input image can be further improved. The accuracy of the description text of the image area generated by the description generation model can also be improved. Subsequently, the description text of the image area can be accurately generated using the trained description generation model.

[0142] In one embodiment, step S101 includes:

[0143] Step 1: for each current first sample image, when the proportion of the designated image area in the first sample image is less than a preset threshold, the designated image area in the first sample image is expanded to obtain an image to be processed.

[0144] Step 2: Input the obtained image to be processed and the first text prompt into the current description generation model to obtain a description text for describing the specified image area in the first sample image as the current first sample description text.

[0145] Step three: When the proportion of the specified image area in the first sample image is not less than a preset threshold, the first sample image and the first text prompt are input into the current description generation model to obtain a description text for describing the specified image area in the first sample image as the current first sample description text.

[0146] In an embodiment of the present application, for each current first sample image, when the proportion of the designated image area in the first sample image is less than a preset threshold, it means that the designated image area in the first sample image is small, and the designated image area in the first sample image can be expanded to obtain an image to be processed. For example, the preset threshold can be 30% or 25%. The designated image area can be expanded based on the center of the designated image area in the first sample image, and the expanded designated image area can be cropped out from the first sample image to obtain an image to be processed. For example, the bounding box of the designated image area can be expanded based on the center of the designated image area in the first sample image, such as the length and width of the bounding box of the designated image area can be expanded by 10% or 12% of the original length and width, respectively. Furthermore, the obtained image to be processed and the first text prompt can be input into the current description generation model to obtain the first sample description text.

[0147] When the designated image area in a first sample image is small, the image content of the designated image area in the first sample image is also small, resulting in the low sensitivity of the description generation model to the features of the designated image area, which will also affect the accuracy of the description generation model in identifying the features of the designated image area, resulting in low accuracy of the first sample description text generated by the description generation model. In the above manner, when the designated image area in a first sample image is small, the designated image area in the first sample image can be expanded, and the description text can be generated based on the expanded image. In this way, the problem of low accuracy of the first sample description text generated by the description generation model due to the small designated image area in the first sample image can be avoided. That is, the perception ability of the description generation model to the smaller designated image area (also referred to as a small target) in the input image can be improved. It is also possible to improve the accuracy of the first sample description text of the smaller designated image area generated by the description generation model, and further improve the accuracy of the description text used to describe the small target in the training data used in each round of iteration. Accordingly, the reasoning performance of the description generation model for the small target can be improved.

[0148] For each current first sample image, when the proportion of the designated image area in the first sample image is not less than a preset threshold, it means that the designated image area in the first sample image is large (also called a large target), the description generation model is more sensitive to the features of the designated image area, and the accuracy of feature recognition of the designated image area is also high. In this case, the first sample image and the first text prompt can be directly input into the current description generation model to obtain a description text for describing the designated image area in the first sample image as the current first sample description text. In this way, when the designated image area in a first sample image is large, the first sample image can be directly used as the input of the description generation model to generate the first sample description text. The context information of the designated image area can be kept intact, improving the ability of the description generation model to generate description text containing context information.

[0149] Subsequently, in the process of training the description generation model using the screened second sample description text and the corresponding first sample image, the complete first sample image is used as the input of the description generation model for both large and small targets to increase the difficulty of the training samples, thereby improving the robustness of the trained description generation model.

[0150] Based on the above processing, the description generation model can be improved in its ability to perceive small targets while being able to obtain more image content other than the specified image area in the image, so as to fully retain the contextual information of the specified image area and encourage the description generation model to generate a description of the relationship between the subject in the specified image area in the input image and the full image information.

[0151] In one embodiment, see Figure 2 , Figure 2 A second flow chart of a method for generating model training is provided in an embodiment of the present application. Step S102 includes:

[0152] Step S1021: input each current first sample image, the first sample description text corresponding to the first sample image, and the second text prompt into a pre-trained reward model to obtain a matching result between a specified image area in the first sample image and the corresponding first sample description text.

[0153] Among them, the second text prompt is used to indicate whether the specified image area in the input image matches the input description text; the reward model is pre-trained based on positive sample pairs and negative sample pairs; a positive sample pair includes: a second sample image, a specified image area in the second sample image, and a description text that matches the specified image area in the second sample image; a negative sample pair includes: a third sample image, a specified image area in the third sample image, and a description text that does not match the specified image area in the third sample image.

[0154] Step S1022: when the obtained matching result indicates that the designated image area in the first sample image matches the corresponding first sample description text, the first sample description text corresponding to the first sample image is determined as the corresponding second sample description text.

[0155] In an embodiment of the present application, positive sample pairs and negative sample pairs can be obtained in advance, and based on the obtained positive sample pairs and negative sample pairs, the reward model of the initial structure is trained so that the reward model learns the ability to determine whether the input image matches the input description text. The positive sample pairs and negative sample pairs can come from a public data set, or they can be obtained by collecting images through an image acquisition device and annotating them manually. The second text prompt is used to indicate whether the specified image area in the input image matches the input description text. In one implementation, the second text prompt may include information about the position of the specified image area in the input image (e.g., the bounding box coordinates of the specified image area). For example, the second text prompt may be "Please determine whether the image area in the rectangular box with the upper left vertex coordinates (a, b) and the lower right vertex coordinates (c, d) in the input image matches the input description text". In another implementation, the bounding box of the specified image area in the input image has been annotated, for example, the second text prompt may be "Please determine whether the image area in the annotated bounding box in the input image matches the input description text".

[0156] Using the trained reward model combined with the second text prompt, each first sample image and the first sample description text corresponding to the first sample image are processed to obtain a matching result between the specified image area in the first sample image and the corresponding first sample description text. The obtained matching result can indicate whether the specified image area in the first sample image matches the corresponding first sample description text.

[0157] The reward model may include an existing pre-trained large language model, and the type of large language model included in the reward model may be the same as the description generation model. For example, the reward model may include: a multimodal large language model (i.e., the first multimodal large language model in this application), an adapter (i.e., the first adapter in this application), an image encoder (i.e., the first image encoder in this application), and a tag parser (i.e., the first tag parser in this application). The first image encoder is used to extract features of the input image; the first adapter is used to convert the image features output by the first image encoder into a tag form; the first tag parser is used to convert the input text into a tag form. In this way, the reward model can also process the input image and text prompts to ensure that the reward model can be used to process the input image and description text in combination with the second text prompt to determine whether the specified image area in the input image matches the input description text.

[0158] For each first sample image, when the obtained matching result indicates that the designated image area in the first sample image matches the corresponding first sample description text, the first sample description text corresponding to the first sample image can be determined as the corresponding second sample description text.

[0159] Based on the above processing, the reward model can be used to filter the second sample description text from the first sample description text, that is, the pre-trained reward model can be used to implement rejection sampling to improve the accuracy of the training data and obtain high-quality training data. Subsequently, the current description generation model can be trained based on the filtered second sample description text and the corresponding first sample image to optimize the description generation capability of the description generation model.

[0160] In one embodiment, see Figure 3 , Figure 3 A flowchart of training a description generation model based on a reward model is provided for an embodiment of the present application. The description generation model 31 (also referred to as a region description model) may include: a large language model (i.e., the second large language model in the present application), an adapter (i.e., the second adapter in the present application), an image encoder (i.e., the second image encoder in the present application), and a tag parser (i.e., the second tag parser in the present application). Figure 3The multiple small trapezoids in the figure represent the tokens obtained by the adapter and the token parser.

[0161] For each image (i.e., the first sample image in the above embodiment), it can be determined whether the image is a small target, that is, whether the proportion of the specified image area in the first sample image is less than a preset threshold. If so, a center cropped image can be obtained for the specified image area in the first sample image, and the obtained center cropped image and the bounding box coordinates of the specified image area in the image can be input into the previous version of the region description model (i.e., the current description generation model in the above embodiment) to generate the first sample description text, i.e., steps one and two in the above embodiment. If not, the image and the bounding box coordinates of the specified image area in the image can be input into the previous version of the region description model to generate the first sample description text, i.e., step three in the above embodiment.

[0162] Each first sample image, the first sample description text corresponding to the first sample image, and the bounding box coordinates of the specified image area in the first sample image can be input into the pre-trained reward model, that is, step S1021 in the above embodiment. It is also possible to implement rejection sampling through the reward model to determine the positive sample image that has not been rejected, that is, the first sample image corresponding to the screened second sample description text in the above embodiment. The bounding box of the specified image area in the positive sample image can be called a positive sample bounding box, and the description text corresponding to the positive sample image (that is, the second sample description text in the above embodiment) can be called a positive sample description.

[0163] For each current positive sample image, the positive sample image can be input into the image encoder in the current description generation model 31, and the corresponding positive sample description and the text prompt containing the bounding box information (i.e., the first text prompt in the above embodiment) can be input into the tag parser in the current description generation model 31, so as to obtain the region description (i.e., the first predicted description text in the above embodiment) through inference by the current description generation model 31. For details, please refer to the relevant description of step S103 in the above embodiment. Subsequently, based on the difference between the current first predicted description text and the current second sample description text, the parameters of the current description generation model 31 can be adjusted to obtain a new description generation model.

[0164] In one embodiment, the description text in each sample pair is obtained by processing the sample image in the sample pair using the first text prompt and the current description generation model; the reward model is trained by the following steps:

[0165] Step 1: For each sample pair, the sample image and the specified image area in the sample pair are input into the first image encoder in the reward model of the initial structure to obtain the features of the sample image and the specified image area in the sample pair, and the obtained features are spliced.

[0166] Step 2: Input the obtained splicing result into the first adapter in the reward model of the initial structure to obtain the marked form of the splicing result.

[0167] Step 3: Input the second text prompt and the description text in the sample pair into the first token parser in the reward model of the initial structure to obtain the token form of the second text prompt and the token form of the description text in the sample pair.

[0168] Step 4: Input the marked form of the splicing result, the marked form of the second text prompt, and the marked form of the description text in the sample pair into the first language model in the reward model of the initial structure to obtain the predicted matching result between the specified image area in the sample pair and the description text.

[0169] Step 5: Based on the difference between the predicted matching result and the actual matching result of the sample pair, the parameters of the reward model of the initial structure are adjusted until the preset convergence condition is reached to obtain the trained reward model.

[0170] In an embodiment of the present application, a sample image to be used can be acquired in advance, and the sample image to be used can be the same as the first sample image in the above embodiment, or it can be different. For each sample image to be used, the latest description generation model can be used, combined with the first text prompt, to process the sample image to be used, and obtain the description text of the specified image area in the sample image to be used. For details, please refer to the relevant description of the first sample description text obtained in the above embodiment. Furthermore, for each description text obtained, it can be manually determined according to preset preferences whether the description text matches the specified image area in the corresponding sample image to be used. Alternatively, for each description text obtained, a pre-trained model with the ability to determine whether the specified image area in the input image and the description text match can be used to determine whether the description text matches the specified image area in the corresponding sample image to be used.

[0171] When it is determined that a description text matches a designated image region in a corresponding sample image to be used, the description text, the corresponding sample image to be used, and the designated image region in the sample image to be used can be taken as a positive sample pair. In this case, the corresponding sample image to be used is the second sample image. Conversely, the description text, the corresponding sample image to be used, and the designated image region in the sample image to be used can be taken as a negative sample pair. In this case, the corresponding sample image to be used is the third sample image.

[0172] For each sample pair obtained (including positive sample pairs and negative sample pairs), the sample image in the sample pair and the specified image area in the sample image can be input into the first image encoder in the reward model of the initial structure to obtain the features of the sample image in the sample pair and the features of the specified image area in the sample image, and the obtained features can be spliced. The obtained splicing result is input into the first adapter in the reward model of the initial structure to obtain the splicing result in a marked form. The second text prompt and the description text in the sample pair can be input into the first tag parser in the reward model of the initial structure to obtain the second text prompt in a marked form and the description text in the sample pair. The splicing result in a marked form, the second text prompt and the description text in the sample pair are input into the first large language model in the reward model of the initial structure. Accordingly, the first large language model can infer the predicted matching result between the specified image area in the sample image in the sample pair and the description text.

[0173] The second text prompt may instruct the reward model to obtain an output result based on the thought chain, that is, the first language model in the reward model may output a conclusion on whether the specified image area in the sample image of the sample pair matches the description text, or may output an analysis process for determining whether the two match. That is, the output result of the reward model may include: a conclusion on whether the specified image area in the input image matches the input description text, and an analysis process for determining whether the two match.

[0174] like Figure 4a As shown, Figure 4a A schematic diagram of an image of an input reward model provided in an embodiment of the present application. Figure 4a In the figure, there is a little boy a and a little girl b, sitting on a green swing. The little girl is smiling and wearing a pink hat. The dotted box 41 indicates the designated image area. The descriptive text corresponding to the input reward model can be "a person sitting on a swing and smiling", and the analysis process of the reward model to determine whether the two match can be "the object in the area is a girl wearing a pink hat, she is sitting on the swing and seems to be smiling", and the conclusion of the reward model to determine whether the two match can be "yes". Figure 4b As shown, Figure 4bA schematic diagram of an image of another input reward model provided in an embodiment of the present application. Figure 4b There is a little boy a and a little girl b sitting on a green swing. The dotted box 42 represents the designated image area. The descriptive text corresponding to the input reward model can be "children on the swing", and the analysis process of the reward model to determine whether the two match can be "the object in the area is a green swing. There are two children sitting on the swing. The description 'children on the swing' refers to the children sitting on the swing, not the swing itself", and the conclusion of the reward model to determine whether the two match can be "no".

[0175] Furthermore, based on the difference between the obtained predicted matching result and the actual matching result of the sample pair, the parameters of the reward model of the initial structure can be adjusted until the preset convergence condition is reached to obtain a trained reward model. If the sample pair is a positive sample pair, the actual matching result of the sample pair can represent: the specified image area in the sample image in the sample pair matches the description text. If the sample pair is a negative sample pair, the actual matching result of the sample pair can represent: the specified image area in the sample image in the sample pair does not match the description text. For example, the preset convergence condition can be that the number of times the parameters of the reward model are adjusted reaches a specified number, or it can also be that the difference between the obtained predicted matching result and the actual matching result of the sample pair is less than a second preset difference.

[0176] Based on the above processing, the latest description generation model can be used to obtain sample pairs for training the reward model. By training the reward model based on the obtained sample pairs, the reward model can learn the ability to determine whether the description text generated by the description generation model matches the specified image area in the input image. Furthermore, using the trained reward model, the second sample description text can be accurately screened out from the first sample description text to ensure that the current description generation model can be trained based on the screened second sample description text and the corresponding first sample image to optimize the description generation ability of the description generation model. Subsequently, using the trained description generation model, the description text of the image area can be accurately generated.

[0177] In one embodiment, see Figure 5 , Figure 5 A flowchart of a training reward model provided in an embodiment of the present application. The reward model 51 may include: a large language model (i.e., the first large language model in the present application), an adapter (i.e., the first adapter in the present application), an image encoder (i.e., the first image encoder in the present application), and a tag parser (i.e., the first tag parser in the present application). Figure 5 The multiple small trapezoids in the figure represent the tokens obtained by the adapter and the token parser.

[0178] For each sample image to be used (i.e. Figure 5 ), the previous version of the regional description model (i.e., the current description generation model) can be used, combined with the first text prompt, to process the sample image to be used, and obtain the description text of the specified image area in the sample image to be used. For details, reference can be made to the relevant description of the first sample description text obtained in the above embodiment. Furthermore, for each description text obtained, it can be determined manually whether the description text conforms to human preferences to determine whether the description text matches the specified image area in the corresponding sample image to be used. If so, the description text (also referred to as a positive sample description), the corresponding sample image to be used, and the specified image area in the sample image to be used (i.e., the cropped image of the bounding box area) can be used as a positive sample pair. If not, the description text (also referred to as a negative sample description), the corresponding sample image to be used, and the specified image area in the sample image to be used can be used as a negative sample pair.

[0179] For each sample pair obtained (including positive sample pairs and negative sample pairs), the sample image in the sample pair and the designated image area in the sample image can be input into the image encoder in the reward model of the initial structure, and the text prompt containing the bounding box information (i.e., the second text prompt in the above embodiment) and the description text in the sample pair can be input into the tag parser in the reward model of the initial structure, so as to obtain the thought chain analysis process and conclusion through the reward model 51 reasoning, that is, the thought chain form of the analysis process to determine whether the two match, and the conclusion whether the designated image area in the input image matches the input description text. Furthermore, based on the difference between the obtained predicted matching result and the actual matching result of the sample pair, the parameters of the reward model of the initial structure can be adjusted until the preset convergence condition is reached, and the trained reward model is obtained.

[0180] In one embodiment, see Figure 6 , Figure 6 The first flow chart of directly optimizing the preference of the description generation model provided in the embodiment of the present application. After step S104, the description generation model training method further includes:

[0181] Step S601: Acquire a fourth sample image.

[0182] Among the description texts generated by the current description generation model for each fourth sample image, there are description texts that do not match the designated image area in the corresponding fourth sample image.

[0183] Step S602: using the first text prompt and the current description generation model, processing each fourth sample image acquired according to different sampling parameters to obtain a plurality of description texts of a designated image area in the fourth sample image.

[0184] Step S603: Determine, from the obtained description texts, at least one positive sample description text that matches the specified image region in the fourth sample image, and at least one negative sample description text that does not match the specified image region in the fourth sample image.

[0185] Step S604: input the fourth sample image and the first text prompt into the current description generation model to obtain the description text of the specified image area in the fourth sample image as the current second predicted description text.

[0186] Step S605: Based on the difference between the current second predicted description text and the determined positive sample description text, and the difference between the current second predicted description text and the determined negative sample description text, the parameters of the current description generation model are adjusted to obtain a new description generation model.

[0187] In an embodiment of the present application, in the description text generated by the current description generation model for each fourth sample image, there is description text that does not match the specified image area in the corresponding fourth sample image, which means that the current latest description generation model has unstable reasoning performance for the scene to which the fourth sample image belongs. The scene to which the fourth sample image belongs may represent the scene to which the specified object contained in the specified image area in the fourth sample image belongs. For example, the specified object may be a pedestrian, and the scene to which the fourth sample image belongs is the scene where the pedestrian is located, such as an intersection.

[0188] In one implementation, during the process of the description generation model generating the description text, the description generation model can be manually observed to determine the scene where the reasoning performance of the description generation model is unstable. In another implementation, the description text and the corresponding image generated by the description generation model can be processed by a pre-trained reward model to determine whether the generated description text matches the specified image area in the corresponding image. For each scene, it can be determined whether the reasoning performance of the description generation model for the scene is stable based on the number of occurrences of the mismatch between the generated description text and the specified image area in the corresponding image belonging to the scene. For example, in the description text generated by the description generation model for each fourth sample image, when the proportion of the description text that does not match the specified image area in the corresponding fourth sample image is greater than a preset ratio, it can be determined that the reasoning performance of the description generation model for the scene to which the fourth sample image belongs is unstable.

[0189] After determining a scene in which the reasoning performance of the description generation model is unstable, a sample image (i.e., a fourth sample image) belonging to the scene can be obtained. Using the first text prompt and the current description generation model, each fourth sample image obtained is processed according to different sampling parameters, and multiple description texts of the specified image area in the fourth sample image can be obtained. Furthermore, at least one positive sample description text that matches the specified image area in the fourth sample image and at least one negative sample description text that does not match the specified image area in the fourth sample image can be determined from the obtained description text. In one implementation, the positive sample description text and the negative sample description text can be selected from the obtained description text manually according to preference. In another implementation, the matching result between the obtained description text and the specified image area in the fourth sample image can be determined using a pre-trained reward model. And the positive sample description text and the negative sample description text are selected from the obtained description text according to the obtained matching result. In the case where the matching result obtained by the reward model indicates that the obtained description text matches the specified image area in the fourth sample image, the positive sample description text and the negative sample description text can be selected from the obtained description text manually according to preference. For example, preference may indicate a preference for description texts with richer description features. In this case, description texts with richer description features may be selected from the obtained description texts as positive sample description texts, and description texts with relatively less description features may be selected from the obtained description texts as negative sample description texts.

[0190] The fourth sample image and the first text prompt are input into the current description generation model to obtain the description text of the specified image area in the fourth sample image as the current second predicted description text. Furthermore, based on the difference between the current second predicted description text and the determined positive sample description text, and the difference between the current second predicted description text and the determined negative sample description text, the parameters of the current description generation model can be adjusted to obtain a new description generation model. For example, the parameters of the current description generation model can be adjusted until the second convergence condition is reached to obtain a new description generation model. For example, the second convergence condition can be that the number of times the parameters of the description generation model are adjusted reaches a second specified number, or it can also be that the difference between the current second predicted description text and the determined positive sample description text is less than the first difference, and the difference between the current second predicted description text and the determined negative sample description text is greater than the second difference.

[0191] Based on the above processing, direct preference optimization can be performed for the scene where the reasoning performance of the latest description generation model is unstable, so as to optimize the description generation model's ability to generate description text for the specified image area in the image belonging to this scene. In this way, the description generation model's preference for positive sample description text can be improved, and the preference for negative sample description text can be reduced. It is also possible to further improve the probability that the description generation model generates description text that matches the specified image area in the input image. It is also possible to improve the accuracy of the description text of the image area generated by the description generation model. Subsequently, the description text of the image area can be accurately generated using the trained description generation model.

[0192] In one embodiment, see Figure 7 , Figure 7 A second flow chart of directly optimizing the preference of the description generation model provided in the embodiment of the present application. The description generation model 71 may include: a large language model (i.e., the second large language model in the present application), an adapter (i.e., the second adapter in the present application), an image encoder (i.e., the second image encoder in the present application), and a tag parser (i.e., the second tag parser in the present application). Figure 5 The multiple small trapezoids in the figure represent the tokens obtained by the adapter and the token parser.

[0193] For each fourth sample image (i.e. Figure 7 ), the first text prompt containing the coordinates of the bounding box and the previous version of the regional description model (i.e., the current description generation model in the above embodiment) can be used to process the fourth sample image according to different sampling parameters to obtain multiple description texts of the specified image area in the fourth sample image (i.e., samples obtained by multiple inferences). Furthermore, positive sample description texts (also referred to as positive sample descriptions) and negative sample description texts (also referred to as negative sample descriptions) can be manually selected from the obtained description texts according to preferences, that is, to determine whether the obtained description text meets human preferences. If so, it is determined to be a positive sample description, and if not, it is determined to be a negative sample description.

[0194] By inputting the fourth sample image and the text prompt containing the bounding box information (i.e., the first text prompt in the above embodiment) into the current description generation model 71, the description text of the specified image area in the fourth sample image can be obtained as the current second predicted description text. Furthermore, based on the difference between the current second predicted description text and the determined positive sample description text, and the difference between the current second predicted description text and the determined negative sample description text, the parameters of the current description generation model can be adjusted to obtain a new description generation model. The description generation model can increase its preference for the positive sample description text and reduce its preference for the negative sample description text, so that the preference for the positive sample description text is much greater than the preference for the negative sample description text.

[0195] In one embodiment, see Figure 8 , Figure 8 A schematic diagram of a flow chart for iterating a description generation model provided in an embodiment of the present application. For each first sample image, the latest description generation model (also called image region description model) can be used to perform K inferences under different sampling parameters to obtain K groups of results (i.e. Figure 8 The same data list in K groups of results is obtained). That is, K first sample description texts of the specified image area in the first sample image are obtained. The obtained first sample description text can be firstly subjected to rule-based data cleaning, for example, removing description texts containing erroneous symbols and / or being too long. Then, the pre-trained reward model is used for rejection sampling to obtain supervised fine-tuning training data (i.e., the second sample description text). And the latest description generation model can be used to inject human preferences, that is, the positive sample pairs and negative sample pairs used to train the reward model are manually determined according to the preferences. The reward model can be supervised fine-tuned according to the obtained sample pairs.

[0196] The current description generation model can be fine-tuned in a supervised manner according to the obtained supervised fine-tuning training data to obtain a new description generation model, i.e., steps S103 and S104 in the above embodiment. The latest description generation model obtained can be directly optimized, i.e., steps S601 to S605 in the above embodiment.

[0197] In one embodiment, the description generation model can also be trained to generate description texts with different description styles. Fig. 9 , Fig. 9 A flowchart of the ability of the description generation model to generate a description text of a specified description style provided in the embodiment of the present application. The first text prompt is also used to indicate: the current description generation model generates a description text that conforms to the first specified description style; a description style is used to indicate the language of the description text and / or the number of sentences contained therein.

[0198] After step S104, the description generation model training method further includes:

[0199] Step S901: using the first text prompt and the current description generation model, processing each current first sample image to obtain a description text of a first specified description style for a specified image area in the first sample image as the current third sample description text.

[0200] Step S902: adjusting the description style of the current third sample description text from the first specified description style to the second specified description style to obtain the current fourth sample description text.

[0201] Step S903: Determine the description text that matches the designated image area in the corresponding first sample image from the current fourth sample description text to obtain the current fifth sample description text.

[0202] Step S904: input the first sample image and the third text prompt corresponding to each fifth sample description text into the current description generation model to obtain the description text of the second specified description style of the specified image area in the first sample image as the current third predicted description text.

[0203] Step S905: Based on the difference between the current third predicted description text and the current fifth sample description text, the parameters of the current description generation model are adjusted to obtain a new description generation model.

[0204] In an embodiment of the present application, the ability of the description generation model to generate description texts of different description styles can be trained. A description style can be used to indicate the language of the description text and / or the number of sentences contained. According to the number of sentences contained, it can be determined whether the description text is a long description or a short description, for example, the number of sentences in the long description is not less than the preset sentence number threshold. The number of sentences in the short description is not greater than the preset sentence number threshold. For example, the number of sentences can be the number of periods in the description text. The preset sentence number threshold can be set as needed without specific limitation. For example, the preset sentence number threshold can be 1, in which case the number of sentences in the long description can be greater than or equal to 1, and the number of sentences in the short description is 1. In this way, it can be avoided that the number of sentences in the long description is set to be greater than 1 for image areas with simpler image content, resulting in errors in the generated description text, and the accuracy of the description text generated by the description generation model can be further guaranteed.

[0205] For example, description styles can include Chinese long description, Chinese short description, English long description, and English short description. Fig.10 , Fig.10A schematic diagram of an image including multiple image regions for which description texts of different description styles need to be generated is provided in an embodiment of the present application. The description texts of different styles corresponding to each bounding box can be displayed near the bounding box. Specifically, there is a gray stone ( Fig.10 The corresponding Chinese short description can be "a small gray stone", the English short description can be "A gray rock on the ground", the English long description can be "A gray rock is visible, which is part of a larger pile of rocks. The rock's surface is rough and uneven", and the Chinese long description can be "a gray stone with a rough surface and irregular shape".

[0206] The image region in the second bounding box 12 contains a person riding a motorcycle wearing a white, blue, and orange jacket ( Fig.10 The corresponding Chinese short description could be "A person wearing a white, blue, and orange jacketriding a motorcycle", and the English short description could be "A person wearing a white, blue, and orange jacketriding a motorcycle", and the English long description could be "A person riding a motorcycle is visible. The rider is wearing a white and blue jacket, black pants, and a redand white helmet. The motorcycle is red and black. The rider is traveling ona dirt road surrounded by greenery and hills. The rider is moving away from the camera, and there are no other vehicles or people in sight", and the Chinese long description could be "A person riding a motorcycle, wearing a blue and white jacket, black pants, and a red and white helmet. The rider's posture shows that he is in control of the motorcycle, and the rear wheel of the motorcycle kicks up a cloud of dust".

[0207] There is an exhaust pipe of a motorcycle in the image area in the third bounding box 13, and the corresponding Chinese short description may be “Motorcycle exhaust pipe”, the English short description may be “A motorcycle exhaust pipe on a motorcycle”, the English long description may be “A motorcycle exhaust pipe is visible, which is a component of the motorcycle's engine system. The exhaust pipe is cylindrical and appears to be made of metal, with a reflective surface. It is attached to the engine and extends towards the rear of the motorcycle”, and the Chinese long description may be “A motorcycle exhaust pipe is part of the motorcycle's exhaust system, which allows exhaust gases to be discharged from the engine. The exhaust pipe is cylindrical, the surface is made of metal, and there is a muffler at the end to reduce noise”.

[0208] In the process of iterating the description generation model in one round, the first text prompt can be used to instruct the description generation model to generate a description text that conforms to the first specified description style. The first specified description style can be set as needed and is not specifically limited. By using the first text prompt and the current description generation model, each current first sample image is processed to obtain a description text of the specified image area in the first sample image as the current third sample description text. The new description style that the description generation model currently needs to learn can be called the second specified description style.

[0209] The description style of the current third sample description text can be adjusted from the first specified description style to the second specified description style to obtain the current fourth sample description text. For example, if the first specified description style is a short description in Chinese and the second specified description style is a short description in English, the current third sample description text can be translated to obtain the current fourth sample description text. Alternatively, if the first specified description style is a short description in Chinese and the second specified description style is a long description in Chinese, the current third sample description text can be continued to obtain the current fourth sample description text. Both translation and continuation can use existing translation tools or continuation tools. For example, it can be implemented using a large language model with translation or continuation functions.

[0210] From the current fourth sample description text, determine the description text that matches the specified image area in the corresponding first sample image to obtain the current fifth sample description text. This process can refer to the above-mentioned embodiment of filtering the relevant description of the second sample description text from the first sample description text. The third text prompt can instruct the description generation model to generate a description text of the second specified description style for the specified image area in the input image. For example, the third text prompt can be "Please generate a description text for the image area in the marked boundary box in the input image, and the description style of the generated description text is an English short description."

[0211] The first sample image and the third text prompt corresponding to each of the current fifth sample description texts are input into the current description generation model, and the description text of the second specified description style of the specified image area in the first sample image can be obtained as the current third predicted description text. Based on the difference between the current third predicted description text and the current fifth sample description text, the parameters of the current description generation model are adjusted to obtain a new description generation model. For details, please refer to the relevant description of step S104 in the above embodiment.

[0212] Based on the above processing, the description generation model can learn the ability to generate description texts of different description styles. Subsequently, the user can select the required description style from the description styles that the description generation model can provide, and indicate the selected description style in the text prompt, so that the description generation model can generate the description text of the description style selected by the user. And in the process of training the description generation model to generate the description text of a new description style, the trained description generation model with the ability to generate description texts of other description styles can be used to generate training samples. In this way, the ability of the trained description generation model to generate description texts of other description styles can be used to obtain high-quality training data of the new description style. That is, the optimization results of the description generation model for other description styles can be reused to obtain high-quality training data. It is also possible to achieve efficient optimization of the ability to generate description texts of different description styles.

[0213] In one embodiment, after step S905, the description generation model training method further includes: inputting each current first sample image and third text prompt into the current description generation model to obtain a description text of a second specified description style for a specified image area in the first sample image as the current fourth sample description text; returning to execute step S903 until the first preset condition is met to obtain a new description generation model.

[0214] In an embodiment of the present application, through the above steps S901-S905, a round of iteration of the ability of the description generation model to generate a description text of the second specified description style can be achieved. Multiple rounds of iterations of the ability of the description generation model to generate a description text of the second specified description style can be achieved. After step S905, each current first sample image and the third text prompt can be input into the current latest description generation model to obtain the description text of the second specified description style of the specified image area in the first sample image as a new fourth sample description text. Furthermore, a new fifth sample description text can be screened from the new fourth sample description text. And the description generation model is trained using the new fifth sample description text, the corresponding first sample image and the third text prompt. It is also possible to achieve a new round of iteration of the ability of the description generation model to generate a description text of the second specified description style.

[0215] The first preset condition may be that the number of times the parameters describing the generated model are adjusted reaches a first preset number of times. The first preset number of times may be set as required and is not specifically limited.

[0216] Alternatively, after completing a round of iterations, the latest description generation model can be used to process a first preset number of first sample images to obtain description texts of the designated image area in each first sample image. The first preset number can be set as needed, without specific limitation, and the first preset number can be the same as or different from the second preset number. And it can be determined whether the proportion of description texts matching the designated image area in the corresponding first sample image in each description text obtained is greater than the first ratio. If the proportion of description texts matching the designated image area in the corresponding first sample image in each description text obtained is greater than the first ratio, it is determined that the first preset condition is met. The first ratio can be set as needed, without specific limitation, and the first ratio can be the same as or different from the second ratio. For each description text obtained, it can be manually determined according to a preset preference whether the description text matches the designated image area in the corresponding image. Alternatively, a pre-trained model having the ability to determine whether the designated image area in the input image matches the description text can be used to determine whether the description text matches the designated image area in the corresponding image.

[0217] Based on the above processing, for each description style, the description generation model can be iterated multiple times. In this way, the accuracy of the description generation model in generating the description text of the specified description style can be further improved. Subsequently, the description text of the image area can be accurately generated using the trained description generation model.

[0218] In one embodiment, see Fig.11 , Fig.11A schematic diagram of the reuse of capabilities of different description styles provided in an embodiment of the present application. For each description style, the description generation model can be iterated one or more times, that is, the task inner loop optimization is performed. In the process of training the description generation model to generate the description text of a new description style, the description generation model that has been trained and has the ability to generate description texts of other description styles can be used to generate training samples. For example, the description generation model that has been trained and has the ability to generate description texts of short description style of language A can be used to process the first sample image to generate description texts of short description style of language A. And the description text of short description style of language B can be translated to obtain training data of short description style of language B. Alternatively, the description text of long description style of language A can be continued to obtain training data of long description style of language A. The description generation model that has been trained and has the ability to generate description texts of long description style of language A can be used to process the first sample image to generate description texts of long description style of language A, and the description text of long description style of language B can be translated to obtain training data of long description style of language B. In this way, the ability of the trained description generation model to generate description texts of other description styles can be used to obtain high-quality training data for new description styles, thereby achieving a mapping between the abilities to generate different description styles.

[0219] Based on the same inventive concept, the present application embodiment also provides a description generation method, see Fig.12 , Fig.12 A flowchart of a description generation method provided in an embodiment of the present application, the method comprising:

[0220] Step S1201: Obtain the image to be used and the text prompt to be used.

[0221] Step S1202: input the image to be used and the text prompt to be used into a pre-trained description generation model to obtain a description text of the image area indicated by the text prompt to be used in the image to be used.

[0222] The description generation model is obtained by training based on any of the description generation model training methods described above.

[0223] Based on the above processing, the description generation model can generate training samples using the description generation capability of the description generation model obtained in the previous training during the training process, and further optimize the description generation capability of the existing description generation model to obtain a new description generation model. That is, the probability of the description generation model generating a description text that matches the specified image area in the input image is further improved. The accuracy of the description text of the image area generated by the description generation model can also be improved. Accordingly, the description text of the image area can be accurately generated using the trained description generation model.

[0224] In one embodiment, the position of the image area indicated by the to-be-used text prompt in the to-be-used image is obtained by the following steps:

[0225] Displaying images to be utilized;

[0226] In response to a user's frame selection operation on the displayed image to be used, a position indicated by the frame selection operation in the image to be used is determined, and a position of an image area indicated by the text prompt to be used in the image to be used is obtained.

[0227] In an embodiment of the present application, an image to be used can be displayed so that a user can browse the image to be used, and can frame an image area in the displayed image to be used for which a description text needs to be generated by a frame selection operation. Accordingly, in response to the user's frame selection operation on the displayed image to be used, the position indicated by the frame selection operation in the image to be used can be determined, and the position of the image area indicated by the text prompt to be used in the image to be used can be obtained. In this way, it can be ensured that the description generation model can determine the position of the image area in the image to be used for which a description text needs to be generated, so as to accurately generate the description text for the image area indicated by the text prompt to be used in the image to be used.

[0228] In one embodiment, the description style of the obtained description text is the description style indicated by the text prompt to be used; a description style is used to indicate the language of the description text and / or the number of sentences included.

[0229] In the embodiment of the present application, the user can select the description style indicated by the to-be-used text prompt as needed. Accordingly, the to-be-used text prompt can instruct the description generation model to generate a description text that conforms to the description style selected by the user. In this way, it can be ensured that the description generation model can generate a description text that conforms to the user's needs.

[0230] In one embodiment, see Fig.13 , Fig.13A flowchart of another description generation method provided for an embodiment of the present application. The trained description generation model 131 may include: a large language model (i.e., the second large language model in the present application), an adapter (i.e., the second adapter in the present application), an image encoder (i.e., the second image encoder in the present application), and a tag parser (i.e., the second tag parser in the present application). Figure 3 The multiple small trapezoids in the figure represent the tokens obtained by the adapter and the token parser.

[0231] The image (i.e., the image to be used in the above embodiment) can be input into the image encoder in the current description generation model 31, and the text prompt containing the bounding box information (i.e., the text prompt to be used in the above embodiment) can be input into the tag parser in the description generation model 131, so as to obtain the description text (also referred to as the area description) of the image area indicated by the text prompt to be used in the image to be used through the description generation model 131 inference. The text prompt to be used can include the bounding box coordinates. The user can perform a box selection operation on the displayed image to be used, and the position indicated by the box selection operation is the bounding box area drawn by the user. According to the border area drawn by the user, the bounding box coordinates can also be parsed. Accordingly, the parsed bounding box coordinates can be added to the text prompt to be used. And the user can select the description style indicated by the text prompt to be used (i.e., specify the description style) as needed. Accordingly, the text prompt to be used can also include a specified description style, which can also instruct the description generation model to generate a description text that conforms to the specified description style.

[0232] It can be seen that the description generation method in the present application can generate description text based on an end-to-end multimodal large model, and the description generation model in the present application supports the description generation function of any area of ​​the image in multiple languages ​​and multiple styles, and the user can select the image area and the description style as needed, with strong interactivity.

[0233] Based on the same inventive concept, the present application embodiment also provides a description generation model training device, see Fig.14 , Fig.14 A schematic diagram of a structure of a description generation model training device provided in an embodiment of the present application. The device comprises:

[0234] The first sample description text generation module 1401 is used to process each current first sample image using the first text prompt and the current description generation model to obtain a description text for describing a specified image area in the first sample image as the current first sample description text; wherein the first text prompt is used to indicate the generation of a description text for a specified image area in an input image;

[0235] The second sample description text determination module 1402 is used to determine, from the current first sample description texts, a description text that matches the designated image area in the corresponding first sample image as the current second sample description text;

[0236] The first predicted description text generation module 1403 is used to input the first sample image and the first text prompt corresponding to each current second sample description text into the current description generation model to obtain the description text of the specified image area in the first sample image as the current first predicted description text;

[0237] The first parameter adjustment module 1404 is used to adjust the parameters of the current description generation model based on the difference between the current first predicted description text and the current second sample description text to obtain a new description generation model.

[0238] Based on the description generation model training device provided in the embodiment of the present application, the current latest description generation model can be used to generate a description text (i.e., a first sample description text) of a specified image area in a sample image. Due to the influence of the accuracy of the description generation model, the first sample description text inferred by the description generation model may not match the specified image area in the corresponding first sample image. Therefore, the description text (i.e., the second sample description text) that matches the specified image area in the corresponding first sample image can be further screened from the obtained first sample description text. Furthermore, each second sample description text can be used as a true value label, and the first sample image corresponding to each second sample description text can be used as a training sample to train the current description generation model to obtain a new description generation model. In this way, the description generation capability of the existing description generation model can be used to generate training samples, and the description generation capability of the existing description generation model can be further optimized to obtain a new description generation model. That is, the probability of the description generation model generating a description text that matches the specified image area in the input image is further improved. It is also possible to improve the accuracy of the description text of the image area generated by the description generation model. Subsequently, using the trained description generation model, the description text of the image area can be accurately generated.

[0239] In one embodiment, the second sample description text determination module is specifically used to input each current first sample image, the first sample description text corresponding to the first sample image, and the second text prompt into a pre-trained reward model to obtain a matching result between a specified image area in the first sample image and the corresponding first sample description text; wherein the second text prompt is used to indicate whether the specified image area in the input image matches the input description text; the reward model is pre-trained based on positive sample pairs and negative sample pairs; a positive sample pair includes: a second sample image, a specified image area in the second sample image, and a description text that matches the specified image area in the second sample image; a negative sample pair includes: a third sample image, a specified image area in the third sample image, and a description text that does not match the specified image area in the third sample image; when the matching result obtained indicates that the specified image area in the first sample image matches the corresponding first sample description text, the first sample description text corresponding to the first sample image is determined as the corresponding second sample description text.

[0240] In one embodiment, the reward model includes: a first large language model, a first adapter, a first image encoder and a first token parser; the first image encoder is used to extract features of an input image; the first adapter is used to convert the image features output by the first image encoder into a token form; the first token parser is used to convert the input text into a token form.

[0241] In one embodiment, the description text in each sample pair is obtained by processing the sample image in the sample pair using the first text prompt and the current description generation model; the device further comprises:

[0242] The reward model training module is used to input the sample image and the designated image area in the sample pair into the first image encoder in the reward model of the initial structure for each sample pair, obtain the features of the sample image and the designated image area in the sample pair, and splice the obtained features; input the obtained splicing result into the first adapter in the reward model of the initial structure to obtain the marked form of the splicing result; input the second text prompt and the description text in the sample pair into the first mark parser in the reward model of the initial structure to obtain the marked form of the second text prompt and the marked form of the description text in the sample pair; input the marked form of the splicing result, the marked form of the second text prompt and the marked form of the description text in the sample pair into the first large language model in the reward model of the initial structure to obtain the predicted matching result between the designated image area in the sample pair and the description text; based on the difference between the obtained predicted matching result and the actual matching result of the sample pair, adjust the parameters of the reward model of the initial structure until the preset convergence condition is reached to obtain a trained reward model.

[0243] In one embodiment, the first sample description text generation module 1401 is specifically used to, for each current first sample image, expand the specified image area in the first sample image when the proportion of the specified image area in the first sample image is less than a preset threshold, so as to obtain an image to be processed; input the obtained image to be processed and the first text prompt into the current description generation model to obtain a description text for describing the specified image area in the first sample image as the current first sample description text; and input the first sample image and the first text prompt into the current description generation model when the proportion of the specified image area in the first sample image is not less than the preset threshold, so as to obtain a description text for describing the specified image area in the first sample image as the current first sample description text.

[0244] In one embodiment, the first sample description text generation module 1401 is specifically used to utilize the first text prompt and the current description generation model to process the first sample image according to different sampling parameters to obtain multiple description texts of the specified image area in the first sample image as the current first sample description text.

[0245] In one embodiment, the apparatus further comprises:

[0246] A fourth sample image acquisition module, configured to acquire a fourth sample image after adjusting the parameters of the current description generation model based on the difference between the current first predicted description text and the current second sample description text to obtain a new description generation model; wherein, among the description texts generated by the current description generation model for each fourth sample image, there are description texts that do not match the specified image area in the corresponding fourth sample image;

[0247] A description text inference module, used to process each fourth sample image acquired according to different sampling parameters by using the first text prompt and the current description generation model, to obtain a plurality of description texts of a specified image area in the fourth sample image;

[0248] a positive and negative sample description text determination module, used to determine, from the obtained description texts, at least one positive sample description text that matches the specified image area in the fourth sample image, and at least one negative sample description text that does not match the specified image area in the fourth sample image;

[0249] A second predicted description text generation module, used for inputting the fourth sample image and the first text prompt into a current description generation model to obtain a description text of a specified image area in the fourth sample image as a current second predicted description text;

[0250] The second parameter adjustment module is used to adjust the parameters of the current description generation model based on the difference between the current second predicted description text and the determined positive sample description text, and the difference between the current second predicted description text and the determined negative sample description text to obtain a new description generation model.

[0251] In one embodiment, the first text prompt is further used to indicate that: the current description generation model generates a description text that conforms to a first specified description style; a description style is used to indicate the language of the description text and / or the number of sentences included;

[0252] The device also includes:

[0253] A third sample description text generation module is used for adjusting the parameters of the current description generation model based on the difference between the current first predicted description text and the current second sample description text to obtain a new description generation model, and then using the first text prompt and the current description generation model to process each current first sample image to obtain a description text of the first specified description style for a specified image area in the first sample image as the current third sample description text;

[0254] a fourth sample description text generating module, configured to adjust the description style of the current third sample description text from the first specified description style to the second specified description style, to obtain a current fourth sample description text;

[0255] A fifth sample description text determination module determines, from the current fourth sample description text, a description text that matches the designated image area in the corresponding first sample image, to obtain the current fifth sample description text;

[0256] A third predicted description text generation module is used to input the first sample image and the third text prompt corresponding to each fifth sample description text into the current description generation model, and obtain the description text of the second specified description style of the specified image area in the first sample image as the current third predicted description text;

[0257] The third parameter adjustment model is used to adjust the parameters of the current description generation model based on the difference between the current third predicted description text and the current fifth sample description text to obtain a new description generation model.

[0258] In one embodiment, the apparatus further comprises:

[0259] The fourth sample description text updating module is used to adjust the parameters of the current description generation model based on the difference between the current third predicted description text and the current fifth sample description text, and after obtaining the new description generation model, each current first sample image and the third text prompt are input into the current description generation model to obtain the description text of the second specified description style of the specified image area in the first sample image as the current fourth sample description text; trigger the fifth sample description text determination module until the first preset condition is met to obtain the new description generation model.

[0260] In one embodiment, the apparatus further comprises:

[0261] The trigger module adjusts the parameters of the current description generation model based on the difference between the current first predicted description text and the current second sample description text to obtain a new description generation model, and then triggers the first sample description text generation module 1401 until the second preset condition is met to obtain a new description generation model.

[0262] In one embodiment, the description generation model includes: a second large language model, a second adapter, a second image encoder and a second token parser; the second image encoder is used to extract features of an input image; the second adapter is used to convert the image features output by the second image encoder into a token form; the second token parser is used to convert the input text into a token form.

[0263] Based on the same inventive concept, the present application embodiment also provides a description generating device, see Fig.15 , Fig.15 A schematic diagram of a structure of a description generation device provided in an embodiment of the present application. The device comprises:

[0264] The data acquisition module 1501 is used to acquire the image to be used and the text prompt to be used;

[0265] The description text generation module 1502 is used to input the image to be used and the text prompt to be used into a pre-trained description generation model to obtain a description text of the image area indicated by the text prompt to be used in the image to be used; wherein the description generation model is trained based on the description generation model training method described in any of the above embodiments.

[0266] Based on the description generation device provided by the embodiment of the present application, the description generation model can generate training samples using the description generation capability of the description generation model obtained in the previous training during the training process, and further optimize the description generation capability of the existing description generation model to obtain a new description generation model. That is, the probability that the description generation model generates a description text that matches the specified image area in the input image is further improved. The accuracy of the description text of the image area generated by the description generation model can also be improved. Accordingly, using the trained description generation model, the description text of the image area can be accurately generated.

[0267] In one embodiment, the position of the image area indicated by the to-be-used text prompt in the to-be-used image is obtained by the following steps:

[0268] Displaying the image to be used;

[0269] In response to a user's frame selection operation on the displayed image to be used, a position indicated by the frame selection operation in the image to be used is determined, and a position of an image area indicated by the text prompt to be used in the image to be used is obtained.

[0270] In one embodiment, the description style of the obtained description text is the description style indicated by the text prompt to be used; a description style is used to indicate the language of the description text and / or the number of sentences included.

[0271] The present application also provides an electronic device, such as Fig.16 As shown, including:

[0272] Memory 1601, used for storing computer programs;

[0273] The processor 1602 is used to implement any of the above-mentioned description generation model training methods or any of the above-mentioned description generation methods when executing the program stored in the memory 1601.

[0274] In addition, the electronic device may further include a communication bus and / or a communication interface, and the processor 1602, the communication interface, and the memory 1601 communicate with each other via the communication bus.

[0275] The communication bus mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0276] The communication interface is used for communication between the above electronic device and other devices.

[0277] The memory may include a random access memory (RAM) or a non-volatile memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0278] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0279] In another embodiment provided in the present application, a computer-readable storage medium is also provided, in which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above-mentioned description generation model training methods or any of the above-mentioned description generation methods are implemented.

[0280] In another embodiment provided in the present application, a computer program product comprising instructions is also provided, which, when executed on a computer, enables the computer to execute any of the description generation model training methods in the above embodiments or any of the description generation methods in the above embodiments.

[0281] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a solid-state hard disk (SSD), etc.

[0282] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0283] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the description generation method, device, electronic device, computer-readable storage medium, and computer program product embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.

[0284] The above description is only a preferred embodiment of the present application and is not intended to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application are included in the protection scope of the present application.

Claims

1. A description generation model training method, characterized in that: The method comprises: Using the first text prompt and the current description generation model, each current first sample image is processed to obtain a description text for describing a specified image area in the first sample image as the current first sample description text; wherein the first text prompt is used to instruct the generation of a description text for the specified image area in the input image; Determine, from each of the current first sample description texts, a description text that matches the designated image area in the corresponding first sample image as the current second sample description text; Inputting the first sample image and the first text prompt corresponding to each current second sample description text into the current description generation model to obtain the description text of the specified image area in the first sample image as the current first predicted description text; Based on the difference between the current first predicted description text and the current second sample description text, the parameters of the current description generation model are adjusted to obtain a new description generation model.

2. The method according to claim 1, characterized in that: The step of determining, from the current first sample description texts, a description text that matches a designated image area in the corresponding first sample image as the current second sample description text comprises: Input each current first sample image, the first sample description text corresponding to the first sample image, and the second text prompt into a pre-trained reward model to obtain a matching result between a specified image area in the first sample image and the corresponding first sample description text; wherein the second text prompt is used to indicate whether the specified image area in the input image matches the input description text; the reward model is pre-trained based on positive sample pairs and negative sample pairs; a positive sample pair includes: a second sample image, a specified image area in the second sample image, and a description text that matches the specified image area in the second sample image; a negative sample pair includes: a third sample image, a specified image area in the third sample image, and a description text that does not match the specified image area in the third sample image; When the obtained matching result indicates that the designated image area in the first sample image matches the corresponding first sample description text, the first sample description text corresponding to the first sample image is determined as the corresponding second sample description text.

3. The method according to claim 2, characterized in that The reward model includes: a first large language model, a first adapter, a first image encoder and a first tag parser; the first image encoder is used to extract features of an input image; the first adapter is used to convert the image features output by the first image encoder into a tag form; the first tag parser is used to convert the input text into a tag form.

4. The method according to claim 3, characterized in that The description text in each sample pair is obtained by processing the sample image in the sample pair using the first text prompt and the current description generation model; the reward model is trained by the following steps: For each sample pair, the sample image and the specified image region in the sample pair are input into the first image encoder in the reward model of the initial structure to obtain the features of the sample image and the features of the specified image region in the sample pair, and the obtained features are spliced; Input the obtained splicing result into the first adapter in the reward model of the initial structure to obtain a marked form of the splicing result; Inputting the second text prompt and the description text in the sample pair into the first token parser in the reward model of the initial structure to obtain the token form of the second text prompt and the token form of the description text in the sample pair; Inputting the marked form of the splicing result, the marked form of the second text prompt, and the marked form of the description text in the sample pair into the first language model in the reward model of the initial structure, and obtaining a predicted matching result between the specified image area in the sample pair and the description text; Based on the difference between the predicted matching result and the actual matching result of the sample pair, the parameters of the reward model of the initial structure are adjusted until the preset convergence condition is reached to obtain a trained reward model.

5. The method according to claim 1, characterized in that The method of using the first text prompt and the current description generation model to process each current first sample image to obtain a description text for describing a specified image area in the first sample image as the current first sample description text includes: For each current first sample image, when the proportion of the designated image area in the first sample image is less than a preset threshold, the designated image area in the first sample image is expanded to obtain an image to be processed; Inputting the obtained image to be processed and the first text prompt into the current description generation model to obtain a description text for describing the specified image area in the first sample image as the current first sample description text; When the proportion of the specified image area in the first sample image is not less than the preset threshold, the first sample image and the first text prompt are input into the current description generation model to obtain a description text for describing the specified image area in the first sample image as the current first sample description text.

6. The method according to claim 1, characterized in that The method of using the first text prompt and the current description generation model to process each current first sample image to obtain a description text for describing a specified image area in the first sample image as the current first sample description text includes: The first sample image is processed according to different sampling parameters using the first text prompt and the current description generation model to obtain a plurality of description texts of a designated image area in the first sample image as the current first sample description text.

7. The method according to claim 1, characterized in that After adjusting the parameters of the current description generation model based on the difference between the current first predicted description text and the current second sample description text to obtain a new description generation model, the method further includes: Acquire a fourth sample image; wherein, among the description texts generated by the current description generation model for each fourth sample image, there are description texts that do not match the specified image area in the corresponding fourth sample image; Using the first text prompt and the current description generation model, each fourth sample image obtained is processed according to different sampling parameters to obtain a plurality of description texts for a specified image area in the fourth sample image; Determine, from the obtained description texts, at least one positive sample description text that matches the specified image region in the fourth sample image, and at least one negative sample description text that does not match the specified image region in the fourth sample image; Inputting the fourth sample image and the first text prompt into a current description generation model to obtain a description text of a specified image area in the fourth sample image as a current second predicted description text; Based on the difference between the current second predicted description text and the determined positive sample description text, and the difference between the current second predicted description text and the determined negative sample description text, the parameters of the current description generation model are adjusted to obtain a new description generation model.

8. The method according to claim 1, characterized in that The first text prompt is also used to indicate that: the current description generation model generates a description text that conforms to a first specified description style; a description style is used to indicate the language of the description text and / or the number of sentences included; After adjusting the parameters of the current description generation model based on the difference between the current first predicted description text and the current second sample description text to obtain a new description generation model, the method further includes: Using the first text prompt and the current description generation model, each current first sample image is processed to obtain a description text of the first specified description style for a specified image area in the first sample image as the current third sample description text; Adjusting the description style of the current third sample description text from the first specified description style to the second specified description style to obtain a current fourth sample description text; Determine, from the current fourth sample description text, a description text that matches the designated image area in the corresponding first sample image, to obtain the current fifth sample description text; Inputting the first sample image and the third text prompt corresponding to each fifth sample description text into the current description generation model, obtaining the description text of the second specified description style of the specified image area in the first sample image as the current third predicted description text; Based on the difference between the current third predicted description text and the current fifth sample description text, the parameters of the current description generation model are adjusted to obtain a new description generation model.

9. The method according to claim 8, characterized in that After adjusting the parameters of the current description generation model based on the difference between the current third predicted description text and the current fifth sample description text to obtain a new description generation model, the method further includes: Input each current first sample image and the third text prompt into the current description generation model to obtain a description text of the second specified description style for a specified image area in the first sample image as the current fourth sample description text; Return to the step of determining, from the current fourth sample description text, a description text that matches the designated image area in the corresponding first sample image to obtain the current fifth sample description text, until the first preset condition is met to obtain a new description generation model.

10. The method according to any one of claims 1 to 9, characterized in that: The description generation model includes: a second language model, a second adapter, a second image encoder and a second token parser; the second image encoder is used to extract features of an input image; the second adapter is used to convert image features output by the second image encoder into a token form; the second token parser is used to convert input text into a token form.

11. A description generation method, characterized in that: The method comprises: Obtaining images to be used and text prompts to be used; The image to be used and the text prompt to be used are input into a pre-trained description generation model to obtain a description text of the image area indicated by the text prompt to be used in the image to be used; wherein the description generation model is trained based on any one of the methods described in claims 1-10.

12. The method according to claim 11, characterized in that The position of the image area indicated by the to-be-used text prompt in the to-be-used image is obtained by the following steps: Displaying the image to be used; In response to a user's frame selection operation on the displayed image to be used, a position indicated by the frame selection operation in the image to be used is determined, and a position of an image area indicated by the text prompt to be used in the image to be used is obtained.

13. A description generation model training device, characterized in that: The device comprises: A first sample description text generation module is used to process each current first sample image using the first text prompt and the current description generation model to obtain a description text for describing a specified image area in the first sample image as the current first sample description text; wherein the first text prompt is used to indicate the generation of a description text for the specified image area in the input image; A second sample description text determination module is used to determine, from the current first sample description texts, a description text that matches the designated image area in the corresponding first sample image as the current second sample description text; A first predicted description text generation module, used for inputting the first sample image and the first text prompt corresponding to each current second sample description text into the current description generation model, and obtaining the description text of the specified image area in the first sample image as the current first predicted description text; The first parameter adjustment module is used to adjust the parameters of the current description generation model based on the difference between the current first predicted description text and the current second sample description text to obtain a new description generation model.

14. The device according to claim 13, characterized in that The second sample description text determination module is specifically used to input each current first sample image, the first sample description text corresponding to the first sample image, and the second text prompt into a pre-trained reward model to obtain a matching result between a specified image area in the first sample image and the corresponding first sample description text; wherein the second text prompt is used to indicate whether the specified image area in the input image matches the input description text; the reward model is pre-trained based on positive sample pairs and negative sample pairs; a positive sample pair includes: a second sample image, a specified image area in the second sample image, and a description text that matches the specified image area in the second sample image; a negative sample pair includes: a third sample image, a specified image area in the third sample image, and a description text that does not match the specified image area in the third sample image; when the matching result obtained indicates that the specified image area in the first sample image matches the corresponding first sample description text, the first sample description text corresponding to the first sample image is determined as the corresponding second sample description text; and / or, The reward model includes: a first large language model, a first adapter, a first image encoder and a first token parser; the first image encoder is used to extract features of an input image; the first adapter is used to convert the image features output by the first image encoder into a token form; the first token parser is used to convert the input text into a token form; and / or, The description text in each sample pair is obtained by processing the sample image in the sample pair using the first text prompt and the current description generation model; the device also includes: A reward model training module is used to input, for each sample pair, the sample image and the designated image area in the sample pair into the first image encoder in the reward model of the initial structure, obtain the features of the sample image and the designated image area in the sample pair, and splice the obtained features; input the obtained splicing result into the first adapter in the reward model of the initial structure to obtain the marked form of the splicing result; input the second text prompt and the description text in the sample pair into the first mark parser in the reward model of the initial structure to obtain the marked form of the second text prompt and the marked form of the description text in the sample pair; input the marked form of the splicing result, the marked form of the second text prompt and the marked form of the description text in the sample pair into the first large language model in the reward model of the initial structure to obtain the predicted matching result between the designated image area in the sample pair and the description text; based on the difference between the obtained predicted matching result and the actual matching result of the sample pair, adjust the parameters of the reward model of the initial structure until the preset convergence condition is reached to obtain a trained reward model; and / or, The first sample description text generation module is specifically used to, for each current first sample image, when the proportion of the specified image area in the first sample image is less than a preset threshold, expand the specified image area in the first sample image to obtain an image to be processed; input the obtained image to be processed and the first text prompt into the current description generation model to obtain a description text for describing the specified image area in the first sample image as the current first sample description text; when the proportion of the specified image area in the first sample image is not less than the preset threshold, input the first sample image and the first text prompt into the current description generation model to obtain a description text for describing the specified image area in the first sample image as the current first sample description text; and / or, The first sample description text generation module is specifically used to use the first text prompt and the current description generation model to process the first sample image according to different sampling parameters to obtain multiple description texts of the specified image area in the first sample image as the current first sample description text; and / or, The device also includes: A fourth sample image acquisition module, configured to acquire a fourth sample image after adjusting the parameters of the current description generation model based on the difference between the current first predicted description text and the current second sample description text to obtain a new description generation model; wherein, among the description texts generated by the current description generation model for each fourth sample image, there are description texts that do not match the specified image area in the corresponding fourth sample image; A description text inference module, used to process each fourth sample image acquired according to different sampling parameters by using the first text prompt and the current description generation model, to obtain a plurality of description texts of a specified image area in the fourth sample image; a positive and negative sample description text determination module, used to determine, from the obtained description texts, at least one positive sample description text that matches the specified image area in the fourth sample image, and at least one negative sample description text that does not match the specified image area in the fourth sample image; A second predicted description text generation module, used for inputting the fourth sample image and the first text prompt into a current description generation model to obtain a description text of a specified image area in the fourth sample image as a current second predicted description text; A second parameter adjustment module, used to adjust the parameters of the current description generation model based on the difference between the current second predicted description text and the determined positive sample description text, and the difference between the current second predicted description text and the determined negative sample description text, to obtain a new description generation model; and / or, The first text prompt is also used to indicate that: the current description generation model generates a description text that conforms to a first specified description style; a description style is used to indicate the language of the description text and / or the number of sentences included; The device also includes: A third sample description text generation module is used for adjusting the parameters of the current description generation model based on the difference between the current first predicted description text and the current second sample description text to obtain a new description generation model, and then using the first text prompt and the current description generation model to process each current first sample image to obtain a description text of the first specified description style for a specified image area in the first sample image as the current third sample description text; a fourth sample description text generating module, configured to adjust the description style of the current third sample description text from the first specified description style to the second specified description style, to obtain a current fourth sample description text; A fifth sample description text determination module determines, from the current fourth sample description text, a description text that matches the designated image area in the corresponding first sample image, to obtain the current fifth sample description text; A third predicted description text generation module is used to input the first sample image and the third text prompt corresponding to each fifth sample description text into the current description generation model, and obtain the description text of the second specified description style of the specified image area in the first sample image as the current third predicted description text; a third parameter adjustment model, used to adjust the parameters of the current description generation model based on the difference between the current third predicted description text and the current fifth sample description text, so as to obtain a new description generation model; and / or, The device also includes: a fourth sample description text updating module, for adjusting the parameters of the current description generation model based on the difference between the current third predicted description text and the current fifth sample description text, and after obtaining a new description generation model, inputting each current first sample image and the third text prompt into the current description generation model, and obtaining a description text of the second specified description style of the specified image area in the first sample image as the current fourth sample description text; and triggering the fifth sample description text determination module until the first preset condition is met, and a new description generation model is obtained; and / or, The device also includes: a triggering module, which adjusts the parameters of the current description generation model based on the difference between the current first predicted description text and the current second sample description text to obtain a new description generation model, and then triggers the first sample description text generation module until a second preset condition is met to obtain a new description generation model; and / or, The description generation model includes: a second language model, a second adapter, a second image encoder and a second token parser; the second image encoder is used to extract features of an input image; the second adapter is used to convert image features output by the second image encoder into a token form; the second token parser is used to convert input text into a token form.

15. A description generating device, characterized in that: The device comprises: A data acquisition module, used to acquire images to be used and text prompts to be used; A description text generation module is used to input the image to be used and the text prompt to be used into a pre-trained description generation model to obtain a description text of the image area indicated by the text prompt to be used in the image to be used; wherein the description generation model is trained based on any one of the methods described in claims 1-10.

16. The device according to claim 15, characterized in that The position of the image area indicated by the to-be-used text prompt in the to-be-used image is obtained by the following steps: Displaying the image to be used; In response to a user's frame selection operation on the displayed image to be used, determining a position indicated by the frame selection operation in the image to be used, and obtaining a position of an image area indicated by the text prompt to be used in the image to be used; and / or, The description style of the obtained description text is the description style indicated by the text prompt to be used; a description style is used to indicate the language of the description text and / or the number of sentences included.

Images