Image description generation model processing method and apparatus, and computer device and storage medium

Abstract

The embodiments of the present application belong to the technical field of computers. Disclosed are an image description generation model processing method and apparatus, and a computer device and a storage medium. The method comprises: by means of a first image description generation model, generating first description text corresponding to a first image, and by means of a description correction model, acquiring first corrected description text; by means of a second image description generation model, generating second description text corresponding to the first image, and by means of the description correction model, acquiring a quality score of the second description text and second corrected description text; and when the quality score of the second description text does not satisfy a quality condition, determining a loss value on the basis of the first description text, the first corrected description text, the second description text and the second corrected description text, and updating the second image description generation model with the aim of reducing the loss value.

Description

Image description generation model processing method and device, computer device, and storage medium

[0001] The present application claims priority from the Chinese patent application No. 202410729166.0 filed on June 5, 2024, and entitled "Model processing method and device, computer device, and storage medium", the content of which is incorporated herein by reference in its entirety. TECHNICAL FIELD

[0002] Embodiments of the present application relate to the field of computer technology, and particularly relate to an image description generation model processing method and device, a computer device, and a storage medium. BACKGROUND

[0003] Image description generation technology refers to a technology of generating a text describing an image given the image. With the development of computer vision technology and natural language processing technology, image description generation technology has achieved wide application.

[0004] In the related art, an image description generation model is usually trained based on training samples. After training, the image description generation model can be used to generate a description text corresponding to any image. The training samples include images and description texts corresponding to the images.

[0005] TECHNICAL CONTENT

[0006] Embodiments of the present application provide an image description generation model processing method and device, a computer device, and a storage medium. The technical solution is as follows:

[0007] Some embodiments of the present application provide an image description generation model processing method, which includes:

[0008] generating a first description text corresponding to a first image by a first image description generation model, and correcting the first description text by a description correction model to obtain a first corrected description text;

[0009] generating a second description text corresponding to the first image by a second image description generation model, obtaining a quality score of the second description text by the description correction model, and correcting the second description text by the description correction model to obtain a second corrected description text;

[0010] In a case where the quality score of the second description text does not satisfy the quality condition, a loss value is determined based on the first description text, the first revised description text, the second description text and the second revised description text, and the second image description generation model is updated to reduce the loss value, the updated second image description generation model being used to generate a text describing an input image according to the input image, and the second image description generation model before the update having the same structure and state as the first image description generation model.

[0011] Some embodiments of the present application further provide an image description generation model processing apparatus, the apparatus comprising:

[0012] a first description generation module configured to generate a first description text corresponding to a first image by a first image description generation model, and revise the first description text by a description revision model to obtain a first revised description text;

[0013] a second description generation module configured to generate a second description text corresponding to the first image by a second image description generation model, obtain a quality score of the second description text by the description revision model, and revise the second description text by the description revision model to obtain a second revised description text;

[0014] a model update module configured to, in a case where the quality score of the second description text does not satisfy a quality condition, determine a loss value based on the first description text, the first revised description text, the second description text and the second revised description text, and update the second image description generation model to reduce the loss value, the updated second image description generation model being used to generate a text describing an input image according to the input image, and the second image description generation model before the update having the same structure and state as the first image description generation model.

[0015] Some embodiments of the present application further provide a computer device, the computer device comprising a processor and a memory, the memory storing at least one computer program, the at least one computer program being loaded and executed by the processor to implement the operations performed by the model processing method according to the above aspect.

[0016] Some embodiments of the present application further provide a computer readable storage medium, the computer readable storage medium storing at least one computer program, the at least one computer program being loaded and executed by a processor to implement the operations performed by the model processing method according to the above aspect.

[0017] Some embodiments of the present application further provide a computer program product comprising a computer program loaded and executed by a processor to implement the operations performed by the model processing method according to the above aspects.

[0018] BRIEF DESCRIPTION OF DRAWINGS

[0019] FIG. 1 is a schematic diagram of an implementation environment according to an embodiment of the present application;

[0020] FIG. 2 is a flowchart of an image description generation model processing method according to an embodiment of the present application;

[0021] FIG. 3 is a schematic diagram of an update process of a second image description generation model according to an embodiment of the present application;

[0022] FIG. 4 is a flowchart of another image description generation model processing method according to an embodiment of the present application;

[0023] FIG. 5 is a flowchart of a combined multi-dimensional description text according to an embodiment of the present application;

[0024] FIG. 6 is a flowchart of a training image description generation model according to an embodiment of the present application;

[0025] FIG. 7 is a flowchart of a training description correction model according to an embodiment of the present application;

[0026] FIG. 8 is a flowchart of another image description generation model processing method according to an embodiment of the present application;

[0027] FIG. 9 is an architecture diagram of an image description generation model according to an embodiment of the present application;

[0028] FIG. 10 is a schematic diagram of a training phase and an update phase according to an embodiment of the present application;

[0029] FIG. 11 is a schematic diagram of another training phase according to an embodiment of the present application;

[0030] FIG. 12 is a structural schematic diagram of an image description generation model processing apparatus according to an embodiment of the present application;

[0031] FIG. 13 is a structural schematic diagram of another image description generation model processing apparatus according to an embodiment of the present application;

[0032] FIG. 14 is a structural schematic diagram of a terminal according to an embodiment of the present application;

[0033] FIG. 15 is a structural schematic diagram of a server according to an embodiment of the present application. DETAILED DESCRIPTION

[0034] In order to make the purposes, technical solutions, and advantages of the embodiments of the present application clearer, the following further describes the embodiments of the present application in detail with reference to the drawings.

[0035] It can be understood that the terms "first", "second", and the like used in the present application can be used herein to describe various concepts, but unless specifically stated, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another concept. For example, without departing from the scope of the present application, a first image can be referred to as a second image, and similarly, a second image can be referred to as a first image.

[0036] At least two refers to two or more than two, for example, at least two images can be two images, three images, or any integer greater than or equal to two images. Each refers to each of the at least two, for example, each image refers to each of the at least two images, and if the at least two images are three images, each image refers to each of the three images.

[0037] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data for analysis, stored data, displayed data, etc.), and signals (including but not limited to signals transmitted between user terminals and other devices, etc.) involved in the present application are all fully authorized by the user or relevant parties, and the collection, use, and processing of relevant data need to comply with relevant laws, regulations, and standards of relevant countries and regions.

[0038] The scheme provided by the embodiments of the present application relates to computer vision and natural language processing technologies of artificial intelligence, and is specifically explained by the following embodiments:

[0039] The method provided by the embodiments of the present application is used in a computer device. In some embodiments, the computer device is a terminal or a server. In some embodiments, the terminal is a smart phone, a tablet computer, a notebook computer, a desktop computer, a smart speaker, a smart watch, a smart voice interaction device, a smart home appliance, a vehicle-mounted terminal, an aircraft, etc., but is not limited thereto. In some embodiments, the server is a standalone physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and basic cloud computing services such as big data and artificial intelligence platforms. The embodiments of the present application can be applied to various scenarios, including but not limited to cloud technology, artificial intelligence, intelligent transportation, and assisted driving.

[0040] In some embodiments, the computer program related to the embodiments of the present application can be deployed to execute on one computer device, or on multiple computer devices located in one place, or on multiple computer devices distributed in multiple places and interconnected through a communication network, which can constitute a blockchain system.

[0041] In some embodiments, the computer device in the embodiments of the present application is a node in the blockchain system, which can store the model in the blockchain, and then the node or the corresponding node in other devices in the blockchain can call the model by accessing the blockchain.

[0042] In the related art, a description generation model is usually trained based on training samples, and after training, the description generation model can be used to generate a description text corresponding to any image. The training samples include images and description texts corresponding to the images. Since the training samples are usually obtained from the Internet, the quality of the training samples is poor, which leads to poor performance of the trained image description generation model. The embodiments of the present application provide an image description generation model processing scheme. With the aid of a first image description generation model and a description correction model, even if the quality score of the second description text generated by the second image description generation model does not meet the quality condition, the second description text, the second corrected description text obtained by correcting the second description text by the description correction model, the first description text generated by the first image description generation model, and the first corrected description text obtained by correcting the first description text by the description correction model can be used to update the second image description generation model, thereby improving the performance of the second image description generation model, saving the number of samples, and avoiding wasting low-quality description texts.

[0043] FIG. 1 is a schematic diagram of an implementation environment provided by the embodiments of the present application. Referring to FIG. 1, the implementation environment includes a terminal 101 and a server 102. The terminal 101 and the server 102 are connected through a wired network or a wireless network.

[0044] The terminal 101 is installed and runs a client 111. The client 111 can be a video sharing client, a social application client, a game client, or the like. When the terminal 101 runs the client 111, a user interface of the client 111 is displayed on a screen of the terminal 101.

[0045] Those skilled in the art can know that the number of the terminals can be more or less. For example, the terminals can be only one, or the terminals can be six or eight or more. The number of the terminals and the type of the devices are not limited in the embodiments of the present application.

[0046] Only one terminal is shown in FIG. 1, but there are multiple other terminals 103 that can access the server 102 in different embodiments. In some embodiments, there is also one or more terminals 103 that are corresponding terminals of developers, and a development and editing platform of the client is installed on the terminal 103, and the developers can edit and update the client on the terminal 103, and transmit the updated client installation package to the server 102 through a wired or wireless network, and the terminal 101 can download the client installation package from the server 102 to implement updating of the client.

[0047] The terminal 101 and the other terminals 103 are connected to the server 102 through a wired network or a wireless network.

[0048] The server 102 is configured to provide background services for the client 111 on the terminal 101. In some embodiments, the server 102 undertakes main computing work, and the terminal 101 undertakes secondary computing work; or the server 102 undertakes secondary computing work, and the terminal 101 undertakes main computing work; or the server 102 and the terminal 101 adopt a distributed computing architecture to perform collaborative computing.

[0049] In the embodiments of the present application, the server 102 updates the second image description generation model by using the method provided in the embodiments of the present application, so as to improve the performance of the second image description generation model. Then, the server 102 generates a description text corresponding to an image by using the updated second image description generation model, so as to ensure that the description text can accurately and comprehensively describe the image.

[0050] In some embodiments, a user inputs an image by using the terminal 101, and the terminal 101 or the server 102 generates a description text corresponding to the image by using the second image description generation model.

[0051] For example, in the scenario of publishing a dynamic information, the terminal 101 determines an image to be published by a user, and sends the image to the server 102, and then the server 102 generates a description text corresponding to the image by using the updated second image description generation model, and sends the description text to the terminal 101, and the terminal 101 displays the image and the description text, and after the user clicks to confirm, the image and the description text are combined into a dynamic message and published to a social platform. The user only needs to select an image to be published, and the description text corresponding to the image can be automatically generated as a script matched with the image, without the need of the user to separately input the script, so that the dynamic message can be quickly published.

[0052] In another possible implementation, the server 102 generates a description text corresponding to an image by using the updated second image description generation model, and trains an image generation model based on the image and the description text corresponding to the image, so that the trained image generation model can generate an image conforming to any description text according to the description text. A user inputs a description text by using the terminal 101, and the terminal 101 or the server 102 generates an image conforming to the description text by using the image generation model.

[0053] For example, in the scenario of generating a creative image, a user inputs a description text in the terminal 101, the terminal 101 sends the description text to the server 102, and the server 102 generates an image conforming to the description text and having a unique creativity by using the image generation model, and sends the image to the terminal 101. Subsequently, the user can share the image with friends, or post the image on a social platform, or generate a video including the image, and the like.

[0054] It should be noted that the implementation environment described above is only an example, and the method provided in the embodiments of the present application can also be executed by the server 102 alone or by other computer devices, which are not limited in the embodiments of the present application.

[0055] FIG. 2 is a flowchart of an image description generation model processing method provided in the embodiments of the present application, and FIG. 3 is a schematic diagram of an updating flow of a second image description generation model provided in the embodiments of the present application. The embodiments of the present application are executed by a computer device, which is a terminal 101 or a server 102, and the like as shown in FIG. 1, and the embodiments of the present application describe a process of updating the image description generation model. Referring to FIGS. 2 and 3, the method includes the following steps.

[0056] 201. The computer device generates a first description text corresponding to a first image by using a first image description generation model, and corrects the first description text to obtain a first corrected description text by using a description correction model.

[0057] In the embodiments of the present application, the computer device obtains a first image description generation model, a second image description generation model, and a description correction model. The three models will be introduced respectively as follows.

[0058] First, the first image description generation model is used to generate a description text corresponding to an image, so as to describe the image with the description text. Since the first image description generation model has been trained for one or more times and has the ability to generate a description text corresponding to an image, the first image description generation model can be put into use, that is, for any image, the first image description generation model can generate a description text corresponding to the image. In some embodiments, the first image description generation model used by the embodiments of the present application is a multi-modal large language model, which is an artificial intelligence model capable of processing multiple types of data inputs such as text, image, audio, etc. and generating natural language output, and thus can be used to generate a description text corresponding to an image.

[0059] Second, the second image description generation model also has the function of generating a description corresponding to an image, so as to describe the image with the description text. Since the second image description generation model has been trained for one or more times and has the ability to generate a description text corresponding to an image, the second image description generation model can be put into use, that is, for any image, the second image description generation model can generate a description text corresponding to the image.

[0060] In some embodiments, the image description generation model (for example, the first image description generation model and the second image description generation model described above) can include two main components: an image feature extraction component and a text generation component.

[0061] The image feature extraction component is used to convert an image into a feature vector to facilitate subsequent text generation. This can be achieved by a convolutional neural network (CNN). The basic structure of CNN includes an input layer, multiple convolutional layers, a pooling layer and a fully connected layer. The convolutional layer is the core part of CNN, which is used to extract local features of the image. The pooling layer is used to simplify data and reduce computational complexity. The fully connected layer is used to integrate the features extracted by the convolutional layer and the pooling layer, and outputs a feature vector.

[0062] The text generation component is used to map the feature vector to a text sequence, which can be achieved by a recurrent neural network (RNN), a long short-term memory network (LSTM) or a Transformer.

[0063] The difference between the first image description generation model and the second image description generation model is that the first image description generation model remains unchanged, and the initial state of the second image description generation model is the same as that of the first image description generation model, that is, before the second image description generation model is updated by the method provided in the embodiments of the present application, the second image description generation model is the same as the first image description generation model, and after the second image description generation model is updated one or more times by the method provided in the embodiments of the present application, the second image description generation model has changed, although both the first image description generation model and the second image description generation model are used to generate description texts corresponding to images, the second image description generation model has better performance and the generated description texts are more accurate.

[0064] Again, the description correction model is used to correct the description texts corresponding to the images, and can correct the wrong words and inappropriate language in the description texts corresponding to the images to obtain more accurate description texts. In addition, the description correction model is also used for quality detection of the description texts corresponding to the images to obtain a quality score of the description texts, which indicates the quality of the description texts. In the embodiments of the present application, the description correction model has also been trained one or more times and has the ability to correct and quality detect the description texts. In some embodiments, the description correction model used in the embodiments of the present application is a multi-modal large language model.

[0065] In the embodiments of the present application, although the second image description generation model has been trained based on the training samples, the performance of the second image description generation model will be affected by the training samples, in order to improve the performance of the second image description generation model, the second image description generation model will be updated after being trained in the embodiments of the present application. The second image description generation model can be trained by a computer device and then updated, or trained by other devices and then sent to the computer device for updating.

[0066] Moreover, the process of updating the second image description generation model also needs the participation of the first image description generation model, and the computer device will take the first image description generation model as the reference model of the second image description generation model, and take the description text generated by the first image description generation model as the reference description text, together with the description text generated by the second image description generation model, to update the second image description generation model. In some embodiments, the first image description generation model can also be referred to as a reference image description generation model, and the first description text generated by the first image description generation model can also be referred to as a reference description text.

[0067] Embodiments of the present application take the first image as an example, which is any image that needs to generate a description text, such as an image input on a computer device, an image sent to the computer device by other devices (such as a terminal used by a user or a terminal used by a technician, etc.) or an image obtained in other ways, etc. The computer device generates a first description text corresponding to the first image (which can also be referred to as a sample image) through a first image description generation model. The first description text is a text used to describe the first image, such as describing the theme, background, style, composition, culture and emotion of the first image, and the quality of the first description text can reflect the performance of the first image description generation model. Moreover, a first modified description text corresponding to the first description text is obtained through a description modification model. Since the first modified description text has been modified by the description modification model, it can be considered that the accuracy of the first modified description text is higher than that of the first description text.

[0068] It should be noted that after the computer device generates the first description text in step 201, the first description text can also be applied to other tasks other than updating the first image description generation model, such as semantic analysis tasks, text classification tasks, reply dialogue tasks, etc., which are not limited by embodiments of the present application. For example, in the scene of intelligent question answering, after the computer device generates the first description text according to the image input by the user, the computer device performs semantic analysis on the first description text, generates a reply text based on the semantics of the first description text, and feeds back the reply text to the user, thereby realizing the effect of the computer device and the user's image dialogue.

[0069] 202、The computer device generates a second description text corresponding to the first image through a second image description generation model, obtains a quality score of the second description text through a description modification model, and modifies the second description text through the description modification model to obtain a second modified description text.

[0070] For the same first image, the computer device also generates a second description text corresponding to the first image through a second image description generation model, and obtains a quality score of the second description text and a second modified description text corresponding to the second description text through a description modification model.

[0071] The second description text is a text used to describe the first image, such as describing the theme, background, style, composition, culture and emotion of the first image, and the quality of the second description text can reflect the performance of the second image description generation model. Moreover, a second modified description text corresponding to the second description text is obtained through a description modification model.

[0072] For example, the second description text is "a man wearing a green shirt and gray pants", and the second modified description text obtained after modification is "a man wearing a green shirt and brown pants, the front of the shirt has green prints and English", the color of the pants is modified to the correct color, and more detailed description of the shirt is added.

[0073] In the initial state, the first image description generation model and the second image description generation model are the same model, and it can be considered that the accuracy of the second description text is equal to the accuracy of the first description text. Then, since the first image description generation model remains unchanged, and the second image description generation model is updated one or more times, the performance gradually improves, so it can be considered that the accuracy of the second description text is higher than the accuracy of the first description text. Since the second modified description text has been modified by the description modification model, it can be considered that the accuracy of the second modified description text is higher than the accuracy of the second description text, and is also very likely to be higher than the accuracy of the first modified description text.

[0074] The quality score of the second description text represents the quality of the second description text, and also represents the performance of the second image description generation model to some extent. In some embodiments, the higher the quality score, the more accurately and comprehensively the second description text describes the first image. The lower the quality score, the less accurately and comprehensively the second description text describes the first image.

[0075] In some embodiments, the quality score can be represented in numerical form or in alphabetical form, for example, the quality score can be represented as 1, 2, 3, etc. The higher or lower the quality score is determined by the size of the score. Or the quality score can be represented as A level, B level, etc. The higher or lower the quality score is determined by the level.

[0076] 203、In the case that the quality score of the second description text does not satisfy the quality condition, the computer device determines a loss value based on the first description text, the first modified description text, the second description text and the second modified description text, and updates the second image description generation model with the loss value as the target.

[0077] The quality threshold refers to the condition that the quality score of a high-quality description text should satisfy. The quality condition can be set by default by the computer device or by the technical personnel according to the needs.

[0078] In the embodiments of the present application, considering that the quality of the description text will affect the performance of the second image description generation model, the quality condition is used to divide the description text, and different ways are used to update the second image description generation model for the description text that satisfies the quality condition and the description text that does not satisfy the quality condition.

[0079] The embodiments of the present application only take the first image as an example. After the second description text corresponding to the first image is generated by the second image description generation model and the quality score of the second description text is obtained by the description correction model, it is determined whether the quality score of the second description text meets the quality condition. In the case where the quality score of the second description text does not meet the quality condition, the second description text is a low-quality description text. At this time, if the first image and the second description text are directly used to update the second image description generation model, the second image description generation model will learn incorrect knowledge, which will affect the performance of the second image description generation model. If the second description text is directly filtered out, the training samples will be wasted, and the number of samples will be reduced.

[0080] Therefore, after the computer device obtains four kinds of description texts corresponding to the first image: the first description text, the first corrected description text, the second description text and the second corrected description text, in the case where the quality parameter of the second description text does not meet the quality condition, the four kinds of description texts can be considered together to update the second image description generation model through the four kinds of description texts.

[0081] In the case of the given first image, the first image can be described by multiple description texts, and the probability of occurrence of each description text in the multiple description texts is different. The description text with high occurrence probability can more accurately describe the first image. If the performance of the second image description generation model is to be improved, the second image description generation model is required to generate description texts with high accuracy as much as possible and generate description texts with low accuracy as little as possible.

[0082] Since the accuracy of the second corrected description text is likely to be the highest among the above four kinds of description texts, the updating target of the second image description generation model is to increase the probability of the second corrected description text obtained by the updated second image description generation model and the description correction model, that is, to generate more accurate description texts. Therefore, although the quality score of the second description text does not meet the quality condition, as long as the second image description generation model is updated according to the above updating target, the performance of the updated second image description generation model can still be higher than that of the second image description generation model before updating.

[0083] In some embodiments, step 203 comprises: determining a first probability that the first image description generation model outputs the first description text, a second probability that the description revision model outputs the first revised description text, a third probability that the second image description generation model outputs the second description text, and a fourth probability that the description revision model outputs the second revised description text. The loss value is determined according to the first probability, the second probability, the third probability, and the fourth probability, wherein the loss value is positively correlated with a logarithm of a ratio of the third probability to the first probability, and is negatively correlated with a logarithm of a ratio of the fourth probability to the second probability, and the second image description generation model is updated with a target of reducing the loss value.

[0084] wherein the logarithm of the ratio of the third probability to the first probability represents a difference between a logarithm of the third probability and a logarithm of the first probability, and the logarithm of the ratio of the fourth probability to the second probability represents a difference between a logarithm of the fourth probability and a logarithm of the second probability.

[0085] The target of updating the second image description generation model is to reduce the loss value, and the smaller the loss value is, the smaller the difference between the logarithm of the third probability and the logarithm of the first probability is, which means that the probability of the second image description generation model generating the second description text is closer to the probability of the first image description generation model generating the first description text, i.e., the probabilities of outputting the first description text and the second description text for the first image are both reduced, and the probability of outputting the second description text is reduced more because the quality score of the second description text does not satisfy the quality condition. The smaller the loss value is, the greater the difference between the logarithm of the fourth probability and the logarithm of the second probability is, i.e., the fourth probability is greater and the second probability is smaller, i.e., the second image description generation model is more inclined to output the second revised description text and less inclined to output the first revised description text for the first image. Therefore, updating the second image description generation model with this target can increase the probability of obtaining the second revised description text through the updated second image description generation model and the description revision model, thereby improving the performance of the second image description generation model.

[0086] For example, the loss value, the first probability, the second probability, the third probability, and the fourth probability satisfy the following relationship:

[0087] wherein L(π θ ; π ref ) represents the loss value, E represents expectation, θ refers to the second image description generation model, ref refers to the first image description generation model, x refers to the image, y l refers to the description text generated by the image description generation model, y w refers to the description text revised by the description revision model, β represents a hyperparameter, and σ represents a sigmiod function (S-shaped function). Therefore, π θ(y w / x) represents the fourth probability, π ref (y w / x) represents the second probability, π θ (y l / x) represents the third probability, π ref (y l / x) represents the first probability.

[0088] In some embodiments, the method can further include:

[0089] 204、The computer device updates the second image description generation model based on the first image and the second description text in a case where the quality score of the second description text meets the quality condition.

[0090] In a case where the quality score of the second description text meets the quality condition, it indicates that the second description text is a high-quality description text, and the first image and the second description text can be used to update the second image description generation model, thereby increasing the number of samples and enabling the second image description generation model to learn the correct knowledge contained in the second description text, thereby effectively improving the performance of the second image description generation model.

[0091] In some embodiments, in a case where the quality score of the second description text meets the quality condition, the first image and the second description text constitute a set of samples and are added to the training set of the second image description generation model, and then the second image description generation model is updated based on one or more sets of samples in the training set. Since only high-quality description texts are stored in the training set, updating the second image description generation model based on high-quality description texts can enable the second image description generation model to learn correct knowledge, thereby effectively improving the performance.

[0092] In the related art, an image description generation model is usually trained based on training samples, and after training, the image description generation model can be used to generate a description text corresponding to any image. The training samples include images and description texts corresponding to the images. However, the accuracy of the description text generated by the description generation model is insufficient, and there is a serious "hallucination" problem, that is, the description text contains incorrect description information, and the description generation model cannot be updated by self-learning.

[0093] The method provided by the embodiments of the present application can use the second description text, the second corrected description text, the first description text, and the first corrected description text to update the second image description generation model even in a case where the quality of the second description text generated by the second image description generation model does not meet the quality condition, thereby improving the performance of the second image description generation model, saving the number of samples, and avoiding wasting low-quality description texts.

[0094] In addition, in a case where the quality score of the second description text generated by the second image description generation model satisfies the quality condition, it is indicated that the second description text is a high-quality description text, and then the high-quality second description text and the first image corresponding to the second description text are used to update the second image description generation model, so that the second image description generation model is updated by self-learning, and the performance of the second image description generation model is improved.

[0095] For example, in a scenario of continuously processing images, the second image description generation model can realize self-learning update based on newly processed images, and iteratively update the performance of the second image description generation model to adapt to the demand of continuously processing images in real-time scenarios.

[0096] On the basis of the above-mentioned embodiments, the following describes a manner of determining whether the quality score of the second description text satisfies the quality condition.

[0097] In some embodiments, the quality condition includes a quality score threshold, and in a case where the quality score of the second description text is less than or equal to the quality score threshold, it is determined that the second description text does not satisfy the quality condition.

[0098] The quality score threshold is a threshold for distinguishing high-quality description texts and low-quality description texts. For example, the value range of the quality score is the interval [0, 1], the quality score threshold is 0.5 or 0.8, or the value range of the quality score is the interval [0, 10], the quality score threshold is 5 or 8, or the quality score belongs to the set {A, B, C, D, E}, and the quality score threshold is C, that is, the description text with the quality score of A, B or C is a low-quality description text, and the description text with the quality score of D or E is a high-quality description text.

[0099] The quality score threshold can be set by the computer device or set by the technician according to the demand. Alternatively, in a case where the computer device obtains a plurality of description texts, the quality score threshold is determined based on the quality scores of the plurality of description texts, for example, the quality score threshold is equal to the average of the quality scores of the plurality of description texts, or the quality score threshold is equal to the product of the sum of the quality scores of the plurality of description texts and a preset proportion.

[0100] In a case where the quality score of the second description text is greater than the quality score threshold, it is indicated that the second description text is a high-quality description text, which satisfies the quality condition. In another embodiment, in a case where the quality score of the second description text is not greater than the quality score threshold, it is indicated that the second description text is a low-quality description text, which does not satisfy the quality condition.

[0101] The embodiment of the application screens the description text with the quality score greater than the quality score threshold as the high-quality description text, and screens the description text with the quality score not greater than the quality score threshold as the low-quality description text, so as to improve the flexibility by updating the second image description generation model in different ways based on the high-quality description text and the low-quality description text.

[0102] In some embodiments, the second image description generation model has the ability to generate multi-dimensional description text, and the second description text generated by the second image description generation model can describe the first image from N dimensions, N being an integer greater than 1, the N dimensions including at least two of theme, background, style, composition, culture and emotion, and other dimensions can also be included. The description modification model has the ability to detect the quality of the multi-dimensional description text, so the quality score obtained by detecting the quality of the second description text includes N sub-scores, each sub-score representing the quality of the second description text in describing the first image in the dimension corresponding to the sub-score. The quality condition includes N quality score thresholds of the dimensions, which are used to determine whether the quality of the second description text meets the requirements from the N dimensions.

[0103] The sub-score greater than the quality score threshold of the corresponding dimension is referred to as a high-quality sub-score, and the method further includes any one of the following:

[0104] (1) In the case where the N sub-scores are not all high-quality sub-scores, it is determined that the second description text does not meet the quality condition.

[0105] In another embodiment, the N sub-scores of the second description text are all high-quality sub-scores, indicating that the second description text can accurately describe the first image from each dimension, and belongs to high-quality description text, and it is determined that the second description text meets the quality condition.

[0106] (2) In the case where the number of high-quality sub-scores in the N sub-scores is not greater than a preset number, it is determined that the second description text does not meet the quality condition, and the preset number is less than N.

[0107] The preset number is set by default by the computer device or set by the technician according to the value of N. The number of high-quality sub-scores in the N sub-scores of the second description text is greater than the preset number, indicating that the second description text can accurately describe the first image from enough dimensions, and therefore belongs to high-quality description text and meets the quality condition. The number of high-quality sub-scores in the N sub-scores of the second description text is not greater than the preset number, indicating that the second description text can accurately describe the first image from fewer dimensions, and therefore belongs to low-quality description text and does not meet the quality condition.

[0108] (3) if the proportion of the high-quality sub-scores in the N-dimensional sub-scores is not greater than a preset proportion, it is determined that the second description text does not satisfy the quality condition, and the preset proportion is less than 100%.

[0109] The preset proportion is set by default by the computer device or set by the technician according to the demand. If the proportion of the high-quality sub-scores in the N-dimensional sub-scores of the second description text is greater than the preset proportion, it indicates that the second description text can accurately describe the first image from enough dimensions, and thus belongs to a high-quality description text and satisfies the quality condition. If the proportion of the high-quality sub-scores in the N-dimensional sub-scores of the second description text is not greater than the preset proportion, it indicates that the second description text can accurately describe the first image from fewer dimensions, and thus belongs to a low-quality description text and does not satisfy the quality condition.

[0110] In the embodiments of the present application, considering that the second description text generated by the second image description generation model is a multi-dimensional description text, a screening scheme suitable for the multi-dimensional description text is provided. A sub-score greater than the quality score threshold of the corresponding dimension in the quality score of the second description text is referred to as a high-quality sub-score. Based on the number of high-quality sub-scores in the N-dimensional sub-scores of the second description text, high-quality description texts and low-quality description texts are screened out. The embodiments of the present application provide a scheme for accurately screening high-quality and low-quality multi-dimensional description texts, improve the accuracy of screening, and thus improve the performance of the second image description generation model.

[0111] In another possible implementation, the computer device can generate second description texts corresponding to a plurality of first images respectively, thereby obtaining a plurality of second description texts, and obtain quality scores of the plurality of second description texts through the description correction model respectively. Then, the plurality of second description texts are filtered based on the quality scores of the plurality of second description texts, and high-quality second description texts and corresponding first images, and low-quality second description texts and corresponding first images are distinguished, so that the second image description generation model is updated in different ways.

[0112] In some embodiments, the computer device stores, as a set of samples, each first image and a second description text corresponding to the first image generated by the computer device each time, and stores the set of samples into a sample set. As the number of samples gradually increases, when the number of samples in the sample set reaches a target sample number, the computer device obtains a quality score of each second description text in the sample set by using the description correction model, and divides the plurality of second description texts based on the quality scores of the plurality of second description texts. Alternatively, the computer device stores, as a set of samples, each first image and a second description text corresponding to the first image generated by the computer device each time, and stores the set of samples into a sample set. And every pre-set time length, the computer device obtains a quality score of the second description text by using the description correction model for the samples collected in the sample set within the pre-set time length, and divides the plurality of second description texts based on the quality scores of the plurality of second description texts.

[0113] In some embodiments, the quality condition comprises a quality ranking threshold.

[0114] When the position of the quality score of the second description text in the plurality of second description texts in a quality ranking from high to low is after the quality ranking threshold, it is determined that the second description text does not meet the quality condition. The plurality of second description texts are description texts corresponding to a plurality of images respectively and generated by the second image description model.

[0115] For example, the quality condition is that the quality score ranks in the top K of the plurality of second description texts. The method further comprises:

[0116] The computer device determines, according to the quality scores of the plurality of second description texts from large to small, that the second description text not ranking in the top K does not meet the quality condition.

[0117] Wherein K is a positive integer, and K is less than the number of the plurality of second description texts. K can be set by default by the computer device, set by a technician according to demand, or K is equal to the product of the number of the plurality of second description texts and a pre-set proportion, or K is equal to other numerical values, etc.

[0118] The computer device ranks the plurality of second description texts according to the quality scores of the plurality of second description texts from large to small. For any second description text, if the second description text ranks in the top K, it means that the second description text is a high-quality description text and meets the quality condition. If the second description text does not rank in the top K, it means that the second description text is a low-quality description text and does not meet the quality condition.

[0119] The embodiments of the present application rank the plurality of second description texts in descending order of the quality scores of the plurality of second description texts, take the top K second description texts as high-quality description texts, and take the second description texts not in the top K as low-quality description texts, so as to update the second image description generation model based on the high-quality description texts and the low-quality description texts in different manners respectively, thereby improving flexibility. The embodiments of the present application provide an accurate description text classification scheme, improve the accuracy of screening, and thereby improve the performance of the second image description generation model.

[0120] In some embodiments, the quality score of the second description text includes N-dimensional sub-scores, each of which represents the quality of the second description text in describing the first image in a dimension corresponding to the sub-score. The top K sub-scores in the same dimension of the plurality of second description texts are referred to as high-quality sub-scores, and the method further includes any one of the following:

[0121] (1) In the case where not all of the N-dimensional sub-scores are high-quality sub-scores, it is determined that the second description text does not meet the quality condition.

[0122] In another embodiment, the N-dimensional sub-scores of the second description text are all high-quality sub-scores, indicating that the second description text can accurately describe the first image from each dimension and belongs to a high-quality description text, and it is determined that the second description text meets the quality condition.

[0123] (2) In the case where the number of high-quality sub-scores in the N-dimensional sub-scores is not greater than a preset number, it is determined that the second description text does not meet the quality condition, and the preset number is less than N.

[0124] The preset number is set by default by the computer device or by the technical personnel according to the value of N. The number of high-quality sub-scores in the N-dimensional sub-scores of the second description text is greater than the preset number, indicating that the second description text can accurately describe the first image from enough dimensions and therefore belongs to a high-quality description text, meeting the quality condition. The number of high-quality sub-scores in the N-dimensional sub-scores of the second description text is not greater than the preset number, indicating that the second description text can accurately describe the first image from fewer dimensions and therefore belongs to a low-quality description text, not meeting the quality condition.

[0125] (3) In the case where the proportion of high-quality sub-scores in the N-dimensional sub-scores is not greater than a preset proportion, it is determined that the second description text does not meet the quality condition, and the preset proportion is less than 100%.

[0126] The preset proportion is set by default by the computer device or set by a technician according to a requirement. If the proportion of the high-quality sub-scores in the N-dimensional sub-scores of the second description text is greater than the preset proportion, it indicates that the second description text can accurately describe the first image from enough dimensions, and thus the second description text is a high-quality description text and meets the quality condition. If the proportion of the high-quality sub-scores in the N-dimensional sub-scores of the second description text is not greater than the preset proportion, it indicates that the second description text can accurately describe the first image from fewer dimensions, and thus the second description text is a low-quality description text and does not meet the quality condition.

[0127] In addition, the computer device can also use other ways to divide the quality of the plurality of second description texts, and the embodiment of the present application does not limit the division method.

[0128] In the embodiment of the present application, considering that the second description text generated by the second image description generation model is a multi-dimensional description text, a screening scheme suitable for the multi-dimensional description text is provided, the sub-scores of the plurality of second description texts are sorted according to different dimensions, the sub-scores in the same dimension of the plurality of second description texts that are ranked in the top K positions are referred to as high-quality sub-scores, and the high-quality description text and the low-quality description text are screened based on how many high-quality sub-scores in the N-dimensional sub-scores of the second description text. The embodiment of the present application provides a scheme for accurately screening high-quality multi-dimensional description texts, improves the accuracy of screening, and thus improves the performance of the second image description generation model.

[0129] On the basis of the above-mentioned embodiment, the process of training the second image description generation model is described below.

[0130] In some embodiments, after the computer device determines a plurality of groups of high-quality samples (the first image and the second description text meeting the quality condition), the process of training the second image description generation model includes:

[0131] For each group of samples, the description text corresponding to the first image is generated based on the current second image description generation model, and the model parameters of the second image description generation model are adjusted based on the error between the generated description text and the second description text to reduce the error to a target.

[0132] After the training of one or more groups of samples, the performance of the second image description generation model is improved, so that the probability of the second image description generation model generating a high-quality description text is increased.

[0133] On the basis of the above-mentioned embodiments, after the computer device updates the second image description generation model, the computer device can generate a description text corresponding to any image through the updated second image description generation model. Since the performance of the updated second image description generation model is stronger, the quality of the generated description text is improved. Alternatively, the image generation model can also be trained through the updated second image description generation model.

[0134] The image generation model is configured to generate an image conforming to the description text based on any description text. For example, a user inputs a piece of description text describing the appearance of an image that the user wants to generate, and the image generation model can generate an image conforming to the user's requirements.

[0135] The process of training the image generation model includes: generating a sample description text corresponding to a sample image through the updated second image description generation model, the sample description text being used to describe the sample image, and training the image generation model based on the sample image and the sample description text.

[0136] For the convenience of distinction, the image used for training the image generation model is referred to as a sample image, and the description text corresponding to the sample image is referred to as a sample description text. The sample image and the first image in the above-mentioned embodiments can include the same image or different images. The sample image can be sent to the computer device by another device, stored in the computer device by a technician, or obtained in other ways.

[0137] Since the current second image description generation model has been trained and updated and has strong performance, the sample description text generated through the second image description generation model has high quality and can accurately and comprehensively describe the sample image. Therefore, the image generation model can be trained based on the sample image as a reference sample and the sample description text as an input sample, so as to effectively improve the performance of the image generation model and enable the image generation model to generate an image more conforming to the input description text.

[0138] In some embodiments, the image generation model is trained based on the sample image and the sample description text, including: generating an image corresponding to the sample description text through the image generation model, and adjusting the model parameters of the image generation model based on the error between the generated image and the sample image to reduce the error to a target.

[0139] After the training of one or more groups of samples, the performance of the image generation model can be improved, so that the image generation model can generate high-quality images.

[0140] The training samples used for training the image generation model in the related art can contain a large amount of noise information, which can result in poor performance of the trained image generation model.

[0141] The embodiment of the present application also provides a scheme for training an image generation model. The image corresponding description text is generated by means of the second image description generation model, so that the image generation model is trained based on the image and the image corresponding description text. Since the current second image description generation model has been trained and updated, the performance is strong, so the quality of the description text generated by the second image description generation model is high, which can accurately and comprehensively describe the image, thereby effectively improving the performance of the image generation model, so that the image generation model can generate an image more consistent with the input description text. Moreover, the description text corresponding to the image does not need to be determined manually, which saves the labor cost and improves the training efficiency.

[0142] Moreover, since the second image description generation model realizes self-learning and updating, it is suitable for real-time scenarios. While continuously improving the performance of the second image description generation model, the image generation model can also be provided with more high-quality description texts, thereby enriching the sample quantity of the image generation model and effectively improving the performance of the image generation model.

[0143] On the basis of the above embodiment, the embodiment of the present application also provides another image description generation model processing method. FIG. 4 is a flowchart of another image description generation model processing method provided by the embodiment of the present application. The embodiment of the present application is executed by a computer device, which is a terminal 101 or a server 102 as shown in FIG. 1. In some embodiments, the first image description generation model and the second image description generation model are obtained by training an image description generation model to be trained. The embodiment of the present application describes the process of training the image description generation model to obtain the first image description generation model and the second image description generation model. Referring to FIG. 4, the method comprises:

[0144] 401. Generating a combined description text corresponding to a second image, wherein the combined description text is used to describe the second image from multiple dimensions.

[0145] In some embodiments, the generating of the combined description text corresponding to the second image comprises:

[0146] 4011. Generating, by a plurality of third image description generation models, a plurality of dimension description texts corresponding to the second image respectively; wherein each dimension description text is used to describe the second image from the dimension; and each third image description generation model is used to generate a dimension description text corresponding to the second image.

[0147] 4012. Combining the plurality of dimension description texts by a second large language model to obtain the combined description text.

[0148] For the convenience of distinguishing, the image used for training the first image description generation model is referred to as a second image, which can include different images from the first image in the above embodiment. The second image can be sent to the computer device by other devices, stored in the computer device by technical personnel, or obtained in other ways.

[0149] Considering that if the description text corresponding to the second image is manually determined, the artificial cost is large and the training efficiency is low, therefore, in the process of training the first image description generation model, the third image description generation model is needed to generate the description text corresponding to the second image, so as to save the artificial cost and improve the training efficiency. The third image description generation model has similar functions to the first image description generation model, and is also a model for generating description text corresponding to an image. Therefore, step 401 is the same as step 201 described above. The difference is that the first image description generation model and the third image description generation model are different description generation models. For example, the model architecture of the first image description generation model and the third image description generation model is the same, the training samples used in the training process are different, or the model architecture of the first image description generation model and the third image description generation model is different, and the training samples used in the training process include the same training samples and different training samples.

[0150] In addition, the plurality of third image description generation models can also be the same model or different models, for example, the model architecture of the plurality of third image description generation models is the same, and the training samples used in the training process are different, so that the description texts generated by the plurality of third image description generation models for the same image can be different.

[0151] In the embodiments of the present application, since the plurality of third image description generation models have been trained one or more times and have the ability to generate description texts corresponding to images, the generation results of the plurality of third image description generation models can be used as training samples to train the first image description generation model.

[0152] In some embodiments, the third image description generation model used in the embodiments of the present application is a multi-modal large language model, which is an artificial intelligence model capable of processing multiple types of data inputs such as text, images, audio, etc. and generating natural language output, so it can be used to generate description texts corresponding to images.

[0153] In some embodiments, the plurality of third image description generation models are stored on the computer device, and the computer device can generate the description text by invoking the third image description generation model. Alternatively, the plurality of third image description generation models are stored on other devices, and different third image description generation models can be stored on the same device or different devices. The computer device sends a request for generating the description text to the device storing the third image description generation model, and the device storing the third image description generation model generates the description text corresponding to the second image based on the third image description generation model, and then sends the third description text to the computer device.

[0154] In some embodiments, each third image description generation model has one or more target dimensions for generating description texts belonging to the target dimensions. For example, the plurality of dimensions include at least two of the following: theme, background, style, composition, culture, and emotion, and other dimensions can also be included. The target dimensions of different third image description generation models can include the same dimensions or different dimensions, and the plurality of third image description generation models include third image description generation models with different target dimensions. Therefore, the plurality of third image description generation models correspond to N dimensions in total, and N is an integer greater than 1.

[0155] For example, for each third image description generation model, the training samples used in the training process include images and description texts belonging to the target dimensions, but do not include description texts belonging to other dimensions. This can effectively reduce the complexity of the description text and the amount of information of the training sample, thereby improving the training speed. The third image description generation model trained based on the above training sample has the ability to generate description texts belonging to the target dimensions based on images, thereby improving the pertinence and better describing the images from the target dimensions. However, the generated description text does not cover all dimensions.

[0156] The first large language model is an artificial intelligence model designed to understand and generate human language. The first large language model needs to be trained on a large amount of text to ensure that it can accurately perform tasks such as text summarization, translation, sentiment analysis, etc. through the first large language model. The first large language model can include GPT (Generative Pre-Trained Transformer), GLM (Generalized Linear Models), CPM (Chinese Pre-trained Models), and meta-mix large model, etc. The embodiments of the present application do not limit the first large language model.

[0157] In the embodiments of the present application, since the plurality of third image description generation models generate description texts of N dimensions in total, N being an integer greater than 1, the description texts generated by the plurality of third image description generation models are combined by the first large language model to mutually supplement and correct the plurality of description texts, thereby generating more accurate combined description texts, which can provide multi-dimensional training samples for the first image description generation model, thereby not only improving the performance of the first image description generation model, but also improving the comprehensiveness of the first image description generation model, so that the first image description generation model can generate multi-dimensional description texts for the input image and more comprehensively describe the image.

[0158] In some embodiments, even if the computer device cannot determine which dimension the target dimension of each third image description generation model is, the description texts generated by each third image description generation model can still be subjected to semantic analysis by the first large language model, so that when the plurality of description texts are combined, whether the dimensions to which the different description texts belong are the same can be considered, and then the description contents of the repeated dimensions in the plurality of description texts can be removed and the description contents of the non-repeated dimensions can be retained, so that the obtained combined description text can describe the second image from multiple dimensions and also does not have repeated contents.

[0159] For example, FIG. 5 is a flowchart of combining multi-dimensional description texts according to an embodiment of the present application. As shown in FIG. 5, the second image is processed by the plurality of third image description generation models to obtain description texts of multiple dimensions, and then the first large language model is used to combine the description texts to obtain a combined description text that can describe the second image from multiple dimensions, the multiple dimensions including theme, background, style, composition, culture, and emotion, etc.

[0160] 402. The computer device trains the image description generation model to be trained by taking the second image and the combined description text as a training sample pair.

[0161] Since the combined description text is obtained by comprehensively considering the generation results of the plurality of third image description generation models and can be considered as a more comprehensive and higher-quality description text, training the first image description generation model based on the second image and the combined description text can effectively improve the performance and comprehensiveness of the first image description generation model.

[0162] In some embodiments, before training the first image description generation model based on the second image and the combined description text, the method further includes: displaying the second image and the combined description text, and obtaining a modified combined description text based on a modification operation performed on the combined description text.

[0163] The computer device can display the second image and the combined description text on a local display screen for a user of the computer device to view whether the combined description text is accurate and comprehensive, and perform a correction operation on the combined description text, such as adding new text content to the combined description text, deleting text content from the combined description text, or modifying text content in the combined description text. After the correction, the computer device can obtain the corrected combined description text.

[0164] Alternatively, the computer device can also send the second image and the combined description text to the device of the correction personnel, so that the second image and the combined description text are displayed on the device of the correction personnel, and a correction operation is performed on the combined description text, such as adding new text content to the combined description text, deleting text content from the combined description text, or modifying text content in the combined description text. After the correction, the device can obtain the corrected combined description text and send it to the computer device.

[0165] For example, based on the combined description text shown in FIG. 5, the process of training the first image description generation model is shown in FIG. 6. Although performing a correction operation on the combined description text consumes a certain amount of human cost, since the combined description text has been generated, only the correction needs to be performed based on the combined description text, and the human cost is not too high, and the accuracy of the combined description text can be improved, thereby ensuring that the subsequent training of the first image description generation model based on the second image and the combined description text can effectively improve the performance of the first image description generation model.

[0166] It should be noted that the training process shown in the embodiments of the present application can be performed before the updating process in the embodiments shown in FIG. 2, that is, after the first image description generation model is trained using the embodiments of the present application, the second image description generation model is updated using the embodiments shown in FIG. 2. Alternatively, the training process shown in the embodiments of the present application can be performed alternately with the updating process in the embodiments shown in FIG. 2, that is, after the first image description generation model is trained using the embodiments of the present application, the second image description generation model is updated using the embodiments shown in FIG. 2, and then the first image description generation model can be trained again using the embodiments of the present application. The present application does not limit the time of training the first image description generation model and updating the second image description generation model.

[0167] 403. The computer device copies the trained description generation model to obtain the first image description generation model and the second image description generation model.

[0168] After the training is completed, the trained image description generation model can be generated as a first image model generation model, and then the first image description generation model is copied to obtain a second image description generation model, so that the initial state of the first image description generation model and the second image description generation model is the same. Subsequently, the second image description generation model is updated while the first image description generation model remains unchanged.

[0169] It should be noted that the embodiments of the present application take the copying of the first image description generation model to obtain the second image description generation model as an example. In another embodiment, the trained image description generation model can be taken as the second image description generation model, and then the second image description generation model is copied to obtain the first image description generation model. That is, after training an image description generation model, another image description generation model is obtained by copying. At this time, any one of the two image description generation models can be taken as the first image description generation model and kept unchanged, while the other one is taken as the second image description generation model and updated by using the updating method in the above embodiment.

[0170] In the related art, the image description generation model is usually trained based on the training sample. After training, the image description generation model can be used to generate a description text corresponding to any image. The training sample includes an image and a description text corresponding to the image. However, the image description generation model is limited by the dimension of the description text in the training sample, resulting in that the generated description text belongs to a single dimension and cannot cover multiple dimensions, and the image cannot be described comprehensively.

[0171] The embodiments of the present application provide a scheme for training an image description generation model, which can generate a plurality of dimension description texts corresponding to a second image by means of a plurality of third image description generation models, and then mutually supplement and correct the plurality of dimension description texts by using a first large language model, so as to obtain a combined description text of multiple dimensions. The combined description text can describe the second image from multiple dimensions, and is a more comprehensive and higher quality description text. For example, the combined description text can cover multiple dimensions such as theme, background, style, composition, culture and emotion. Therefore, training a first image description generation model based on the second image and the combined description text can effectively improve the performance and comprehensiveness of the first image description generation model, so that the first image description generation model can generate a description text of multiple dimensions for an input image, and more accurately and comprehensively describe the image. Moreover, manual determination of the description text is not required, which saves the labor cost and improves the training efficiency.

[0172] And, after generating the combined description text, only need to modify the combined description text to obtain a more accurate combined description text, the cost of human resources will not be too high, and the accuracy of the combined description text can be improved, thereby ensuring that the image description generation model can be trained based on the second image and the combined description text, and the performance of the image description generation model can be effectively improved.

[0173] And, since the image description generation model can generate multi-dimensional description texts for the input image, the image can be more accurately and comprehensively described, thereby providing more multi-dimensional description texts for the image generation model, and the performance of the image generation model can be effectively improved based on the description texts generated by the image description generation model.

[0174] After step 401, the method further includes a process of training a description modification model based on the combined description text, that is, the method further includes steps 404 and 405:

[0175] In step 404, the computer device obtains a quality score labeled for the combined description text.

[0176] In some embodiments, the quality score includes N-dimensional sub-scores, N is an integer greater than 1, each sub-score represents the quality of the combined description text in describing the second image in the dimension corresponding to the sub-score, and the description modification model is trained based on the combined description text and the quality score of the combined description text.

[0177] The quality score can be manually labeled by a labeler who determines the quality score of the combined description text by quality detection of the combined description text. Since the combined description text is used to describe the second image from N dimensions, the labeler can perform quality detection on the description text of each dimension respectively, thereby obtaining N-dimensional sub-scores. Then, the description modification model is trained based on the combined description text and the quality score of the combined description text, so that the description modification model has the ability to perform quality detection on the description text of each dimension in the description text of the image.

[0178] For example, referring to FIG. 7, taking the description modification model as a scoring model as an example, the combined description text is manually scored to obtain multiple dimension scores, and the scoring model is trained based on the combined description text and the multiple dimension scores.

[0179] The process of training the description modification model in step 404 can be performed in parallel with or after the process of training the image description generation model in step 402, and the application embodiments do not limit the training time of the description modification model and the first image description generation model.

[0180] In step 405, the computer device trains the description modification model based on the combined description text and the quality score of the combined description text.

[0181] The embodiment of the present application also provides a scheme for training the description correction model. The second image corresponding multi-dimensional description texts can be generated by means of the plurality of third image description generation models, and the multi-dimensional description texts are mutually supplemented and corrected by the first large language model, so as to obtain the combined description texts of multi-dimensions. The combined description texts can describe the second image from multiple dimensions and are more comprehensive and higher in quality. Therefore, the description correction model is trained based on the combined description texts, the quality scores labeled for the combined description texts, and the combined description texts and the corrected combined description texts, so that the performance and comprehensiveness of quality detection can be effectively improved. The description correction model can determine the sub-scores of multiple dimensions for the description texts of multiple dimensions, more accurately and comprehensively perform quality detection, and more accurately correct the description texts. Moreover, the artificial determination of the description texts is not needed, the artificial cost is saved, and the training efficiency is improved.

[0182] On the basis of the above-mentioned embodiment, the embodiment of the present application also provides another model processing method. FIG. 8 is a flowchart of another image description generation model processing method provided by the embodiment of the present application. The embodiment of the present application is executed by a computer device, which is a terminal 101 or a server 102 as shown in FIG. 1 or the like. In some embodiments, before step 401, the training of the image description generation model further includes at least one of a first training stage and a second training stage, wherein: the first training stage is used to train the visual model and the connector; and the second training stage is used to train the first large language model and the connector.

[0183] The embodiment of the present application describes the process of training the image description generation model. Referring to FIG. 8, the method includes:

[0184] 801. The computer device generates an image description generation model to be trained, wherein the image description generation model includes a visual model, a second large language model, and a connector located between the visual model and the second large language model.

[0185] An architecture diagram of the image description generation model is shown in FIG. 9. Among them, the visual model is a model for processing visual information, which can be in the form of pictures or videos, etc. The visual model can include a Transformer model (a neural network model based on self-attention mechanism), a ViT model, etc. The second large language model is an artificial intelligence model designed to understand and generate human language. The second large language model needs to be trained on a large amount of text to ensure that it can accurately perform tasks through the first large language model, including text summarization, translation, sentiment analysis, etc. The second large language model can include GPT, GLM, CPM, meta large model, etc., and the embodiments of the present application do not limit the second large language model.

[0186] The connector connects the visual model and the second large language model, and is used to convert the first feature space to which the features output by the visual model belong into the second feature space to which the features input by the second large language model belong, so that the second large language model can recognize the features output by the visual model, i.e., aligning the visual model with the second large language model.

[0187] It should be noted that in some embodiments, when the computer device newly generates the image description generation model, the model parameters in the visual model, the connector and the second large language model can be initialized, and then steps 802 and 803 described below are executed to train the image description generation model. In another possible implementation, the visual model and the second large language model obtained by the computer device have been trained, i.e., the model parameters of the visual model and the second large language model have been trained, and then the computer device combines the visual model and the second large language model and the connector to obtain the image description generation model. Among them, the model parameters of any unit (including the visual model, the connector and the second large language model) in the image description generation model include the weights or bias terms in the unit, or other types of model parameters, etc.

[0188] 802. The computer device trains the visual model and the connector based on the first training sample while keeping the model parameters of the second large language model unchanged, and the first training sample includes a third image and a third description text corresponding to the third image.

[0189] Among them, keeping the model parameters of the second large language model unchanged includes freezing the model parameters of the second large language model. After step 802 is executed, the model parameters of the second large language model are unfrozen.

[0190] In the embodiments of the present application, in order to distinguish, the image used by the training image description generation model in this stage is referred to as a third image. The third image can include different images from the first image in the above embodiments, and can include the same images or different images from the second image in the above embodiments. The third image can be sent to the computer device by other devices, stored in the computer device by a technician, or obtained in other ways. The description text corresponding to the third image is referred to as third description text. The third description text can be determined by a human or by a third description generation model, and the embodiments of the present application do not limit this.

[0191] In the case of keeping the model parameters of the second large language model unchanged, the visual model and the connector are trained, so as to align the first feature space to which the features output by the visual model belong and the second feature space to which the features of the second large language model belong, so that the connector can convert the features belonging to the first feature space output by the visual model into features belonging to the second feature space and input into the second large language model. In this way, the second large language model can recognize the features belonging to the second feature space, so as to process the features belonging to the second feature space.

[0192] 803. The computer device trains the second large language model and the connector based on the second training sample in the case of keeping the model parameters of the visual model unchanged, the second training sample comprising a fourth image and a dialogue text for the fourth image.

[0193] The keeping of the model parameters of the visual model unchanged comprises freezing the model parameters of the visual model. After the step 803 is executed, the model parameters of the visual model are unfreezed.

[0194] In the embodiments of the present application, in order to distinguish, the image used by the training image description generation model in this stage is referred to as a fourth image, which can include different images from the first image in the above embodiments, and can include the same images or different images from the second image in the above embodiments and the third image in step 802. The fourth image can be sent to the computer device by other devices, stored in the computer device by technicians, or obtained in other ways. The fourth image corresponds to the dialogue text used for dialogue for the fourth image. In some embodiments, the dialogue text includes at least two dialogue sentences, each of which is related to the fourth image. The dialogue text also belongs to the description text for describing the fourth image. Compared with ordinary description text, at least two dialogue sentences in the description text of the fourth image have a context relationship and contain more information. For example, the dialogue text for the fourth image includes two dialogue sentences, the first dialogue sentence is a question "What is in the image?" for the fourth image, and the second dialogue sentence is an answer "There are two animals in the image" to the above question. The dialogue text for the fourth image can be determined by artificial or obtained by crawling from the Internet, for example, the comments on the fourth image and the replies to the comments constitute the dialogue text for the fourth image, which is not limited in the embodiments of the present application.

[0195] In the case of keeping the model parameters of the visual model unchanged, training the second large language model and the connector can enhance the understanding ability of the second large language model, so that the second large language model can accurately understand the features output by the connector, and thus accurately process the image description text. It should be noted that step 802 and step 803 can be regarded as two training stages of the image description generation model, and the two training stages can be executed in sequence, and the embodiments of the present application take the execution of step 802 before step 803 as an example, and step 803 can also be executed before step 802 in another embodiment. Moreover, step 802 and step 803 are optional solutions, that is, in the embodiments of the present application, only step 802 can be executed without step 803, or only step 803 can be executed without step 802, and the training process of the image description generation model is not limited in the embodiments of the present application.

[0196] The embodiment of the present application provides another scheme for training the image description generation model, which can be trained in stages. First, the visual model and the connector are trained based on the third image and the third description text corresponding to the third image while keeping the model parameters of the second large language model unchanged, the visual model is aligned with the second large language model, and then the second large language model and the connector are trained based on the fourth image and the dialogue text for the fourth image while keeping the model parameters of the visual model unchanged, so that the understanding ability of the second large language model can be enhanced, thereby performing targeted training at each stage, improving the performance of the image description generation model, and improving the training effect.

[0197] In some embodiments, the description correction model has the same architecture as the image description generation model, and the description correction model can also be trained by using the embodiment shown in FIG. 8. The difference is that the training sample of the image description generation model includes an image and a description text or dialogue text corresponding to the image, and the training sample of the description correction model includes a description text corresponding to the image, a quality score of the description text and a corrected description text. The training process of the description correction model is not limited in the embodiment of the present application.

[0198] In addition, the training method provided by the embodiment of the present application can be combined with the embodiments shown in FIGS. 2 and 4, so as to train the image description generation model in multiple ways, thereby more comprehensively improving the performance of the image description generation model.

[0199] For example, as shown in FIG. 10, step 802 is taken as a first training stage, step 803 is taken as a second training stage, steps 401-402 and steps 404-405 are taken as a third training stage, and steps 201-204 are taken as an updating stage after the third training stage. For example, after the first training stage, the second training stage and the third training stage are performed, a trained image description generation model is obtained as the first image description generation model, a second image description generation model and a trained description correction model are copied, and then the updating stage is performed based on the first image, the first image description generation model, the second image description generation model and the description correction model.

[0200] Or, as shown in FIG. 11, step 802 is taken as a first training stage, step 803 is taken as a second training stage, steps 401-402 and steps 404-405 are taken as a third training stage, and steps 201-204 are also taken as part of the third training stage, that is, the third training stage includes steps 401-405 and steps 201-204. After the first training stage and the second training stage are performed, a trained image description generation model is obtained, and then the third training stage is performed, the image description generation model and the description correction model are trained based on the multi-dimensional description text corresponding to the second image, the trained image description generation model is taken as the first image description generation model, the second image description generation model and the description correction model are copied, and then the second image description generation model is updated based on the first image, the first image description generation model, the second image description generation model and the description correction model.

[0201] The embodiment of the present application integrates the multi-dimensional description texts generated by the plurality of third description generation models through the first large language model to obtain a multi-dimensional combined description text, and obtains a multi-dimensional quality score of the combined description text through artificial labeling. The multi-dimensional description correction model is trained using the labeled quality score, and the image description generation model is trained using the multi-dimensional combined description text. The result of the image description generation model is detected and filtered by the description correction model, thereby updating the performance of the image description generation model, and realizing the generation of multi-dimensional description texts and the self-learning of the image description generation model.

[0202] Experiments show that the self-learning method provided by the embodiment of the present application has excellent performance, and the correlation between the description text generated by the image description generation model in the embodiment of the present application and the image can reach 78%, which is much higher than that of the description generation model in the related art.

[0203] FIG. 12 is a structural schematic diagram of a model processing device provided by an embodiment of the present application. Referring to FIG. 12, the device includes:

[0204] The first description generation module 1201 is configured to generate a first description text corresponding to a first image through a first image description generation model, and correct the first description text through a description correction model to obtain a first corrected description text.

[0205] The second description generation module 1202 is configured to generate a second description text corresponding to the first image through a second image description generation model, obtain a quality score of the second description text through the description correction model, and correct the second description text through the description correction model to obtain a second corrected description text.

[0206] The model updating module 1203 is configured to, in a case where the quality score of the second description text does not satisfy the quality condition, determine a loss value based on the first description text, the first modified description text, the second description text, and the second modified description text, and update the second image description generation model by taking the loss value as a target, wherein the updated second image description generation model is used to generate a text describing an input image according to the input image, and the second image description generation model before the update has the same structure and state as the first image description generation model.

[0207] In some embodiments, referring to FIG. 13, the model updating module 1203 includes:

[0208] The first updating unit 1213 is configured to determine a first probability that the first image description generation model outputs the first description text, a second probability that the description modification model outputs the first modified description text, a third probability that the second image description generation model outputs the second description text, and a fourth probability that the description modification model outputs the second modified description text, and determine a loss value according to the first probability, the second probability, the third probability, and the fourth probability, wherein the loss value is positively correlated with a logarithm of a ratio of the third probability to the first probability, and is negatively correlated with a logarithm of a ratio of the fourth probability to the second probability, and update the second image description generation model by taking a decrease of the loss value as a target.

[0209] In some embodiments, referring to FIG. 13, the model updating module 1203 includes:

[0210] The second updating unit 1223 is configured to, in a case where the quality score of the second description text satisfies the quality condition, update the second image description generation model based on the first image and the second description text.

[0211] It should be noted that the model processing apparatus provided in the above embodiments is only exemplified by the division of the above functional modules, and in actual applications, the above functions can be completed by different functional modules according to needs, that is, the internal structure of the computer device is divided into different functional modules to complete all or part of the functions described above. In addition, the model processing apparatus and the model processing method provided in the above embodiments belong to the same concept, and the specific implementation process is detailed in the method embodiments, which will not be repeated here.

[0212] The embodiments of the present application also provide a computer device, which includes a processor and a memory, and the memory stores at least one computer program, which is loaded and executed by the processor to implement the operations performed in the model processing method of the above embodiments.

[0213] In some embodiments, the computer device is provided as a terminal. FIG. 14 shows a structural schematic diagram of a terminal 1400 provided by an example embodiment of the present application.

[0214] The terminal 1400 includes a processor 1401 and a memory 1402.

[0215] The processor 1401 can include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 1401 can be implemented in at least one of a hardware form of a DSP (Digital Signal Processing), an FPGA (Field Programmable Gate Array), a PLA (Programmable Logic Array). The processor 1401 can also include a main processor and a coprocessor, the main processor being a processor for processing data in an awake state, also known as a CPU (Central Processing Unit), and the coprocessor being a low-power processor for processing data in a standby state. In some embodiments, the processor 1401 can be integrated with a GPU (Graphics Processing Unit) for rendering and drawing content required to be displayed by a display screen. In some embodiments, the processor 1401 can further include an AI (Artificial Intelligence) processor for processing machine learning-related computing operations.

[0216] The memory 1402 can include one or more computer-readable storage media, which can be non-transitory. The memory 1402 can also include a high-speed random access memory, and a non-volatile memory such as one or more disk storage devices, flash storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 1402 is used to store at least one computer program for being executed by the processor 1401 to implement the model processing method provided by the method embodiments in the present application.

[0217] In some embodiments, the terminal 1400 can further include a peripheral device interface 1403 and at least one peripheral device. The processor 1401, the memory 1402, and the peripheral device interface 1403 can be connected through a bus or a signal line. Each peripheral device can be connected to the peripheral device interface 1403 through a bus, a signal line, or a circuit board. In some embodiments, the peripheral device includes at least one of a radio frequency circuit 1404, a display screen 1405, a camera assembly 1406, and a power supply 1407.

[0218] The peripheral interface 1403 can be used to connect at least one peripheral device related to I / O (Input / Output) to the processor 1401 and the memory 1402. In some embodiments, the processor 1401, the memory 1402 and the peripheral interface 1403 are integrated on the same chip or circuit board; in some other embodiments, any one or two of the processor 1401, the memory 1402 and the peripheral interface 1403 can be implemented on a separate chip or circuit board, and the present embodiments are not limited in this regard.

[0219] Those skilled in the art can understand that the structure shown in FIG. 14 does not constitute a limitation on the terminal 1400, and can include more or fewer components than those shown, or combine certain components, or use different component arrangements.

[0220] In some embodiments, the computer device is provided as a server. FIG. 15 is a structural schematic diagram of a server provided by an embodiment of the present application. The server 1500 can differ greatly in configuration or performance and can include one or more processors (CPU) 1501 and one or more memories 1502, wherein the memory 1502 stores at least one computer program, which is loaded and executed by the processor 1501 to implement the method provided by each of the above-mentioned method embodiments. Of course, the server can also have a wired or wireless network interface, a keyboard, an input / output interface and other components for implementing device functions, and the like, so as to perform input and output. The server can also include other components for implementing device functions, which are not described here.

[0221] The embodiments of the present application also provide a computer readable storage medium, which stores at least one computer program, and the at least one computer program is loaded and executed by a processor to implement the operations performed by the model processing method of the above-mentioned embodiments.

[0222] The embodiments of the present application also provide a computer program product, which includes a computer program, and the computer program is loaded and executed by a processor to implement the operations performed by the model processing method of the above-mentioned embodiments.

[0223] Those of ordinary skill in the art can understand that all or part of the above-mentioned embodiments can be completed by hardware, or by a program instructing relevant hardware, and the program can be stored in a computer readable storage medium, and the above-mentioned storage medium can be a read-only memory, a magnetic disk or an optical disk, etc.

[0224] The above merely describes optional embodiments of the present application, and is not intended to limit the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. An image description generation model processing method, executed by an electronic device, comprising: generating, by a first image description generation model, a first description text corresponding to a first image, and correcting the first description text by a description correction model to obtain a first corrected description text; generating, by a second image description generation model, a second description text corresponding to the first image, and obtaining, by the description correction model, a quality score of the second description text, and correcting the second description text by the description correction model to obtain a second corrected description text; in a case where the quality score of the second description text does not satisfy a quality condition, determining a loss value based on the first description text, the first corrected description text, the second description text and the second corrected description text, updating the second image description generation model to obtain an updated second image description generation model, so as to generate a text describing an input image according to the input image, wherein the second image description generation model before the update has the same structure and state as the first image description generation model.

2. The method of claim 1, wherein, The determining of the loss value based on the first description text, the first corrected description text, the second description text and the second corrected description text comprises: determining a first probability that the first image description generation model outputs the first description text, a second probability that the description correction model outputs the first corrected description text, a third probability that the second image description generation model outputs the second description text, and a fourth probability that the description correction model outputs the second corrected description text; determining the loss value according to the first probability, the second probability, the third probability and the fourth probability, wherein the loss value is positively correlated with the logarithm of the ratio of the third probability to the first probability, and is negatively correlated with the logarithm of the ratio of the fourth probability to the second probability.

3. The method of claim 1 or 2, wherein, The first image description generation model and the second image description generation model are obtained by training a to-be-trained image description generation model, and the method further comprises: training the image description generation model to obtain the first image description generation model and the second image description generation model, wherein the to-be-trained image description generation model comprises a visual model, a first large language model and a connector located between the visual model and the first large language model; the training comprises at least one of a first training phase and a second training phase, wherein: the first training phase is used for training the visual model and the connector; and the second training phase is used for training the first large language model and the connector.

4. The method of claim 3, wherein, The training further comprises a third training phase; the third training phase comprises: generating a combined description text corresponding to a second image, wherein the combined description text is used for describing the second image from multiple dimensions; training the image description generation model by taking the second image and the combined description text as a training sample pair.

5. The method of claim 4, wherein, The generating of the combined description text corresponding to the second image comprises: generate, by a plurality of third image description generation models, a plurality of dimensions of description texts corresponding to the second image respectively, wherein each dimension of description text is used to describe the second image from the dimension, and each third image description generation model is used to generate a dimension of description text corresponding to the second image; combine, by a second large language model, the plurality of dimensions of description texts to obtain a combined description text.

6. The method of claim 4 or 5, wherein, The combined description text is a revised combined description text.

7. The method according to any one of claims 4 to 6, wherein, The description revision model is a trained description revision model, and the method further comprises: obtaining a quality score labeled for the combined description text; training the description revision model based on the combined description text and the quality score of the combined description text.

8. The method of claim 7, wherein, The quality score includes a sub-score of each dimension of the plurality of dimensions, and each sub-score represents the quality of the combined description text in describing the second image from the dimension corresponding to the sub-score.

9. The method according to any one of claims 4 to 8, wherein, The plurality of dimensions includes at least two of the following: theme, background, style, composition, culture, and emotion.

10. The method of any one of claims 1 to 9, further comprising: updating the second image description generation model based on the first image and the second description text in a case where the quality score of the second description text satisfies the quality condition.

11. The method according to any one of claims 1 to 10, wherein, The quality condition includes a quality score threshold, and the method further comprises: determining that the second description text does not satisfy the quality condition in a case where the quality score of the second description text is less than or equal to the quality score threshold.

12. The method according to any one of claims 1 to 10, wherein, The quality condition includes a quality ranking threshold, and the method further comprises: determining that the second description text does not satisfy the quality condition when the position of the quality score of the second description text in a ranking of a plurality of second description texts from high to low is after the quality ranking threshold, wherein the plurality of second description texts are second description texts corresponding to a plurality of first images respectively and generated by the second image description model.

13. The method of claim 11 or 12, wherein, The quality score of the second description text includes N dimensions of sub-scores, N being an integer greater than 1, and each sub-score represents the quality of the second description text in describing the first image from the dimension corresponding to the sub-score.

14. The method of any one of claims 1 to 13, further comprising: generating a sample description text corresponding to a sample image by the updated second image description generation model; training an image generation model by taking the sample image and the sample description text as training samples.

15. An image description generation model processing apparatus, comprising: a first description generation module configured to generate a first description text corresponding to a first image by a first image description generation model, and to revise the first description text by a description revision model to obtain a first revised description text; a second description generation module configured to generate a second description text corresponding to the first image by a second image description generation model, to obtain a quality score of the second description text by the description revision model, and to revise the second description text by the description revision model to obtain a second revised description text. The model updating module is configured to, in a case where the quality score of the second description text does not satisfy a quality condition, determine a loss value based on the first description text, the first revised description text, the second description text, and the second revised description text, update the second image description generation model to reduce the loss value, and the updated second image description generation model is configured to generate a text describing an input image according to the input image, and the second image description generation model before the update has the same structure and state as the first image description generation model.

16. A computer device, comprising a processor and a memory, and the memory stores at least one computer program, and the at least one computer program is loaded and executed by the processor to implement the operations performed by the model processing method according to any one of claims 1 to 14.

17. A computer-readable storage medium, and the computer-readable storage medium stores at least one computer program, and the at least one computer program is loaded and executed by a processor to implement the operations performed by the model processing method according to any one of claims 1 to 14.

18. A computer program product, comprising a computer program, and the computer program is loaded and executed by a processor to implement the operations performed by the model processing method according to any one of claims 1 to 14.

Images