Method and apparatus for generating description text

Abstract

The application provides a method and device for generating description text, the method comprising: obtaining an image to be processed; obtaining world knowledge information of the image to be processed; inputting the image to be processed and the world knowledge information of the image to be processed into a first description generation model to generate description text of the image to be processed. The scheme of the embodiment of the application is beneficial to improving the quality of the generated description text.

Description

[0001] This application claims priority to Chinese Patent Application No. 202510570733.7, filed with the China National Intellectual Property Administration on April 29, 2025, entitled "Method and Apparatus for Generating Descriptive Text", the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of artificial intelligence, and more specifically, to a method and apparatus for generating descriptive text. Background Technology

[0003] Caption data, such as descriptive text for images or videos, is crucial in multimodal tasks and can improve the performance of visual models. High-quality caption data helps models better understand visual content, thereby enhancing user experience and task performance.

[0004] Current methods for obtaining caption data suffer from several problems, making it difficult to meet practical application needs. For example, some methods rely on manual annotation to obtain caption data. This method is challenging, costly, and inefficient. Another approach involves generating caption data based on open-source multimodal large models. This method has high inference costs, cannot be mass-produced, and suffers from inconsistent generation quality, potentially containing noise. Summary of the Invention

[0005] This application provides a method and apparatus for generating descriptive text, which helps to improve the quality of the generated descriptive text.

[0006] In a first aspect, a method for generating descriptive text is provided, comprising: acquiring an image to be processed; acquiring world knowledge information of the image to be processed; and inputting the image to be processed and the world knowledge information of the image to be processed into a first description generation model to generate descriptive text of the image to be processed.

[0007] According to the solution of this application embodiment, the world knowledge of the image is introduced in the process of generating descriptive text by the model, thereby avoiding relying solely on the model's own world knowledge to describe the image. This is equivalent to expanding the model's world knowledge, which is beneficial to improving the model's content description ability and thus improving the quality of the descriptive text.

[0008] Furthermore, compared to acquiring visual aids, acquiring world knowledge from images requires relatively less time and computational resources. The solution adopted in this application is beneficial to improve the quality of descriptive text while increasing generation efficiency and avoiding excessive resource consumption.

[0009] The image to be processed can be a single image or an image frame from a video.

[0010] Optionally, the world knowledge information of the image to be processed is obtained by retrieving the image to be processed.

[0011] For example, the image to be processed is input into a search tool for retrieval to obtain relevant knowledge information about the image. For instance, the relevant knowledge information could be obtained by searching for text related to the image using an image-to-text search method.

[0012] In conjunction with the first aspect, in some implementations of the first aspect, the method further includes: acquiring visual auxiliary information of the image to be processed, the visual auxiliary information of the image to be processed being obtained by visual processing of the image to be processed; and inputting the image to be processed and world knowledge information of the image to be processed into a first description generation model to generate descriptive text of the image to be processed, including: inputting the image to be processed, world knowledge information of the image to be processed and visual auxiliary information of the image to be processed into the first description generation model to generate descriptive text of the image to be processed.

[0013] According to the scheme of this application embodiment, visual auxiliary information of an image can be obtained through visual processing. Visual auxiliary information and world knowledge information are simultaneously introduced during the model's generation of descriptive text, further expanding the model's world knowledge and improving its content description capabilities, thereby enhancing the quality of the descriptive text. Compared to schemes that generate descriptive text and then fuse it with other information to optimize it, in the scheme of this application embodiment, the image to be processed can serve as visual guidance. Based on this visual guidance, the fusion of multimodal information (such as the image to be processed, world knowledge information, and visual auxiliary information) is achieved. Visual guidance helps avoid descriptive errors and illusions, thus further improving the quality of the descriptive text.

[0014] In conjunction with the first aspect, in some implementations of the first aspect, the visual auxiliary information of the image to be processed includes at least one of the following: the label of the image to be processed, the location information of the target in the image to be processed, or the OCR information of the image to be processed.

[0015] For example, the label of the image to be processed can be used to indicate one or more of the following: target, scene, attribute, and behavior.

[0016] For example, the location information of a target in the image to be processed can be indicated by the detection bounding box of the target in the image to be processed.

[0017] For example, the OCR information of the image to be processed may include the text content in the image. Furthermore, the OCR information of the image to be processed may also include metadata about the text content in the image.

[0018] In conjunction with the first aspect, in some implementations of the first aspect, the visual auxiliary information of the image to be processed also includes information on the number of targets in the image to be processed.

[0019] For example, the number of targets in the image to be processed can be indicated by counting the number of detection boxes corresponding to each target in the image to be processed.

[0020] In conjunction with the first aspect, in some implementations of the first aspect, the first description generation model is obtained by training the second description generation model based on the first dataset. The training data in the first dataset includes the first image sample, the preferred sample of the first image sample, and the non-preferred sample of the first image sample. During the training process, the parameters of the second description generation model are adjusted so that the probability of the second description generation model generating the preferred sample of the first image sample is higher than the probability of the second description generation model generating the non-preferred sample of the first image sample. The preferred sample and the non-preferred sample of the first image sample are two descriptive texts of the first image sample.

[0021] According to the scheme of the embodiments of this application, the model can be trained based on the preference data of the descriptive text. This makes the model more inclined to generate preferred samples rather than non-preferred samples, which is conducive to further optimizing the model and improving the quality of the generated descriptive text.

[0022] In conjunction with the first aspect, in some implementations of the first aspect, the non-preference sample of the first image sample is the content in the predicted description text of the first image sample that is inconsistent with the target description text of the first image sample, the preference sample of the first image sample is the content in the target description text of the first image sample that is inconsistent with the predicted description text of the first image sample, and the predicted description text of the first image sample is generated by the second description generation model.

[0023] According to the scheme of this application embodiment, inconsistent content can be determined from two complete descriptive texts of an image, and used as non-preference samples and preference samples, respectively. The preference data pairs obtained in this way are of higher quality, which helps the model to specifically strengthen the model against erroneous text, eliminates the interference of correct text on optimization, and thus helps to further improve the generation quality of the model.

[0024] In conjunction with the first aspect, in some implementations of the first aspect, the target description text of the first image sample is obtained by correcting the predicted description text of the first image sample.

[0025] In conjunction with the first aspect, in some implementations of the first aspect, the first description generation model is obtained by training the third description generation model on the second dataset. The second image sample in the second dataset is a hard example, and the second image sample being a hard example is determined based on the predicted description text of the second image sample. The predicted description text of the second image sample is generated by the third description generation model.

[0026] According to the solution of the embodiments of this application, difficult examples can be mined and the model can be optimized using difficult examples, which is beneficial to improving the generation quality of the model.

[0027] In conjunction with the first aspect, in some implementations of the first aspect, the determination that a second image sample is a hard example is based on the predicted description text of the second image sample and a quality control strategy, which is used to determine whether an image sample is a hard example.

[0028] Optionally, the quality control strategy may include a quality control model.

[0029] For example, the quality control model can be a multimodal large model. For instance, the input to the quality control model can include an image sample and its descriptive text. The output of the quality control model can include a score for the descriptive text of the image sample, which can be used to indicate the generation quality. The comparison between the descriptive text score of the image sample and a threshold #1 can be used to determine whether the image sample belongs to the difficult examples category.

[0030] Optionally, a quality control strategy may include one or more quality control rules.

[0031] For example, the one or more quality control rules can be used to determine whether an image sample is a difficult example based on errors in the descriptive text of the image sample.

[0032] In conjunction with the first aspect, in some implementations of the first aspect, the third image sample in the second dataset is retrieved based on the category of the second image sample and / or the predicted descriptive text of the second image sample.

[0033] According to the scheme of the embodiments of this application, relevant image samples can be retrieved based on the category and / or descriptive text of the hard examples that have been mined, and used as new hard examples to expand the training dataset.

[0034] Secondly, a training method for a descriptive generative model is provided. The method includes: acquiring a third dataset, wherein the training data in the third dataset includes a fourth image sample, a preferred sample of the fourth image sample, and a non-preferred sample of the fourth image sample, wherein the preferred sample and the non-preferred sample of the fourth image sample are two descriptive texts of the fourth image sample; and training the fourth descriptive generative model by adjusting the training data, wherein during the training process, the parameters of the fourth descriptive generative model are adjusted so that the probability of the fourth descriptive generative model generating a preferred sample is higher than the probability of the fourth descriptive generative model generating a non-preferred sample.

[0035] According to the solution of this application embodiment, the description generation model can be trained based on the preference data of the description text. This is beneficial to make the model more inclined to generate preferred samples rather than non-preferred samples, thereby facilitating further optimization of the model and improving the quality of the generated description text.

[0036] In conjunction with the second aspect, in some implementations of the second aspect, the method further includes: inputting the fourth image sample into the fourth description generation model to generate a predicted description text for the fourth image sample, wherein the non-preference sample and the preference sample of the fourth image sample are inconsistent contents in the predicted description text and the target description text of the fourth image sample, respectively.

[0037] In conjunction with the second aspect, in some implementations of the second aspect, the target description text of the fourth image sample is obtained by correcting the predicted description text of the fourth image sample.

[0038] In conjunction with the second aspect, in some implementations of the second aspect, the method further includes: acquiring world knowledge information and / or visual auxiliary information of the fourth image sample, wherein the visual auxiliary information of the fourth image sample is obtained by visual processing of the fourth image sample; and inputting the fourth image sample into the fourth description generation model to generate a predicted description text of the fourth image sample, including: inputting the fourth image sample and the auxiliary information of the fourth image sample into the fourth description generation model to generate a predicted description text of the fourth image sample, wherein the auxiliary information of the fourth image sample includes world knowledge information and / or visual auxiliary information of the fourth image sample.

[0039] In conjunction with the second aspect, in some implementations of the second aspect, the visual auxiliary information of the fourth image sample includes at least one of the following: the label of the fourth image sample, the location information of the target in the fourth image sample, or the OCR information of the fourth image sample.

[0040] In conjunction with the second aspect, in some implementations of the second aspect, the visual auxiliary information of the image to be processed also includes information on the number of targets in the image to be processed.

[0041] In conjunction with the second aspect, in some implementations of the second aspect, the fourth description generation model is obtained by training the fifth description generation model on the fourth dataset. The fifth image sample in the fourth dataset is a hard example, and the fact that the fifth image sample is a hard example is determined based on the predicted description text of the fifth image sample. The predicted description text of the fifth image sample is generated by the fifth description generation model.

[0042] In conjunction with the second aspect, in some implementations of the second aspect, the determination that the fifth image sample is a hard example is based on the predicted description text of the fifth image sample and the quality inspection strategy, which is used to determine whether an image is a hard example.

[0043] In conjunction with the second aspect, in some implementations of the second aspect, the sixth image sample in the fourth dataset is retrieved based on the category of the fifth image sample and / or the predicted descriptive text of the fifth image sample.

[0044] Thirdly, a training apparatus for describing a generative model is provided, the apparatus comprising modules / units for performing the methods of the first aspect and any implementation thereof.

[0045] Fourthly, an apparatus for generating descriptive text is provided, the apparatus comprising modules / units for performing the methods of the second aspect and any implementation thereof.

[0046] It should be understood that the extensions, limitations, interpretations and descriptions of the relevant content in the first aspect above also apply to the same content in the second, third and fourth aspects.

[0047] Fifthly, a computing device is provided, including a processor and a memory, and optionally, an input / output interface. The processor controls the input / output interface to send and receive information, the memory stores a computer program, and the processor retrieves and runs the computer program from the memory, enabling the execution of the method described in the first aspect, the second aspect, or any possible implementation thereof.

[0048] Optionally, the processor can be a general-purpose processor, which can be implemented in hardware or software. When implemented in hardware, the processor can be a logic circuit, integrated circuit, etc.; when implemented in software, the processor can be a general-purpose processor that reads software code stored in memory. This memory can be integrated into the processor or located outside the processor and exist independently.

[0049] Optionally, the aforementioned computing device may be a terminal device / server, or a chip within a terminal device / server.

[0050] Sixthly, a chip is provided that acquires and executes instructions to implement the methods in the first or second aspect and any of the implementation methods described above.

[0051] Optionally, as one implementation, the chip includes a processor and a data interface, through which the processor reads instructions stored in the memory and executes the methods in the first aspect or the second aspect and any of the implementations described above.

[0052] Optionally, as one implementation, the chip may further include a memory storing instructions, and the processor is used to execute the instructions stored in the memory. When the instructions are executed, the processor is used to perform the method in the first aspect or the second aspect and any of the implementations.

[0053] In a seventh aspect, a computer program product containing instructions is provided, which, when executed by a computing device, cause the computing device to perform the methods described in the first or second aspect and any of their implementations.

[0054] Eighthly, a computer-readable storage medium is provided, including computer program instructions that, when executed by a computing device, perform the method as described in the first or second aspect and any of its implementations.

[0055] As examples, these computer-readable storage devices include, but are not limited to, one or more of the following: read-only memory (ROM), programmable ROM (PROM), erasable PROM (EPROM), flash memory, electrically EPROM (EEPROM), and hard drive.

[0056] Alternatively, as one implementation method, the aforementioned storage medium can specifically be a non-volatile storage medium. Attached Figure Description

[0057] Figure 1 This is an architecture diagram of a system architecture according to an embodiment of this application.

[0058] Figure 2 This is a schematic flowchart illustrating a method for generating descriptive text according to an embodiment of this application.

[0059] Figure 3 This is a schematic flowchart illustrating a training method for a generative model according to an embodiment of this application.

[0060] Figure 4 This is a schematic diagram illustrating the training process of a generative model according to an embodiment of this application.

[0061] Figure 5 This is a schematic diagram of an example image from an embodiment of this application.

[0062] Figure 6 This is a schematic flowchart illustrating another method for training a generative model provided in an embodiment of this application.

[0063] Figure 7 This is a schematic diagram illustrating an example of a DPO training process according to an embodiment of this application.

[0064] Figure 8 This is a schematic diagram illustrating an example of a text correction process according to an embodiment of this application.

[0065] Figure 9 This is a schematic diagram of a fine-grained splitting process according to an embodiment of this application.

[0066] Figure 10 This is a schematic diagram of another system architecture according to an embodiment of this application.

[0067] Figure 11 This is a schematic block diagram of an apparatus according to an embodiment of this application.

[0068] Figure 12 This is a schematic diagram of the architecture of a computing device according to an embodiment of this application. Detailed Implementation

[0069] The technical solutions in this application will now be described with reference to the accompanying drawings.

[0070] The terminology used in the following embodiments is for the purpose of describing specific embodiments only and is not intended to be limiting of this application. As used in the specification and appended claims of this application, the singular expressions “a,” “an,” and “the” are intended to include expressions such as “one or more,” unless the context clearly indicates otherwise. It should also be understood that in the following embodiments of this application, “at least one,” “at least one,” and “one or more” refer to one, two, or more than two. “First,” “second,” and various numerical designations are merely distinctions for descriptive convenience and are not intended to limit the scope of the embodiments of this application. “And / or” is used to describe the correspondence between corresponding objects, indicating that three relationships can exist. For example, “A and / or B” can represent: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character “ / ” generally indicates that the preceding and following related objects are in an “or” relationship. The order of the sequence numbers of the processes below does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application. For example, in the embodiments of this application, the words "301", "401", "501" etc. are merely identifiers made for the convenience of description and do not limit the order of execution steps.

[0071] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. In this application, the words "exemplary" or "for example" are used to indicate that something is illustrative, exemplary, or descriptive. Any embodiment or design described as "exemplary" or "for example" in this application should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of the words "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner. The terms "comprising," "including," "having," and variations thereof all mean "including but not limited to," unless otherwise specifically emphasized. In the embodiments of this application, descriptions such as "when," "in the case of," "if," and "if" all refer to the fact that the device will perform a corresponding processing under certain objective circumstances, and are not a limitation on time, nor do they require the device to perform a judgment action during implementation, nor do they imply any other limitations.

[0072] In this application, "for indicating" can include both direct and indirect indication. When describing an indication message as indicating A, it can include whether the indication message directly indicates A or indirectly indicates A, but does not necessarily mean that the indication message carries A.

[0073] The method provided in this application can be applied to fields involving descriptive text generation, such as descriptive text generation, video subtitle generation, and visual question answering.

[0074] To facilitate understanding of the solutions in the embodiments of this application, the terms that may be involved in the embodiments of this application will be explained below.

[0075] (1) Neural Networks: Neural networks can be composed of neural units, which can refer to units represented by x. s The operation unit takes an intercept of 1 as input.

[0076] A neural network is a network formed by connecting multiple individual neural units, meaning that the output of one neural unit can be the input of another. The input of each neural unit can be connected to the local receptive field of the previous layer to extract features from the local receptive field, which can be a region composed of several neural units.

[0077] (2) Deep Neural Networks: A deep neural network (DNN), also known as a multilayer neural network, can be understood as a neural network with multiple hidden layers. Based on the position of the layers, the internal neural network of a DNN can be divided into three categories: input layer, hidden layer, and output layer. Generally, the first layer is the input layer, the last layer is the output layer, and the layers in between are hidden layers. The layers are fully connected, meaning that any neuron in the i-th layer is connected to any neuron in the (i+1)-th layer.

[0078] In deep neural networks, more hidden layers allow the network to better depict complex situations in the real world. Theoretically, the more parameters a model has, the higher its complexity and "capacity," meaning it can accomplish more complex learning tasks. Training a deep neural network is essentially the process of learning the weight matrix, and its ultimate goal is to obtain the weight matrix of all layers in the trained deep neural network (a weight matrix formed by vectors from many layers).

[0079] (3) Loss function: In training a deep neural network, to ensure the output closely approximates the desired predicted value, we compare the network's prediction with the target value. Based on the difference, we update the weight vector of each layer (usually pre-configuring parameters before the initial update). For example, if the prediction is too high, the weight vector is adjusted to predict a lower value. This adjustment continues until the deep neural network predicts the target value or a value very close to it. Therefore, we need to predefine "how to compare the difference between the predicted and target values," which is the loss function or objective function. These are important equations used to measure the difference between the predicted and target values. Taking the loss function as an example, a higher output value (loss) indicates a greater difference, and training the deep neural network becomes a process of minimizing this loss.

[0080] (4) Backpropagation algorithm: Backpropagation (BP) is an algorithm used during training to correct the parameters in the initial model, thereby reducing the model's error loss. Specifically, forward propagation of the input signal to the output generates an error loss; this error loss information is then propagated back to update the parameters in the initial model, leading to convergence of the error loss. The backpropagation algorithm is an error-loss-driven backpropagation process aimed at obtaining optimal model parameters, such as the weight matrix.

[0081] (5) Large language model: Large Language Models (LLMs) are neural network models with a large number of parameters trained on vast corpora, capable of understanding and generating natural language text. Specifically, LLMs are typically based on neural network techniques, learning the syntax, semantics, and contextual information of a language through training on large amounts of text data. During training, the model continuously optimizes its parameters to improve its ability to understand and generate text. Due to their powerful ability to understand natural language, LLMs have been widely applied in many fields to solve natural language understanding and generation problems. LLMs have broad applications in artificial intelligence, such as natural language processing, machine translation, and dialogue systems.

[0082] (6) Description generator (captioner); The captioner model is a machine learning model that generates descriptions of images or videos.

[0083] Caption data is descriptive text for images or videos.

[0084] Taking an image captioner as an example, an image captioner is a model that can automatically generate descriptive text based on an input image. The input to an image captioner includes an image, and the output is descriptive text. An image captioner can generate a natural language text describing an input image—that is, the image's descriptive text. The image's descriptive text aims to accurately and in detail describe the key elements in the image, including but not limited to objects, scenes, and their relationships.

[0085] (7) Direct preference optimization (DPO); DPO is a method for optimizing language models, aiming to directly adjust the model output using human preference data to make it more in line with human expectations.

[0086] The optimization of DPO can be divided into two stages: supervised fine-tuning (SFT) and preference learning.

[0087] In the SFT phase, the pre-trained model is fine-tuned on a task-specific dataset to enable it to generate task-relevant outputs. Specifically, labeled data (such as question-answer pairs) can be used to train the model so that it can generate responses that meet task requirements. For example, in a dialogue task, SFT ensures that the model can generate answers relevant to the user's question.

[0088] In the preference learning phase, preference data is used to further optimize the model, enabling it to generate outputs that better align with human expectations. Specifically, a preference dataset can be constructed. For example, a preference dataset can include multiple triples. Each triple consists of the form (prompt, selected response, rejected response). Here, the selected response can be the preferred answer, and the rejected response can be the preferred answer. The model parameters are then optimized based on the preference dataset so that the probability of generating a selected response is higher than the probability of generating a rejected response.

[0089] (8) Expert model; An expert model is a special neural network architecture that improves the efficiency and performance of a model by breaking down a complex task into multiple sub-tasks, each handled by a different "expert" network (sub-model). Each sub-model focuses on processing a specific type of data or task.

[0090] (9) Image tagging model; Image labeling models are used to perform image labeling tasks. Image labeling tasks, also known as multi-label image recognition tasks, aim to provide multiple semantic labels for a given image. This can be understood as providing multiple labels to describe a given image, which can include: object, scene, attribute, and action.

[0091] The target refers to a specific entity or object in the image, such as a dog, a bicycle, or a tree.

[0092] Scene refers to the overall environment or context of an image, such as a beach, street, or forest.

[0093] Attributes provide additional descriptive information about the target or scene, such as red, tall, fast, or new, etc.

[0094] Actions refer to the actions that the subject in the image is performing, such as running, jumping, reading, or eating.

[0095] (10) Optical character recognition (OCR) model; OCR models can convert printed or handwritten text in different types of documents into machine-readable text formats. They identify human-readable characters by analyzing features such as shape and structure in image files and convert them into a data format that computers can process. OCR information refers to the text content extracted by the OCR model and its related metadata.

[0096] The text content extracted by the OCR model is the text content recognized from the document by the OCR model.

[0097] Metadata can include one or more of the following: location information or font style, etc. Location information refers to the position of the identified text in the original document. Font style refers to the font style of the identified text in the original document, such as font type, size, color, etc.

[0098] (11) Digging out difficult examples; The goal of hard case discovery is to improve model performance by identifying and focusing on training the model to handle samples that are difficult to handle correctly (i.e., "hard cases").

[0099] Difficult examples typically refer to samples that the model struggles to handle correctly during the current training phase, such as samples that the model predicts incorrectly or have low confidence levels.

[0100] The system architecture of the embodiments of this application will be described below.

[0101] Figure 1 A system architecture 100 applicable to embodiments of this application is illustrated. The system architecture 100 may include a local device 120, a local device 130, an execution device 110, and a data storage system 150, wherein the local devices 120 and 130 are connected to the execution device 110 via a communication network.

[0102] For example, the execution device 110 can be a device or server with data processing capabilities, such as a cloud server, network server, application server, or management server. For instance, the execution device 110 may be implemented by one or more servers, optionally in conjunction with other computing devices, such as data storage devices, routers, load balancers, etc. The execution device 110 may be deployed on a single physical site or distributed across multiple physical sites. The execution device 110 may use data from the data storage system 150 or call program code from the data storage system 150 to implement the method for generating descriptive text according to the embodiments of this application. The execution device 110 can also be referred to as a cloud device, in which case the execution device 110 may be deployed in the cloud.

[0103] In one possible implementation, the method executed by the aforementioned execution device 110 can be a training method executed in the cloud.

[0104] Users can interact with execution device 110 by operating their respective user devices (e.g., local device 120 and local device 130). Each local device can represent any computing device, such as a personal computer, computer workstation, smartphone, laptop, tablet, augmented reality (AR) / virtual reality (VR) device, ultra-mobile personal computer (UMPC), netbook, personal digital assistant (PDA), smart camera, smart car, other types of cellular phone, media consumption device, wearable device, set-top box, or game console, etc.

[0105] Each user's local device can interact with the execution device 110 through a communication network of any communication mechanism / standard. The communication network can be a wide area network, a local area network, a point-to-point connection, or any combination thereof.

[0106] In one implementation, local devices 120 and 130 can obtain relevant parameters of the description generation model from execution device 110, and use the description generation model to obtain description text on local devices 120 and 130.

[0107] In another implementation, a description generation model can be directly deployed on execution device 110. Execution device 110 obtains image data from local devices 120 and 130 and obtains description text through the description generation model.

[0108] For example, the data storage system 150 may be deployed in local device 120 or local device 130 for storing training data of the local device.

[0109] For example, the data storage system 150 can be deployed independently on the storage device, separate from the local device 120 or the local device 130. The storage device can interact with the local device to obtain user behavior logs from the local device and store them in the storage device.

[0110] Caption data is crucial in multimodal tasks, as it can improve the performance of visual models. However, current methods for acquiring caption data suffer from a number of problems and fail to meet practical application needs.

[0111] In view of this, embodiments of this application provide a method for generating descriptive text, which helps to improve the quality of generated descriptive text.

[0112] Figure 2 This is a schematic flowchart illustrating a method for generating descriptive text according to an embodiment of this application. Figure 2 The method 200 shown can be executed by a cloud server, by a terminal device, or by a combination of both.

[0113] like Figure 2 As shown, method 200 may include the following steps.

[0114] 210, Obtain the image to be processed.

[0115] 220, Obtain auxiliary information for the image to be processed.

[0116] The auxiliary information of the image to be processed includes at least one of the following: visual auxiliary information of the image to be processed, or world knowledge information of the image to be processed. The visual auxiliary information is obtained by performing visual processing on the image to be processed.

[0117] 230. Input the image to be processed and its auxiliary information into the captioner model #21 (i.e., the first description generation model) to generate the description text of the image to be processed.

[0118] For example, in step 210, the image to be processed can be acquired in a variety of ways.

[0119] For example, the image to be processed can be user input.

[0120] For example, the image to be processed can be stored locally.

[0121] For example, the image to be processed may be received from other devices.

[0122] The embodiments of this application do not limit the specific method of acquiring the image to be processed.

[0123] In this application's embodiments, the "image to be processed" can be a single image or an image frame from a video. In other words, the solution in this application's embodiments can be used to generate descriptive text for a single image or for a video.

[0124] The image description text in the embodiments of this application can also be replaced with other descriptions such as caption data.

[0125] Step 220 will be explained below.

[0126] Optionally, the visual aid information of the image to be processed may include any one or more of the following: the label of the image to be processed, the location information of the target in the image to be processed, the OCR information of the image to be processed, and the number of targets in the image to be processed.

[0127] For example, the label of the image to be processed can be used to indicate one or more of the following: target, scene, attribute, and behavior.

[0128] The target refers to a specific entity or object in the image to be processed, such as a dog, a bicycle, or a tree. The labels in the image's tags that indicate the target can be understood as the target's label, used to indicate the target's category.

[0129] Scene refers to the overall environment or context of the image to be processed, such as a beach, street, or forest. The labels in the image's tagging section that indicate the scene can be understood as scene tags, used to indicate the category of the scene.

[0130] Attributes provide descriptive information about the target or scene, such as red, tall, fast, or new. The labels in the image to be processed that indicate attributes can be understood as attribute labels, used to indicate descriptive information about the target or scene.

[0131] A behavior refers to an action being performed by the subject in the image to be processed, such as running, jumping, reading, or eating. The labels in the image's tags that indicate behaviors can be understood as labels for behaviors, indicating the category of the behavior.

[0132] For example, the location information of a target in the image to be processed can be indicated by the detection bounding box of the target in the image to be processed.

[0133] For example, the OCR information of the image to be processed may include the text content in the image. Furthermore, the OCR information of the image to be processed may also include metadata about the text content, such as the location of the text content.

[0134] For example, the number of targets in the image to be processed can be indicated by counting the number of detection boxes corresponding to each target in the image to be processed.

[0135] Visual auxiliary information of the image to be processed can be obtained through one or more visual processing models.

[0136] For example, in step 220, the image to be processed can be visually processed using an expert model to obtain visual aid information. The expert model can be used to mine various types of visual aid information.

[0137] For example, the sub-models in an expert model can include any of the following: object detection model, image segmentation model, image tagging model, or optical character recognition (OCR) model.

[0138] Object detection models can be used to output the location and label of objects in an image being processed. The location of an object can be indicated by its detection bounding box.

[0139] The image segmentation model can be used to output segmented blocks of entities in the image and their categories. A segmented block can be a region surrounded by a series of coordinate points.

[0140] Alternatively, in step 220, the image to be processed can be visually processed by one or more visual processing models to obtain visual aid information.

[0141] For example, one or more vision processing models may include any one or more of the following: a model for classifying images to be processed, a model for detecting objects, a model for segmenting images to be processed, a model for labeling images to be processed, or an OCR model.

[0142] The above are merely examples; visual assistance information may also include other content, and this application does not limit this.

[0143] All information related to the image to be processed, excluding visual aids, can be considered as world knowledge information of the image. This world knowledge information can also be referred to as the world knowledge or knowledge aids of the image.

[0144] The types of information related to the image to be processed may include at least one of the following: video related to the image to be processed, text related to the image to be processed, or audio related to the image to be processed.

[0145] For example, information related to the image to be processed may include at least one of the following: Scene information and socio-cultural background information of the image to be processed, or user feedback on the initial descriptive text of the image to be processed.

[0146] For example, scene information may include at least one of the following: time, location, or weather.

[0147] For example, socio-cultural background information can include the context of events (such as festivals or historical events).

[0148] By combining scene information and socio-cultural background information of the image to be processed, the understanding of the image can be enhanced, which is conducive to providing better descriptive text for images in contexts that depend on information from the external world.

[0149] The initial description text of the image to be processed can be any description text of the image to be processed. This description text can be obtained in any way.

[0150] By analyzing user feedback on the initial descriptive text for the image being processed, the captioner model can be continuously improved. User feedback provides information about their understanding and preferences regarding the generated descriptions, helping to progressively optimize the generation process and adjust the content of the descriptive text based on user cognition and expectations in a specific context.

[0151] Optionally, world knowledge information can be retrieved based on the image to be processed.

[0152] Retrieving world knowledge information from the image to be processed can be achieved in various ways. Examples 1 to 4 are used as examples below.

[0153] Example 1: World knowledge information about the image to be processed can be obtained by searching for text in the image.

[0154] For example, the image to be processed is input into a search tool for retrieval. By searching for text related to the image, relevant text information can be obtained. This text information can serve as world knowledge information about the image. For instance, if the image is a portrait, the relevant text information might include the person's biography. If the image is a landscape, the relevant text information might include the location and time the landscape was photographed. Furthermore, if the image is a work of art, the relevant text information might include the artist's information, creation date, and creative concept.

[0155] Example 2: The world knowledge information of the image to be processed may include world knowledge information of candidate images related to the image to be processed, and / or visual aids information of the related images. Candidate images may be obtained through image search.

[0156] For example, the image to be processed is input into a search tool for retrieval, and one or more images are obtained through image-to-image search. These one or more images can serve as candidate images related to the image to be processed. Image-to-text search is then performed on the candidate images to obtain world knowledge information about them. Visual processing or visual-assisted processing is then performed on the candidate images to obtain visual aid information. The visual aid information and / or world knowledge information of the candidate images can be used as world knowledge information for the image to be processed.

[0157] Example 3: World knowledge information about the image to be processed can be obtained through iterative retrieval based on textual information related to the image. The textual information related to the image can be obtained through image-to-text search.

[0158] For example, the image to be processed is input into a search tool for retrieval. Textual information related to the image is obtained through image-to-text search. The search is then iteratively performed based on this textual information until an iteration stopping condition is met, thus obtaining the world knowledge information about the image. The iteration stopping condition can be set as needed. For example, the iteration stopping condition could be that the number of iterations meets a preset threshold.

[0159] For example, based on the image to be processed, a tag for the image is obtained through image-to-text search (an example of text information related to the image to be processed). Based on the tag, images matching the tag are retrieved. World knowledge information and / or visual aid information for the image are obtained through the method in Example 2, and this information is used as the world knowledge information for the image to be processed. The tag for the image to be processed can be used to indicate the category of entities in the image.

[0160] Example 4: World knowledge information about the image to be processed can be retrieved based on the image's tags. For example, the tags of the image can be used to indicate the category of entities in the image. The tags of the image can be provided by the user.

[0161] For example, the tags of the image to be processed can be entered into a search tool to retrieve related textual information. This related textual information can serve as world knowledge information about the image to be processed.

[0162] The above are merely examples. World knowledge information can also be obtained through methods other than retrieval, and this application does not limit this.

[0163] Step 230 will be explained below.

[0164] The captioner model #21 is used to generate descriptive text for the image to be processed.

[0165] The captioner model can also be replaced with other descriptions such as the description generation model.

[0166] The number #21 for captioner model is for descriptive convenience only and has no limiting effect.

[0167] For example, the label of the image to be processed, the detection box of the target in the image to be processed, and the OCR information in the image to be processed are used as visual auxiliary information of the image to be processed, and the world knowledge information of the image to be processed is used as knowledge auxiliary information of the image to be processed. The visual auxiliary information, knowledge auxiliary information, and the image to be processed are input into the captioner model #21, and the captioner model #21 outputs the descriptive text of the image to be processed.

[0168] For example, auxiliary information can be input into the captioner model #21 in the form of a text description. Alternatively, auxiliary information can also be input into the captioner model #21 in other forms, which is not limited in this embodiment. For ease of description, this embodiment mainly uses text description as an example and does not constitute a limitation on the solution of this embodiment.

[0169] As an example, the input data for the captioner model #21 may include: the image to be processed, auxiliary information about the image to be processed, and instruction text.

[0170] The instruction text is used to describe the task to be processed. For example, the instruction text could be "Please describe the content of the image". Taking the auxiliary information of the image to be processed as a text description input into the captioner model #21 as an example, the auxiliary information of the image to be processed and the instruction text can be used as two inputs to the captioner model #21 respectively. Alternatively, the auxiliary information of the image to be processed can be used as instruction follow-up data, that is, after being fused with the instruction text, it can be used as one input to the captioner model #21, for example, as a prompt.

[0171] In the scheme of this application embodiment, world knowledge of the image is introduced during the process of generating descriptive text by the captioner model. This avoids relying solely on the model's own world knowledge to describe the image, effectively expanding the model's world knowledge and improving its content description capabilities. Furthermore, it enhances the ability to describe long-tail elements, thereby improving the quality of the descriptive text. Simultaneously, during the process of generating descriptive text by the captioner model, multimodal information fusion is achieved based on visual guidance. This visual guidance helps avoid problems such as description errors and illusions, further improving the quality of the descriptive text. Moreover, the time and computational resources required to acquire world knowledge of the image are relatively small. The scheme of this application embodiment helps to improve the quality of the descriptive text while increasing generation efficiency and avoiding excessive resource consumption.

[0172] In the scheme of this application embodiment, visual auxiliary information of the image can be obtained through visual processing. This visual auxiliary information is introduced during the process of the captioner model generating descriptive text, thereby avoiding reliance solely on the model's own world knowledge to describe the image. This effectively expands the model's world knowledge, improving its content description capabilities and, more importantly, its ability to describe long-tail elements, thus enhancing the quality of the descriptive text. Simultaneously, during the process of the captioner model generating descriptive text, the fusion of multimodal information is achieved based on visual guidance. This visual guidance helps avoid problems such as description errors and illusions, further improving the quality of the descriptive text.

[0173] In the embodiments of this application, visual auxiliary information and world knowledge of the image are introduced during the process of generating descriptive text by the captioner model. This further expands the model's world knowledge, which is beneficial for improving the model's content description ability and thus further improving the quality of the descriptive text. Simultaneously, in the process of generating descriptive text by the captioner model, the fusion of multimodal information is achieved based on visual guidance. Visual guidance helps avoid problems such as description errors and illusions, thereby further improving the quality of the descriptive text.

[0174] The following provides an illustrative example of the application scenarios for Method 200. Method 200 can be applied to the inference stage (such as in scenarios 1 and 2). In this case, the captioner model #21 can be a pre-trained model.

[0175] Scene 1: Method 200 can be used to describe the processing scenario of text generation tasks.

[0176] As an example, the image to be processed in method 200 can be input by the user. The descriptive text generated by method 200 can be provided to the user.

[0177] For example, step 210 may include receiving an input image to be processed and instruction text. Method 200 may also include outputting a response to the instruction text. For example, if the instruction text is "Please describe the image content", then the response to the instruction text may be a descriptive text of the image to be processed.

[0178] Scene 2: The descriptive text generated by method 200 can be used to construct training data for training other captioner models.

[0179] For example, the descriptive text of image a is generated by method 200, and image a and its descriptive text are used as labeled data for SFT training of other captioner models.

[0180] Furthermore, when method 200 is applied to the inference phase, the captioner model #21 can be trained based on DPO.

[0181] Optionally, captioner model #21 is obtained by training captioner model #22 (i.e., the second description generation model) using dataset #21 (i.e., the first dataset). The training data in dataset #21 includes image sample #21 (i.e., the first image sample), preferred samples of image sample #21, and unpreferred samples of image sample #21. During training, the parameters of captioner model #22 are adjusted so that the probability of captioner model #22 generating preferred samples is higher than the probability of captioner model #22 generating unpreferred samples. The preferred samples and unpreferred samples of image sample #21 are two descriptive texts for image sample #21.

[0182] The numbering of captioner model #22 is merely an example and does not constitute a limitation on the scheme of the embodiments of this application. The captioner model in the training process can be regarded as captioner model #22.

[0183] The numbering of dataset #21 and image sample #21 is merely an example and does not constitute a limitation on the scheme of the embodiments of this application.

[0184] Optionally, the non-preference sample and the preference sample of image sample #21 are inconsistent contents in the predicted description text and the target description text of image sample #21, respectively, and the predicted description text of image sample #21 is generated by captioner model #22.

[0185] Optionally, the target description text of image sample #21 is obtained by correcting the predicted description text of image sample #21.

[0186] For a detailed description of DPO training, please refer to Method 600 below.

[0187] Furthermore, when method 200 is applied to the inference phase, the captioner model #21 can be trained based on hard examples.

[0188] Optionally, captioner model #21 is obtained by training captioner model #23 (i.e., the third description generation model) on dataset #22 (i.e., the second dataset). Image sample #22 (i.e., the second image sample) in dataset #22 is a hard example. The hard example status of image sample #22 is determined based on the predicted description text of image sample #22, which is generated by captioner model #23.

[0189] Optionally, the determination that image sample #22 is a hard example is based on the predicted description text of image sample #22 and a quality control strategy used to determine whether an image is a hard example.

[0190] Optionally, image sample #23 (i.e., the third image sample) in dataset #22 is retrieved based on the category of image sample #22 and / or the predicted descriptive text of image sample #22.

[0191] The numbering of captioner model #23 is merely an example and does not constitute a limitation on the scheme of the embodiments of this application. The captioner model in the training process can be regarded as captioner model #23.

[0192] The numbering of dataset #22 and image sample #22 is merely an example and does not constitute a limitation on the solutions of the embodiments of this application.

[0193] For a detailed description of training based on difficult examples, please refer to Method 300 below.

[0194] Method 200 can also be applied to the training phase (as in scenarios 3 and 4). In this case, the captioner model #21 can be an untrained model, or a model that needs further optimization.

[0195] Scene 3: The descriptive text generated using method 200 can be used to discover hard examples. The discovered hard examples can be used to optimize the captioner model #21.

[0196] For a detailed description, please refer to method 300 below.

[0197] Scene 4: The descriptive text generated by method 200 can be used for DPO training.

[0198] For a detailed description, please refer to method 600 below.

[0199] This application also provides a training method for a descriptive generative model, which mines difficult examples during the training process of the descriptive generative model and improves the generation quality of the descriptive generative model based on the difficult examples.

[0200] Figure 3 A schematic flowchart illustrating a training method for a generative model according to an embodiment of this application is shown. Figure 3 The method 300 shown can be executed by a cloud server, by a terminal device, or by a combination of both.

[0201] like Figure 3 As shown, method 300 may include the following steps.

[0202] 310. Input image sample #31 into captioner model #31 to generate descriptive text for image sample #31.

[0203] 320. Based on the image sample #31 and its description text, image sample #31 is determined to be a difficult example.

[0204] 330. Based on dataset #31, the captioner model #31 is trained to obtain captioner model #32. Image sample #31 belongs to dataset #31.

[0205] Method 300 enables the discovery of difficult examples that describe generative models. In Method 300, "difficult examples" can be understood as image samples, etc., that are difficult for the model to accurately describe.

[0206] The image sample #31 is designated for ease of description only and does not constitute a limitation on the embodiments of this application.

[0207] Image sample #31 comes from dataset #32, which can be any dataset containing one or more image samples. Image sample #31 can be any image sample in dataset #32. For ease of description, this embodiment mainly uses image sample #31 as an example to illustrate method 300. Other image samples in dataset #32 can also be processed in the same way as image sample #31.

[0208] Captioner model #31 and captioner model #32 are the models before and after training based on difficult examples, respectively. The numbering of captioner model #31 and captioner model #32 is for ease of description only and does not constitute a limitation on the scheme of the embodiments of this application.

[0209] In step 310, image sample #31 is input into captioner model #31, and predicted descriptive text for image sample #31 is generated through forward propagation.

[0210] In the embodiments of this application, the descriptive text output by the captioner model can also be referred to as the predicted descriptive text.

[0211] Optionally, step 320 may include: determining that image sample #31 is a difficult example based on the description text of image sample #31 and a quality control strategy. The quality control strategy is used to determine whether an image sample is a difficult example.

[0212] The quality control strategy is used to determine whether the descriptive text of an image sample meets the quality requirements. If it does, the image sample is not considered a difficult example; otherwise, it is considered a difficult example.

[0213] Quality control strategies can be implemented in various ways. The following provides an example of a quality control strategy.

[0214] As an example, a quality control strategy may include a quality control model.

[0215] For example, the quality control model can be a multimodal large model. For instance, the input to the quality control model can include an image sample and its descriptive text. The output of the quality control model can include a score for the descriptive text of the image sample, which can be used to indicate the generation quality. The comparison between the descriptive text score of the image sample and a threshold #1 can be used to determine whether the image sample belongs to the difficult examples category.

[0216] For example, the higher the score of the descriptive text of an image sample, the higher the quality of the generated descriptive text. If the score of the descriptive text of an image sample is less than or equal to threshold #1, the quality of the generated descriptive text of that image sample is poor, and the image sample is classified as a hard example. Taking image sample #31 as an example, image sample #31 and its descriptive text are input into the quality inspection model. The model outputs the score of the descriptive text of image sample #31. If this score is less than or equal to threshold #1, then image sample #31 is determined to be a hard example.

[0217] As another example, a quality control strategy may include one or more quality control rules.

[0218] For example, the one or more quality control rules can be used to determine whether an image sample is a difficult example based on errors in the descriptive text of the image sample.

[0219] Errors in the descriptive text of image samples can be identified manually or through a model. This application does not limit this.

[0220] For example, one or more quality control rules may include classifying an image sample as a difficult example if its descriptive text satisfies at least one of the following: The number of errors in the descriptive text of this image sample is greater than or equal to threshold #2; or, The error type in the description text of this image sample belongs to the preset type.

[0221] Taking image sample #31 as an example, errors in the description text of image sample #31 are manually identified. If the number of errors in the description text of image sample #31 is greater than or equal to the threshold #2, or if the errors belong to a preset type, then image sample #31 is determined to be a difficult case.

[0222] The above is just a simple example. In practical applications, quality inspection rules can be designed based on multiple dimensions. For example, first-level quality inspection rules are used to analyze the descriptive text of image samples from the following perspectives: comprehensiveness, illusion, attribute accuracy, and image-text quality. In second-level quality inspection rules, comprehensiveness includes supplementary points and omissions; illusion includes cueing and severity categories; attribute accuracy includes cueing, general, severity, and subject category errors; and image-text quality includes irrelevant text, irrelevant text, and duplicate text.

[0223] The above quality inspection rules are described below.

[0224] Comprehensiveness can be used to check whether the caption fully describes the key information in the image.

[0225] The omissions in the comprehensive second-level dimension refer to the core elements that are not mentioned in the caption but are prominently present in the image.

[0226] Supplementary points in the comprehensive second-level dimension refer to supplementary information that can be inferred from the image but is not mentioned in the caption.

[0227] Hallucination can be used to determine whether there is fictitious information in the caption that does not match the actual content of the image.

[0228] The secondary dimensions of hallucination are: cue class and severity class, which correspond to different degrees of severity of hallucination problems.

[0229] Attribute accuracy can be used to confirm whether the attribute descriptions of the subject in the image are accurate.

[0230] The secondary dimension of attribute accuracy, namely, the alert category, the general category, and the severe category, corresponds to different degrees of severity of attribute accuracy issues.

[0231] Subject category error in the second-level dimension of attribute accuracy corresponds to the subject category error in the image.

[0232] Image and text quality can be used to assess the overall quality of descriptive text.

[0233] In the second-level dimension of image and text quality, image and text irrelevance refers to the fact that the content described in the caption has no relation to the content displayed in the image.

[0234] In the secondary dimension of image quality, irrelevant text refers to extraneous text information in the caption that is unrelated to the image content.

[0235] Duplicate text in the second-level dimension of image and text quality refers to repeated descriptions of the same content in the caption.

[0236] Furthermore, quality inspection models and quality inspection rules can also be used in combination, and this application does not limit this.

[0237] The above is just an example. In practical applications, other methods, such as manual quality inspection, can also be used to determine whether an image sample is a difficult example. For example, in step 320, image sample #31 can be determined to be a difficult example through manual quality inspection.

[0238] The designation of dataset #31 is for convenience of description only and does not constitute a limitation on the solutions implemented in this application. Dataset #31 is the dataset used to train the model based on difficult examples.

[0239] Dataset #31 may include one or more image samples, which include hard cases determined by the aforementioned scheme.

[0240] Dataset #31 can be a hard example dataset, meaning that all elements in this dataset are hard examples.

[0241] In step 330, the parameters of captioner model #31 can be adjusted based on dataset #31 to optimize captioner model #31.

[0242] For example, the captioner model #31 can be trained based solely on dataset #31.

[0243] For example, the captioner model #31 can be trained together with dataset #31 and other datasets, that is, the training dataset is composed of dataset #31 and other datasets, thereby optimizing the training dataset.

[0244] Furthermore, the proportion of dataset #31 in the training dataset can be set as needed. This allows for adjusting the distribution of the training data by changing the proportion of dataset #31 in the training dataset, thus making the distribution of the training data more reasonable.

[0245] Furthermore, higher weights can be assigned to the image samples in dataset #31 in the loss function, forcing the model to pay more attention to these samples.

[0246] It should be understood that the above are only some examples of optimization models based on difficult examples. Other optimization methods can also be used to optimize the model, and the embodiments of this application do not limit this.

[0247] In the scheme of this application embodiment, difficult examples can be mined and used to optimize the captioner model, which is beneficial to improving the generation quality of the captioner model. For example, the distribution of the training dataset can be optimized based on the mined difficult examples to make the distribution of the training dataset more reasonable, that is, to improve the quality of the training dataset, thereby improving the generation quality of the captioner model.

[0248] Further, optionally, method 300 may also include step 340.

[0249] 340. Retrieve image sample #32 related to image sample #31 based on the category of image sample #31 and / or the descriptive text of image sample #31. Dataset #31 includes image sample #32.

[0250] In step 340, image samples related to the difficult example (such as image sample #31) can be retrieved and added to the training dataset (such as dataset #31) to expand the training dataset.

[0251] The designation of image sample #32 is for convenience of description only and does not constitute a limitation on the embodiments of this application. One or more image samples related to image sample #31 obtained through retrieval can all be regarded as image sample #32.

[0252] For example, image sample #32 may come from dataset #33, i.e., the dataset to be retrieved. In other words, in step 340, a retrieval is performed in dataset #33 based on the category of image sample #31 and / or the descriptive text of image sample #31.

[0253] Image sample #31 and image sample #32 can come from the same dataset, that is, dataset #32 and dataset #33 are the same dataset. Alternatively, image sample #31 and image sample #32 can also come from different datasets, that is, dataset #32 and dataset #33 are different datasets.

[0254] As one possible implementation, image sample #32 could be determined based on the category of image sample #31.

[0255] Optionally, image sample #32 belongs to the same category as image sample #31.

[0256] In other words, an image sample of the same category as a difficult example (such as image sample #31) can be used as an image sample related to that difficult example (such as image sample #32).

[0257] For an image sample, it can be one category or multiple categories.

[0258] For example, a multi-level classification dimension can be designed, so that an image sample can have multiple levels of categories.

[0259] For example, the categories corresponding to the first-level classification dimension include one or more of the following: nature, people, objects, architecture, transportation, or art and culture. In the second-level classification dimension, categories belonging to "nature" include one or more of the following: landscape, animals, or plants; categories belonging to "people" include one or more of the following: portraits, activities, sports, clothing, or accessories; categories belonging to "objects" include one or more of the following: toys, tools, food, or home furnishings; categories belonging to "architecture" include buildings; categories belonging to "transportation" include one or more of the following: cars, ships, airplanes, or high-speed trains; and categories belonging to "art and culture" include: arts and crafts, etc. The categories corresponding to the first-level classification dimension can be called major categories. The categories corresponding to the second-level classification dimension can be called minor categories. For example, the categories of the two-level classification dimension of an image can be represented as "x_y", where x represents the major category and y represents the minor category. Figure 4 Taking "Nature_Landscape" as an example, the main category is "Nature" and the subcategory is "Landscape".

[0260] Image sample #32 has the same category as image sample #31, which means that image sample #32 and image sample #31 have at least one category in common.

[0261] For example, image sample #31 belongs to the same major category as image sample #32, so it can be considered that image sample #31 and image sample #32 belong to the same category.

[0262] Alternatively, image sample #32 may have the same category as image sample #31, or all categories of image sample #32 may be the same as those of image sample #31.

[0263] For example, image sample #31 and image sample #32 have the same major and minor categories, so they can be considered to have the same category.

[0264] By designing multi-level classification dimensions, we can obtain the image categories across multiple dimensions, which facilitates better analysis of model performance and thus helps to further improve it. Specifically, multi-level classification dimensions make it easier to evaluate the model's performance across various dimensions, allowing for more accurate identification of the model's weaknesses. Based on this, optimizing the quality of the training dataset can further improve model performance. For example, using a similar approach to image sample #31, we can determine whether multiple image samples belong to the hard category. Assuming that most of the hard image samples belong to the category "items_food," it indicates that the current model may have difficulty accurately describing images of the "items_food" category. Compared to the classification dimension for "items," the classification dimension for "food" is more specific, allowing for a more accurate identification of the model's problems. Based on this, optimizing the training dataset can improve the model's accuracy in describing images of this category.

[0265] For example, the category of an image sample can be obtained through an image classification model.

[0266] The categories of image sample #31 and image sample #32 can be obtained based on the same one or more image classification models.

[0267] As another possible implementation, image sample #32 can be determined based on the descriptive text of image sample #31.

[0268] For example, the similarity between the descriptive text of image sample #32 and the descriptive text of image sample #31 is greater than or equal to the threshold #3.

[0269] In other words, if the description text of an image sample is similar to the description text of a difficult example (such as image sample #31), then the image sample can be used as an image sample related to that difficult example (such as image sample #32).

[0270] For example, the descriptive text of image sample #32 includes keywords from the descriptive text of image sample #31.

[0271] In other words, if the descriptive text of an image sample includes keywords from the descriptive text of a difficult example (such as image sample #31), then that image sample can be used as an image sample related to that difficult example (such as image sample #32).

[0272] As another possible implementation, image sample #32 can be determined based on the category of image sample #31 and the descriptive text of image sample #31.

[0273] Image sample #32 can be determined based on both implementation methods mentioned above. For detailed descriptions, please refer to the previous text; they will not be repeated here.

[0274] In the scheme of this application embodiment, relevant image samples can be retrieved based on the category and / or descriptive text of the hard examples that have been mined, and these can be used as new hard examples to expand the training dataset.

[0275] Furthermore, dataset #33 and / or dataset #32 may be obtained through image deduplication.

[0276] The following example uses dataset #33 as an illustration.

[0277] For example, dataset #33 can be obtained by deduplicating image samples from the original dataset. Furthermore, image classification can be performed on the image samples retained after deduplication to determine the categories of the image samples in dataset #33.

[0278] The original dataset can also be replaced with the original image library, etc.

[0279] For example, image classification is performed on the image samples in the original dataset to obtain the category of each image sample in the original dataset; image deduplication is performed on image samples of the same category to obtain dataset #33. Thus, dataset #33 includes one or more image samples and their categories.

[0280] In other words, the execution order of image deduplication and image classification is not limited in the embodiments of this application.

[0281] The method for obtaining dataset #32 can also be found in dataset #33. To avoid duplication, it will not be repeated here.

[0282] Image deduplication can be implemented using existing methods, such as image feature encoding. Alternatively, image deduplication can be implemented in other ways, which are not limited in this application.

[0283] In the scheme of this application embodiment, the dataset to be retrieved (such as dataset #33) can be obtained by image deduplication, which is beneficial to improving the diversity and information density of the data, thereby improving retrieval efficiency and retrieval quality.

[0284] Methods 200 and 300 can be used in combination.

[0285] As an example, the captioner model #31 in method 300 and the captioner model #21 in method 200 can be the same model. In other words, the descriptive text generated by method 200 can be used to mine hard cases, i.e., scenario 3 in method 200. In this case, image sample #31 in method 300 is the image to be processed in method 200, and correspondingly, the input of captioner model #31 also includes auxiliary information of image sample #31. The relevant description of the auxiliary information of image sample #31 can be found in the description in method 200; simply replace the image to be processed in method 200 with image sample #31.

[0286] As another example, the captioner model #32 in method 300 and the captioner model #21 in method 200 can be the same model. In other words, the model trained by method 300 can be used as the model in the inference phase of method 200.

[0287] In this case, the captioner model #31 in method 300 and the captioner model #23 in method 200 can be the same model, the image sample #31 in method 300 can be the image sample #22 in method 200, and the dataset #31 in method 300 can be the dataset #22 in method 200.

[0288] Figure 4 This illustration shows a schematic diagram of a training process for a generative model according to an embodiment of this application. Figure 4 The process shown can be regarded as Figure 2 The method shown and Figure 3 This is an example of a combination of the methods shown. For a detailed implementation, please refer to [link / reference]. Figure 2 and Figure 3 To avoid duplication, the relevant descriptions are as follows: Figure 4 Some descriptions may be omitted when appropriate.

[0289] like Figure 4As shown, the process may include the following steps.

[0290] S1, Image Classification.

[0291] In step S1, the images in the original image library are classified.

[0292] like Figure 4 As shown, images in the original image database are classified using an image classification model. There can be one or more image classification models.

[0293] For example, an image category system can be constructed based on multi-level classification dimensions. This allows images to be classified across multiple classification dimensions, resulting in the image's class across those dimensions. Figure 4 As shown.

[0294] For a detailed description of the classification process, please refer to Method 300 above, which will not be repeated here.

[0295] S2, Image deduplication.

[0296] For example, in step S2, the original image library can be deduplicated to obtain the original training dataset.

[0297] Alternatively, in step S2, image deduplication can be performed separately for each category of images to obtain the original training dataset. That is, image deduplication is performed on images belonging to the same category, and the results of deduplication for each category are used to construct the original training dataset.

[0298] For example, the original training dataset can be used as dataset #32 in method 300, and the images in the original training dataset can be used as image samples in method 300.

[0299] S3, obtain auxiliary information of image samples.

[0300] Taking an image sample from the original training dataset as an example, in step S3, auxiliary information for that image sample is obtained. For example, such as... Figure 4 As shown, the auxiliary information for this image sample can include visual auxiliary information and knowledge auxiliary information. The visual auxiliary information for the image sample can include labels, target detection boxes (i.e., "detection boxes" in the figure), and OCR information.

[0301] S4 generates descriptive text for the image samples.

[0302] In step S4, the auxiliary information of the image sample and the image sample are input into the captioner model to generate the descriptive text of the image sample.

[0303] This captioner model can be viewed as captioner model #31 in method 300.

[0304] S5, Difficult Case Mining.

[0305] Determine whether the image sample belongs to the difficult cases.

[0306] For example, if the image sample is determined to be a difficult example, it can be added to the difficult example dataset to achieve difficult example reflow.

[0307] For example, if the image sample is determined to be a difficult example, image samples related to the image sample can be retrieved based on the category of the image sample and / or the descriptive text of the image sample, and the retrieved image samples can be added to the difficult example dataset to achieve continuous expansion of the training dataset.

[0308] For each image sample in the original training dataset, steps S3 to S5 can be performed.

[0309] The original training dataset and the hard example dataset can be used together as the training dataset for the captioner model to optimize the captioner model.

[0310] Figure 5 This is a schematic diagram of an example image. Inputting this image into different description generation models yields the following descriptive text.

[0311] The English description text generated using ShareGPT4V is translated into Chinese as follows: This photograph presents a striking view against a backdrop of a clear blue sky. On the left side of the frame is a large white sphere perched atop a sturdy stone tower. Four pointed antennas rise high from the tower, seemingly overlooking the landscape below. On the right side of the image, a flag flutters in the breeze, adding vibrant color to the scene. In the foreground, a group of trees can be seen. Their lush green leaves create a natural contrast with the man-made structures in the image. The precise placement and relative positions of these objects—the tower, the flag, and the trees—create a visually pleasing composition that draws the viewer's eye. The absence of any text or discernible movement allows the viewer to focus solely on the objects, their interactions, and their surroundings.

[0312] The descriptive text generated using the Internal Vision-Language (InternVL) 2 model is as follows: This image shows a building located on a mountaintop, which may be one of the Antarctic research stations. The main body of the building is a white dome with geometric patterns, resembling an astronomical observation instrument. The dome is surrounded by gray stone walls with several small windows.

[0313] In front of the building stood several pine trees, their branches and leaves thriving in the sunlight. Behind the pine trees was a flag, its bright red color fluttering in the wind.

[0314] The sky was clear, a deep blue, without a single cloud, indicating excellent weather conditions during the shoot. The entire scene evoked a sense of tranquility and solemnity, while also showcasing human exploration and research activities in extreme environments.

[0315] The descriptive text generated using the Qwen2-vision language (VL) model is as follows: This image shows a white, spherical building perched atop a mountain. The building has a dome at the top with two antennas. The main body of the building is constructed of gray stone, appearing very sturdy and stable. To the right of the building, a red flag flutters in the wind. In front of the building, there are several green trees with leaves in varying shades of green, giving a vibrant feeling. The sky is a clear blue, without a single cloud, appearing very bright and open.

[0316] The description text generated using the captioner model of this application is as follows: In this picture, we see a building situated atop a mountain against a clear blue sky. This building is the Huangshan Meteorological Station on Bright Summit of Huangshan Mountain, primarily used for meteorological observation. The main structure is made of gray stone, presenting a sturdy and rustic style. At the top is a white spherical structure with two antennas beside it, possibly for radar or other meteorological equipment.

[0317] Several pine trees surround the building, their dense, deep green foliage contrasting sharply with the architecture. The pines are arranged rather haphazardly, growing naturally on the hillside, adding to the natural, rustic charm.

[0318] On the right side of the building, a bright red flag fluttered in the wind. Positioned near the edge of the building, the flag appeared to sway gently in the breeze.

[0319] The entire scene is exceptionally bright under the sunlight, with a cloudless blue sky, giving a sense of openness and tranquility. Without any obvious human activity, the whole picture appears serene and dignified, embodying the harmonious beauty of nature and man-made structures.

[0320] This application also provides a method for training a description generation model, which trains the model based on preference data of the description text. This method is beneficial for optimizing the performance of the description generation model and improving the quality of the generated description text.

[0321] Figure 6 A schematic flowchart illustrating another training method for a generative model provided in an embodiment of this application is shown. Figure 6 A schematic flowchart illustrating a training method for a generative model according to an embodiment of this application is shown. Figure 6 The method 600 shown can be executed by a cloud server, by a terminal device, or by a combination of both.

[0322] like Figure 6 As shown, method 600 may include the following steps.

[0323] 610. Obtain dataset #61 (i.e., the third dataset). The training data in dataset #61 includes image sample #61 (i.e., the fourth image sample), preferred samples of image sample #61, and unpreferred samples of image sample #61. The preferred and unpreferred samples of image sample #61 are two descriptive texts for image sample #61.

[0324] 620. The captioner model #61 (i.e., the fourth description generation model) is trained using training data. During training, the parameters of the captioner model #61 are adjusted so that the probability of the captioner model #61 generating preferred samples is higher than the probability of the captioner model #61 generating unpreferred samples.

[0325] For example, in method 600, a training data set can be referred to as a preference data pair, comprising: an image sample, a preference sample of that image sample, and a non-preference sample of that image sample. A training data set can be represented in the form of a triple, i.e. . This represents the input data, which includes image samples. Furthermore, the input data may also include prompts, etc. This represents the preferred sample of the image sample. This represents the non-preference sample of the image sample.

[0326] The preferred samples of an image can be understood as the descriptive text that the captioner model wants to output, i.e., the preferred descriptive text. Correspondingly, the unpreferred samples of an image can be understood as the descriptive text that the captioner model does not want to output, i.e., the unpreferred descriptive text.

[0327] In a pair of preference data, preferred samples and non-preferred samples can be understood as relative concepts. Compared to non-preferred samples, preferred samples are "better" descriptive text, that is, higher quality descriptive text.

[0328] The quality of descriptive text can be determined in several ways.

[0329] As an example, a quality control strategy can be used to determine preferred and unpreferred samples.

[0330] For example, a quality inspection strategy may include a quality inspection model.

[0331] For example, the input to a quality control model can include an image sample and its descriptive text. The output of the model can include a score for the descriptive text of the image sample, which can be used to indicate the generation quality. Assuming that a higher score for the descriptive text of an image sample indicates higher generation quality, then the descriptive text with the higher score among the two descriptive texts of the image sample is the preferred sample.

[0332] For example, a quality inspection strategy may include one or more quality inspection rules.

[0333] For example, in two descriptive texts of an image sample, the descriptive text with fewer errors can be used as the preferred sample.

[0334] For example, in the two descriptive texts of an image sample, the descriptive text containing a preset type of error can be used as a non-preferred sample.

[0335] Furthermore, quality inspection models and quality inspection rules can also be used in combination, and this application does not limit this.

[0336] For descriptions of text quality, please refer to the description in the difficult case mining section above; it will not be repeated here.

[0337] It should be understood that the above are merely examples, and in practical applications, preferred and unpreferred samples can be determined in other ways. This application's embodiments do not limit this approach.

[0338] Method 600 can be understood as follows: DPO training of the captioner model based on preference data. The optimization strategy during training is to adjust the parameters of the captioner model so that it is more inclined to generate preferred samples than non-preferred samples. That is, adjusting the parameters of the captioner model so that the probability of generating preferred samples is higher than the probability of generating non-preferred samples. In other words, adjusting the parameters of the captioner model to increase the difference between the probabilities of generating preferred samples and non-preferred samples, or adjusting the parameters of the captioner model with the goal of maximizing the difference between the probabilities of generating preferred samples and non-preferred samples.

[0339] Steps 610 to 620 above can be executed iteratively, i.e., iterative training, to achieve progressive optimization of the captioner model.

[0340] In the scheme of this application embodiment, the model is trained based on the preference data of the descriptive text so that it is more inclined to generate preferred samples rather than non-preferred samples, thereby facilitating the performance optimization of the description generation model and improving the quality of the generated descriptive text.

[0341] The following explains how the training data is constructed.

[0342] As one possible approach, the training data can be constructed during the training process.

[0343] In this case, the execution process of method 600 can be regarded as two stages: the training data construction stage (i.e., step 610) and the model DPO reinforcement stage (i.e., step 620).

[0344] The training data construction phase can also be called the preference data pair generation phase.

[0345] Figure 7 This is an example of a DPO training process according to an embodiment of this application. The following is combined with... Figure 7 The scheme shown illustrates method 600 by way of example.

[0346] like Figure 7 As shown, the training process can be divided into two stages: the preference data pair generation stage and the DPO reinforcement stage.

[0347] As mentioned earlier, steps 610 and 620 in method 600 can be executed iteratively. The following explanation uses one iteration (the t-th iteration) as an example to illustrate steps 610 and 620 in the iterative process. t is a positive integer.

[0348] The captioner model (i.e., captioner model #61) in the t-th iteration is represented as M. t .

[0349] During the t-th iteration, the seed database { x i The input data (including image samples) in} x i Input to captioner model M t This yields N descriptive texts for the image sample, i.e., N responses in the image. y 1 ,y 2 ,…y N ), where i = 1, 2, ..., N are positive integers.

[0350] like Figure 7 As shown in (a), preference data pairs are constructed based on these N responses. xi ,y w ,y l ), add the constructed preference data pairs to the preference data pair library {( x i ,y w ,y l )}middle. y w Indicates a preference sample. y l This represents a non-biased sample.

[0351] Based on preference data, the captioner model M is compared. t DPO training is performed to obtain the parameter-adjusted captioner model. If the current iteration (i.e., the t-th iteration) is not the last iteration, then the parameter-adjusted model can be used as the captioner model M in the next iteration (i.e., the t+1-th iteration). t+1 ,like Figure 7 As shown.

[0352] Optionally, step 610 may include: inputting image sample #61 into captioner model #61 to obtain descriptive text #61 of image sample #61 (such as the predicted descriptive text of the fourth image sample). The non-preference sample of image sample #61 is part or all of the content of descriptive text #61.

[0353] In other words, in the preferred data pair, the non-preferred sample can be obtained based on the descriptive text output by the current captioner model.

[0354] In method 600, the current captioner model refers to the captioner model in the current iteration. For example, as... Figure 7 As shown, if we are currently in the t-th iteration of the DPO training process, then the captioner model #61 can be the captioner model M in the t-th iteration. t .

[0355] It should be understood that during training, for an image sample, the captioner model can output one or more descriptive texts for that image sample. Taking image sample #61 as an example, the captioner model can output one or more descriptive texts for image sample #61, where the descriptive text #61 can be any one of the one or more descriptive texts.

[0356] For example, such as Figure 7As shown, N can be 1 or an integer greater than 1. Non-preferred samples y l It can be part or all of the content of one of the N responses.

[0357] The number #61 in the description text is for convenience only and has no limiting effect. Any description text of image sample #61 output by the captioner model during training can be considered as description text #61.

[0358] For example, if the captioner model #61 outputs multiple descriptive texts for image sample #61, the descriptive text #61 can be any one of the multiple descriptive texts, or the descriptive text #61 can be a descriptive text determined from the multiple descriptive texts based on preset rules, or the descriptive text #61 can be manually selected from the multiple descriptive texts. This application embodiment does not limit the specific method of determining the descriptive text #61.

[0359] The following examples illustrate how to determine preference data pairs using two methods (Method 1 and Method 2).

[0360] Method 1: The preferred sample for image sample #61 can be the target description text of image sample #61.

[0361] That is, the preferred sample of image sample #61 can be the entire content of the target description text of image sample #61.

[0362] The target description text of an image sample is a complete description of what is expected of that image sample.

[0363] The target description text of an image sample can be obtained in a variety of ways.

[0364] For example, the target description text for image sample #61 can be pre-set.

[0365] Furthermore, the unpreferenced sample of image sample #61 can be the entire content of the descriptive text #61.

[0366] In other words, in the preference data pair, the preference sample can be pre-set, and the non-preference sample can be the predicted descriptive text generated by the current captioner model.

[0367] For example, for a given image, such as image sample #61, the descriptive text can be written manually, or the descriptive text can be output by the visual language model and used as the preferred sample, i.e., the preferred sample of image sample #61. During the training process, image sample #61 is input into captioner model #61, and captioner model #61 outputs the predicted descriptive text of image sample #61 (i.e., descriptive text #61), which is used as the non-preferred sample.

[0368] Alternatively, the target description text of image sample #61 can be a corrected description text #61 obtained by correcting the description text #61.

[0369] For example, the corrected description text #61 can be used as a preferred sample, and the description text #61 can be used as a non-preferred sample.

[0370] In other words, the corrected descriptive text can be used as a preferred sample, and the uncorrected descriptive text can be used as a non-preferred sample. The uncorrected descriptive text can be called the original descriptive text.

[0371] Correction of the original description text can be achieved in a variety of ways.

[0372] For example, correction can be achieved through manual error correction. Taking description text #61 as an example, the errors in description text #61 are manually corrected to obtain the corrected description text #61.

[0373] For example, correction can also be achieved using a multimodal large model. Specifically, the descriptive text #61 can be corrected by using chain-thinking prompts to the multimodal large model, resulting in the corrected descriptive text #61.

[0374] by Figure 7 For example, in (b), one response (i.e., description text #61) is selected from the N responses as the initial unbiased sample. y lose By using a multimodal large model (such as VLM) to process the initial unbiased samples y lose After correction, the corrected description text is obtained, which can be used as the initial preference sample. y win Based on the initial preference sample y win and initial unbiased samples y lose Construct preference data pairs ( x i ,y w ,y l For example, the initial preference sample.y win and initial unbiased samples y lose They can be directly used as preference samples. y w Non-preference samples y l .

[0375] Figure 8 An example describing a text correction process is shown.

[0376] After generating descriptive text (e.g., descriptive text #61) for an image sample (e.g., image sample #61) using a captioner model (e.g., captioner model #61), this descriptive text is corrected. This descriptive text can be referred to as the original descriptive text.

[0377] like Figure 8 As shown, the calibration process may include the following steps.

[0378] Step 1: Analyze and extract the content of the image sample.

[0379] For example, step two can be accomplished using a model capable of extracting image content. Alternatively, step two can also be performed manually.

[0380] Step two: Extract factual statements from the content of the image samples.

[0381] In step two, modifying statements can be removed to improve the quality of subsequent comparison results.

[0382] exist Figure 8 This example only uses extracting factual statements from the content of an image sample. In other implementations, factual statements can also be extracted from the original descriptive text, or they can be extracted from both the content of the image sample and the original descriptive text.

[0383] Step two is optional.

[0384] Step 3: Compare the original descriptive text with the factual statements to identify errors in the original descriptive text.

[0385] As mentioned earlier, step two is optional. If step two is not included in the process, step three can also compare the content of the original descriptive text and the image sample to identify errors in the original descriptive text. If factual statements were extracted from the original descriptive text in step two, the original descriptive text in step three can also be replaced with factual statements extracted from the original descriptive text.

[0386] In step three, inconsistencies between the original descriptive text and the image sample can be identified and marked. For example, such as... Figure 8 As shown, inconsistent content indicates incorrect attributes.

[0387] Step four: Correct the errors in the original description text.

[0388] For example, errors in the original description text can be corrected based on the content marked in step three.

[0389] For example, the description text obtained through this process can be used as a preference sample, and the original description text can also be used as a preference sample.

[0390] It should be understood that the above are merely examples, and the correction of the original descriptive text can also be achieved in other ways. This application does not limit this approach.

[0391] Method 2: The preferred sample of image sample #61 can be a portion of the target description text of image sample #61. The non-preferred sample of image sample #61 can be a portion of the description text of image sample #61.

[0392] Optionally, the non-preference sample and the preference sample are the inconsistent content in the description text #61 and the target description text, respectively.

[0393] That is, non-preference samples are the content in description text #61 that is inconsistent with the target description text, and preference samples are the content in the target description text that is inconsistent with description text #61.

[0394] In other words, the description text generated by the captioner and the target description text can be split into fine-grained parts, and inconsistent content can be identified from the split content and used as non-preference samples and preference samples, respectively.

[0395] like Figure 7 As shown in (b), based on the initial preference sample y win and initial unbiased samples y lose Construct preference data pairs ( x i ,y w ,y l For example, preference samples y w Non-preference samples y l These can be the initial preference samples. y win and initial unbiased samples y lose Inconsistent content.

[0396] For example, fine-grained splitting can be achieved using a large model.

[0397] Figure 9 This diagram illustrates a fine-grained splitting process. After fine-grained splitting, non-preference samples and preference samples target content with incorrect attributes, i.e., erroneous text, while content that matches in the two complete descriptive texts (the descriptive text generated by the captioner and the target descriptive text) is excluded, i.e., correct text is excluded.

[0398] Optionally, the target description text for image sample #61 can be the corrected description text #61.

[0399] For example, such as Figure 9 As shown, the target description text is obtained through... Figure 8 The process shown is to correct the description text generated by the captioner, which is the modified description text shown in the figure.

[0400] In this case, the non-preference sample and the preference sample represent the inconsistent content in description text #61 and the corrected description text #61, respectively.

[0401] That is, non-preference samples are the content in description text #61 that is inconsistent with the corrected description text #61, and preference samples are the content in the corrected description text #61 that is inconsistent with description text #61.

[0402] The above is just an example of target description text. For a more detailed description of the target description text, please refer to Method 1. It will not be repeated here.

[0403] In the scheme of this application embodiment, two complete descriptive texts of an image, such as the descriptive text #61 before and after correction, can be finely split. Inconsistent content can be identified from the split content and used as non-preference samples and preference samples, respectively. The preference data pairs obtained in this way are of higher quality, which helps the captioner model to specifically strengthen the erroneous text, eliminate the interference of correct text on DPO optimization, and thus further improve the generation quality of the captioner model.

[0404] As another possible implementation, the training data can be pre-built.

[0405] In this approach, a training dataset can be pre-built, which may include multiple preference data pairs, i.e., multiple training data sets. This training dataset can be used to perform DPO training on the captioner model #61.

[0406] Training data can be constructed in a variety of ways.

[0407] For example, given an image and two descriptive texts for that image, preferred and / or unpreferred samples can be manually labeled.

[0408] Alternatively, you can refer to the online construction method described above to construct training data, and replace the captioner model #61 used to generate unbiased samples with other captioner models.

[0409] Figure 6 The method 600 shown can be used with Figure 2 The methods shown in 200 are used in combination.

[0410] As an example, the captioner model #61 in method 600 and the captioner model #21 in method 200 can be the same model. That is, method 200 is applied during the training phase, corresponding to scenario 4 in method 200. In other words, the structure of the captioner model during the DPO training process is the same as the structure of the captioner model in method 200. In this case, the image sample #61 in method 600 is the image to be processed in method 200, and correspondingly, the input of the captioner model #61 also includes auxiliary information of this image sample #61.

[0411] Optionally, method 600 may further include: acquiring world knowledge information and / or visual aid information of image sample #61, wherein the visual aid information of image sample #61 is obtained by visual processing of image sample #61.

[0412] Image sample #61 is input into captioner model #61 to generate descriptive text #61 for image sample #61, including: Image sample #61 and its auxiliary information are input into captioner model #61 to generate descriptive text #61 for image sample #61. The auxiliary information for image sample #61 includes world knowledge information and / or visual auxiliary information of image sample #61.

[0413] Optionally, the visual aid information of image sample #61 includes at least one of the following: the label of image sample #61, the location information of the target in image sample #61, or the OCR information of image sample #61.

[0414] For a detailed description of the auxiliary information for image sample #61, please refer to the description in method 200. Simply replace the image to be processed in method 200 with image sample #61.

[0415] As another example, the model trained by method 600 can be the same model as the captioner model #21 in method 200. In other words, the model trained by method 600 can be used as the model in the inference phase of method 200.

[0416] In this case, the captioner model #61 in method 600 and the captioner model #22 in method 200 can be the same model, the image sample #61 in method 600 can be the image sample #21 in method 200, and the dataset #61 in method 600 can be the dataset #21 in method 200.

[0417] Figure 6 The method 600 shown can be used with Figure 3 The method shown in 300 is used in combination.

[0418] As an example, the captioner model #61 in method 600 can be trained on hard examples. In other words, the captioner model trained on hard examples can serve as the base model at the start of DPO training. DPO training is then used as post-training.

[0419] Optionally, captioner model #61 is obtained by training captioner model #62 (i.e., the fifth description generation model) on dataset #62 (i.e., the fourth dataset). Image sample #62 (i.e., the fifth image sample) in dataset #62 is a hard example. The hard example status of image sample #62 is determined based on the predicted description text of image sample #62, which is generated by captioner model #62.

[0420] The numbering of dataset #62, captioner model #62, and image sample #62 is for descriptive convenience only and has no limiting effect.

[0421] Dataset #62 and Dataset #31 in Method 300 can be the same dataset, Captioner Model #62 and Captioner Model #31 in Method 300 can be the same model, and Image Sample #62 and Image Sample #31 in Method 300 can be the same image sample.

[0422] Optionally, the determination that image sample #62 is a hard example is based on the predicted description text of image sample #62 and a quality control strategy used to determine whether an image is a hard example.

[0423] Optionally, image sample #63 (i.e., the sixth image sample) in dataset #62 is retrieved based on the category of image sample #62 and / or the predicted descriptive text of image sample #62.

[0424] The image sample number #63 is for descriptive convenience only and has no limiting effect.

[0425] Image sample #63 and image sample #32 in method 300 can be the same image sample.

[0426] As another example, the seed database in method 600 may include dataset #31 from method 300. For instance, the seed database may be a training dataset optimized with hard examples.

[0427] Figure 6 Method 600 shown Figure 2 Method 200 and shown Figure 3 The methods shown in 300 can be used in combination.

[0428] As an example, the captioner model trained on hard examples serves as the base model at the start of DPO training. Throughout the training process, the structure of the captioner model follows the structure of the captioner model in method 200.

[0429] It should be understood that the above combinations are merely examples and do not constitute a limitation on the solutions of the embodiments of this application.

[0430] Figure 10 This application provides a system architecture 1000. Figure 10 In this embodiment, the data acquisition device 1060 is used to acquire training data. For example, in the method for generating descriptive text according to this application embodiment, the training data may include image samples, and further, the training data may also include preferred and unpreferred samples of the image samples.

[0431] After collecting the training data, the data acquisition device 1060 stores the training data in the database 1030, and the training device 1020 trains the target model / rule 1001 based on the training data maintained in the database 1030.

[0432] The following describes how the training device 1020 obtains the target model / rule 1001 based on the training data. The training device 1020 processes the input raw data and compares the output value with the target value until the difference between the output value of the training device 1020 and the target value is less than a certain threshold, thereby completing the training of the target model / rule 1001.

[0433] The aforementioned target model / rule 1001 can be used to implement the method for generating descriptive text in the embodiments of this application. Specifically, the target model / rule 1001 in the embodiments of this application can be a neural network model. It should be noted that in practical applications, the training data maintained in the database 1030 may not all come from the data acquisition device 1060; it may also be received from other devices. Furthermore, it should be noted that the training device 1020 may not necessarily train the target model / rule 1001 entirely based on the training data maintained in the database 1030; it may also obtain training data from the cloud or other sources for model training. The above description should not be construed as limiting the embodiments of this application.

[0434] The target model / rule 1001 trained using training device 1020 can be applied to different systems or devices, such as... Figure 10 The execution device 1010 shown can be a terminal, such as a mobile phone terminal, tablet computer, laptop computer, augmented reality (AR) / virtual reality (VR) device, vehicle terminal, etc., or it can be a server or cloud service. Figure 10 In this embodiment, the execution device 1010 is configured with an input / output (I / O) interface 1012 for data interaction with external devices. Users can input data to the I / O interface 1012 through the client device 1040. The input data may include images input by the client device.

[0435] During the preprocessing of input data by the execution device 1010, for example, by preprocessing through the preprocessing module 1013 and / or the preprocessing module 1014, or during the calculation process performed by the calculation module 1011 of the execution device 1010, the execution device 1010 may call data, code, etc. in the data storage system 1050 for corresponding processing, or store the data, instructions, etc. obtained from the corresponding processing into the data storage system 1050.

[0436] Finally, the I / O interface 1012 returns the processing result, such as the description text obtained above, to the client device 1040, thereby providing it to the user.

[0437] It is worth noting that the training device 1020 can generate corresponding target models / rules 1001 based on different training data for different objectives or tasks. The corresponding target models / rules 1001 can be used to achieve the above objectives or complete the above tasks, thereby providing the user with the required results.

[0438] exist Figure 10In the scenario shown, the user can manually provide input data, which can be done through the interface provided by I / O interface 1012. Alternatively, the client device 1040 can automatically send input data to I / O interface 1012. If user authorization is required for the client device 1040 to automatically send input data, the user can set the corresponding permissions in the client device 1040. The user can view the output results of the execution device 1010 on the client device 1040, which can be presented in various ways such as display, sound, or animation. The client device 1040 can also act as a data acquisition terminal, collecting the input data and output results of the input I / O interface 1012 as new sample data and storing them in database 1030. Alternatively, data can be collected directly from the I / O interface 1012 without going through the client device 1040, using the input data and output results of the input I / O interface 1012 as new sample data and storing them in database 1030.

[0439] It is worth noting that, Figure 10 This is merely a schematic diagram of a system architecture provided in an embodiment of this application. The positional relationships between the devices, components, modules, etc., shown in the diagram do not constitute any limitation. For example, in Figure 10 In this context, the data storage system 1050 is an external memory relative to the execution device 1010. In other cases, the data storage system 1050 may also be placed within the execution device 1010.

[0440] The following is combined with Figures 11 to 12 The apparatus of the embodiments of this application will be described below. It should be understood that the apparatus described below is capable of performing the methods of the foregoing embodiments of this application. To avoid unnecessary repetition, repeated descriptions will be appropriately omitted when describing the apparatus of the embodiments of this application below.

[0441] Figure 11 A schematic block diagram of the apparatus provided in an embodiment of this application is shown. Figure 11 The apparatus 1800 shown can be used to perform the methods of the embodiments of this application, for example, Figure 2 , Figure 3 or Figure 6 The method shown.

[0442] like Figure 11 As shown, the device 1800 may include a first acquisition module 1810, a second acquisition module 1820, and a processing module 1830. The first acquisition module 1810 and the second acquisition module 1820 may be the same acquisition module or different acquisition modules.

[0443] As one possible implementation, device 1800 can be used to perform Figure 2 The method shown.

[0444] The first acquisition module 1810 is used to acquire the image to be processed.

[0445] The second acquisition module 1820 is used to acquire world knowledge information of the image to be processed.

[0446] The processing module 1830 is used to input the image to be processed and the world knowledge information of the image to be processed into the first description generation model to generate descriptive text for the image to be processed.

[0447] Optionally, the device 1800 further includes a third acquisition module (not shown in the figure), for: The visual auxiliary information of the image to be processed is obtained by performing visual processing on the image; and the processing module 1830 is specifically used for: The image to be processed, its world knowledge information, and its visual auxiliary information are input into the first description generation model to generate descriptive text for the image to be processed.

[0448] The third acquisition module and the second acquisition module 1820 can be the same acquisition module or different acquisition modules.

[0449] The third acquisition module and the first acquisition module 1810 can be the same acquisition module or different acquisition modules.

[0450] Optionally, the visual aid information of the image to be processed includes at least one of the following: the label of the image to be processed, the location information of the target in the image to be processed, or the OCR information of the image to be processed.

[0451] Optionally, the first description generation model is obtained by training the second description generation model based on the first dataset. The training data in the first dataset includes the first image sample, the preferred sample of the first image sample, and the non-preferred sample of the first image sample. During the training process, the parameters of the second description generation model are adjusted so that the probability of the second description generation model generating the preferred sample of the first image sample is higher than the probability of the second description generation model generating the non-preferred sample of the first image sample. The preferred sample and the non-preferred sample of the first image sample are two descriptive texts of the first image sample.

[0452] Optionally, the non-preference sample of the first image sample is the content in the predicted description text of the first image sample that is inconsistent with the target description text of the first image sample, and the preference sample of the first image sample is the content in the target description text of the first image sample that is inconsistent with the predicted description text of the first image sample. The predicted description text of the first image sample is generated by the second description generation model.

[0453] Optionally, the target description text of the first image sample is obtained by correcting the predicted description text of the first image sample.

[0454] Optionally, the first description generation model is obtained by training the third description generation model on the second dataset. The second image sample in the second dataset is a hard example. The second image sample is a hard example, which is determined by the predicted description text of the second image sample. The predicted description text of the second image sample is generated by the third description generation model.

[0455] Optionally, the determination of whether a second image sample is a hard example is based on the predicted description text of the second image sample and a quality control strategy, which is used to determine whether an image sample is a hard example.

[0456] Optionally, the third image sample in the second dataset is retrieved based on the category of the second image sample and / or the predicted descriptive text of the second image sample.

[0457] For a detailed description, please refer to Method 200 above; it will not be repeated here.

[0458] As another possible implementation, device 1800 can be used to perform... Figure 8 The method shown.

[0459] Specifically, the first acquisition module 1810 is used to: acquire a third dataset, wherein the training data in the third dataset includes a fourth image sample, a preferred sample of the fourth image sample, and a non-preferred sample of the fourth image sample, wherein the preferred sample of the fourth image sample and the non-preferred sample of the fourth image sample are two descriptive texts of the fourth image sample; The processing module 1830 is used to: train the fourth description generation model by adjusting the training data, and during the training process, adjust the parameters of the fourth description generation model so that the probability of the fourth description generation model generating preferred samples is higher than the probability of the fourth description generation model generating unpreferred samples.

[0460] Optionally, the processing module 1830 is further configured to: input the fourth image sample into the fourth description generation model to generate a predicted description text for the fourth image sample, wherein the non-preference sample and the preference sample of the fourth image sample are inconsistent contents in the predicted description text and the target description text of the fourth image sample, respectively.

[0461] Optionally, the target description text of the fourth image sample is obtained by correcting the predicted description text of the fourth image sample.

[0462] Optionally, the second acquisition module 1820 is configured to: acquire world knowledge information and / or visual auxiliary information of the fourth image sample, wherein the visual auxiliary information of the fourth image sample is obtained by visual processing of the fourth image sample; and the processing module 1830 is configured to: input the fourth image sample and the auxiliary information of the fourth image sample into the fourth description generation model to generate a predicted description text of the fourth image sample, wherein the auxiliary information of the fourth image sample includes world knowledge information and / or visual auxiliary information of the fourth image sample.

[0463] Optionally, the visual aid information of the fourth image sample includes at least one of the following: the label of the fourth image sample, the location information of the target in the fourth image sample, or the OCR information of the fourth image sample.

[0464] Optionally, the fourth description generation model is obtained by training the fifth description generation model on the fourth dataset. The fifth image sample in the fourth dataset is a hard example. The fact that the fifth image sample is a hard example is determined based on the predicted description text of the fifth image sample. The predicted description text of the fifth image sample is generated by the fifth description generation model.

[0465] Optionally, the determination of whether the fifth image sample is a hard example is based on the predicted description text of the fifth image sample and a quality control strategy, which is used to determine whether an image is a hard example.

[0466] Optionally, the sixth image sample in the fourth dataset is retrieved based on the category of the fifth image sample and / or the predicted descriptive text of the fifth image sample.

[0467] For a detailed description, please refer to Method 600 above; it will not be repeated here.

[0468] Each module in device 1800 can be implemented in software or in hardware. For example, the implementation of processing module 1830 will be described below. Similarly, the implementation methods of other modules can be the same as those of processing module 1830.

[0469] As an example of a software functional unit, processing module 1830 may include code running on a computing instance. The computing instance may include at least one of a physical host (computing device), a virtual machine, or a container. Further, the aforementioned computing instance may be one or more. For example, processing module 1830 may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code may be distributed within the same region or in different regions. Further, the multiple hosts / virtual machines / containers used to run the code may be distributed within the same availability zone (AZ) or in different AZs, each AZ including one or more geographically proximate data centers. Typically, a region may include multiple AZs.

[0470] Similarly, multiple hosts / virtual machines / containers used to run this code can be distributed within the same Virtual Private Cloud (VPC) or across multiple VPCs. Typically, a VPC is set up within a region. Communication between two VPCs within the same region, as well as between VPCs in different regions, requires a communication gateway to be set up within each VPC to enable interconnection between VPCs.

[0471] As an example of a hardware functional unit, the processing module 1830 may include at least one computing device, such as a server. Alternatively, the processing module 1830 may also be a device implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be implemented using a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), generic array logic (GAL), or any combination thereof.

[0472] The processing module 1830 includes multiple computing devices that can be distributed within the same region or in different regions. These computing devices can be distributed within the same Availability Zone (AZ), within the same Virtual Private Cloud (VPC), or across multiple VPCs. Furthermore, the multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.

[0473] It should be noted that the division of units in the above device is only a logical functional division. In actual implementation, they can be fully or partially integrated into a single physical entity, or they can be physically separated. In other embodiments, the processing module 1830 can be used to execute any step in the method described above, and other modules can be used to implement any step described above. The steps that each module is responsible for implementing can be specified as needed. By having each module implement different steps described above, all functions of the device 1800 can be achieved.

[0474] Figure 12 This is a schematic block diagram of the apparatus provided in an embodiment of this application. The apparatus 1900 may include a processor 1910, a transceiver 1920, and a memory 1930. The processor 1910, transceiver 1920, and memory 1930 are connected via internal interconnection paths. The memory 1930 is used to store instructions, and the processor 1910 is used to execute the instructions stored in the memory 1930 to receive / send data via the transceiver 1920. Optionally, the memory 1930 may be coupled to the processor 1910 via an interface or integrated with the processor 1910.

[0475] It should be noted that the transceiver 1920 mentioned above may include, but is not limited to, transceiver devices such as input / output interfaces, to enable communication between device 1900 and other devices or communication networks.

[0476] The memory 1930 can be a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM).

[0477] In one implementation, the processor 1910 can be a circuit with instruction read and execute capabilities, such as a central processing unit (CPU), microprocessor, or digital signal processor (DSP). In another implementation, the processor 1910 can implement certain functions through the logical relationships of hardware circuits. These logical relationships can be fixed or reconfigurable. For example, the processor 1910 can be a hardware circuit implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD), such as a field-programmable gate array (FPGA). In a reconfigurable hardware circuit, the process of the processor loading a configuration document and configuring the hardware circuit can be understood as the processor loading instructions to implement some or all of the functions of the aforementioned units.

[0478] This application also provides an electronic device, which may include the above-described device 1800 or device 1900.

[0479] This application also provides a computing device cluster, which includes at least one computing device. The computing device may be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device may also be a terminal device such as a desktop computer, a laptop computer, or a smartphone. The computing device cluster includes at least one device 1900.

[0480] In some possible implementations, the memory 1930 of one or more computing devices in the computing device cluster may also store partial instructions for executing the aforementioned method. In other words, a combination of one or more computing devices can jointly execute the instructions for executing the aforementioned method.

[0481] It should be noted that the memory 1930 in different computing devices within the computing device cluster can store different instructions, each used to execute a portion of the steps described above. That is, the instructions stored in the memory 1930 of different computing devices can implement the functions of one or more modules in device 1900.

[0482] In some possible implementations, one or more computing devices in a computing device cluster can be connected via a network. This network can be a wide area network (WAN) or a local area network (LAN), etc.

[0483] This application also provides a computer program product, which includes computer program code that, when run on a computer, causes the computer to perform the methods described in the above embodiments.

[0484] This application also provides a computer-readable medium storing program code that, when run on a computer, causes the computer to perform the methods described in the above embodiments.

[0485] This application also provides a chip, including circuitry, for performing the methods described in the above embodiments.

[0486] In implementation, each step of the above method can be completed by integrated logic circuits in the processor's hardware or by instructions in software. The method disclosed in the embodiments of this application can be directly implemented by a hardware processor, or by a combination of hardware and software modules within the processor. The software modules can reside in random access memory, flash memory, read-only memory, programmable read-only memory, power-on erasable programmable memory, registers, or other mature storage media in the art. This storage medium is located in memory, and the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above method. To avoid repetition, detailed descriptions are omitted here.

[0487] It should also be understood that, in the various embodiments of this application, the order of the above-mentioned processes does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0488] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0489] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0490] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0491] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0492] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, ROM, RAM, magnetic disks, or optical disks.

[0493] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be covered.

Claims

1. A method for generating descriptive text, characterized in that, include: Obtain the image to be processed; Obtain world knowledge information of the image to be processed; The image to be processed and its world knowledge information are input into the first description generation model to generate descriptive text for the image to be processed.

2. The method according to claim 1, characterized in that, The method further includes: Obtaining visual auxiliary information of the image to be processed, wherein the visual auxiliary information of the image to be processed is obtained by visual processing of the image to be processed; and inputting the image to be processed and the world knowledge information of the image to be processed into a first description generation model to generate descriptive text of the image to be processed, including: The image to be processed, the world knowledge information of the image to be processed, and the visual auxiliary information of the image to be processed are input into the first description generation model to generate a description text for the image to be processed.

3. The method according to claim 2, characterized in that, The visual aid information of the image to be processed includes at least one of the following: the label of the image to be processed, the location information of the target in the image to be processed, or the OCR information of the image to be processed.

4. The method according to claim 3, characterized in that, The visual aid information of the image to be processed also includes information on the number of targets in the image.

5. The method according to any one of claims 1 to 4, characterized in that, The first description generation model is obtained by training the second description generation model on the first dataset. The training data in the first dataset includes a first image sample, a preferred sample of the first image sample, and a non-preferred sample of the first image sample. During the training process, the parameters of the second description generation model are adjusted so that the probability of the second description generation model generating the preferred sample of the first image sample is higher than the probability of the second description generation model generating the non-preferred sample of the first image sample. The preferred sample and the non-preferred sample of the first image sample are two descriptive texts of the first image sample.

6. The method according to claim 5, characterized in that, The non-preference sample of the first image sample is the content in the predicted description text of the first image sample that is inconsistent with the target description text of the first image sample. The preference sample of the first image sample is the content in the target description text of the first image sample that is inconsistent with the predicted description text of the first image sample. The predicted description text of the first image sample is generated by the second description generation model.

7. The method according to claim 6, characterized in that, The target description text of the first image sample is obtained by correcting the predicted description text of the first image sample.

8. The method according to any one of claims 1 to 7, characterized in that, The first description generation model is obtained by training the third description generation model on the second dataset. The second image sample in the second dataset is a hard example. The second image sample is a hard example, which is determined by the predicted description text of the second image sample. The predicted description text of the second image sample is generated by the third description generation model.

9. The method according to claim 8, characterized in that, The determination that the second image sample is a difficult example is based on the predicted description text of the second image sample and the quality inspection strategy, which is used to determine whether an image sample is a difficult example.

10. The method according to claim 8 or 9, characterized in that, The third image sample in the second dataset is retrieved based on the category of the second image sample and / or the predicted descriptive text of the second image sample.

11. An apparatus for generating descriptive text, characterized in that, include: The first acquisition module is used to acquire the image to be processed; The second acquisition module is used to acquire world knowledge information of the image to be processed; The processing module is used to input the image to be processed and the world knowledge information of the image to be processed into the first description generation model to generate a description text for the image to be processed.

12. The apparatus according to claim 11, characterized in that, The device further includes a third acquisition module, used for: The visual aid information of the image to be processed is obtained by performing visual processing on the image to be processed; and the processing module is specifically used for: The image to be processed, the world knowledge information of the image to be processed, and the visual auxiliary information of the image to be processed are input into the first description generation model to generate a description text for the image to be processed.

13. The apparatus according to claim 12, characterized in that, The visual aid information of the image to be processed includes at least one of the following: the label of the image to be processed, the location information of the target in the image to be processed, or the OCR information of the image to be processed.

14. The apparatus according to claim 13, characterized in that, The visual aid information of the image to be processed also includes information on the number of targets in the image.

15. The apparatus according to any one of claims 11 to 13, characterized in that, The first description generation model is obtained by training the second description generation model on the first dataset. The training data in the first dataset includes a first image sample, a preferred sample of the first image sample, and a non-preferred sample of the first image sample. During the training process, the parameters of the second description generation model are adjusted so that the probability of the second description generation model generating the preferred sample of the first image sample is higher than the probability of the second description generation model generating the non-preferred sample of the first image sample. The preferred sample and the non-preferred sample of the first image sample are two descriptive texts of the first image sample.

16. The apparatus according to claim 15, characterized in that, The non-preference sample of the first image sample is the content in the predicted description text of the first image sample that is inconsistent with the target description text of the first image sample. The preference sample of the first image sample is the content in the target description text of the first image sample that is inconsistent with the predicted description text of the first image sample. The predicted description text of the first image sample is generated by the second description generation model.

17. The apparatus according to claim 16, characterized in that, The target description text of the first image sample is obtained by correcting the predicted description text of the first image sample.

18. The apparatus according to any one of claims 11 to 17, characterized in that, The first description generation model is obtained by training the third description generation model on the second dataset. The second image sample in the second dataset is a hard example. The second image sample is a hard example, which is determined by the predicted description text of the second image sample. The predicted description text of the second image sample is generated by the third description generation model.

19. The apparatus according to claim 18, characterized in that, The determination that the second image sample is a difficult example is based on the predicted description text of the second image sample and the quality inspection strategy, which is used to determine whether an image sample is a difficult example.

20. The apparatus according to claim 18 or 19, characterized in that, The third image sample in the second dataset is retrieved based on the category of the second image sample and / or the predicted descriptive text of the second image sample.

21. A computing device, characterized in that, It includes a processor and a memory, the processor being configured to execute instructions stored in the memory to cause the computing device to perform the method as described in any one of claims 1 to 10.

22. A computer program product containing instructions, characterized in that, When the instructions are executed by the computing device, the computing device performs the method as described in any one of claims 1 to 10.

23. A computer-readable storage medium, characterized in that, It includes computer program instructions, which, when executed by a computing device, cause the computing device to perform the method as described in any one of claims 1 to 10.

Images