Data processing method, and training method and device of image text data quality enhancement model
Patent Information
- Application Number
- CN202411471900.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-21
- Publication Date
- 2025-02-21
Smart Images
Figure CN119495001A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of image processing technology, and specifically to a data processing method, a training method and a device for an image text data quality enhancement model. Background Art
[0002] Multimodal Large Language Model (MLLM) has made significant progress by integrating visual and textual modes. A key factor in training MLLM models is the quality of images and description information in multimodal pre-training datasets. However, due to the lack of semantic alignment between images and description information, in order to improve the accuracy of MLLM models, data with a high degree of match between description information and images are usually selected from images and their description information for training MLLM models. For example, data with a high degree of match between description information and images can be selected from multiple pairs of images and their description information through filters. The above process usually causes a large amount of high-quality image data to be filtered out due to inaccurate description information, resulting in low data utilization. Summary of the invention
[0003] In view of this, the present application provides a data processing method, an image description model training method and a device to help solve the problem of low data utilization in the prior art.
[0004] In a first aspect, an embodiment of the present application provides a data processing method, comprising: acquiring an image and initial description information of the image;
[0005] The image and the initial description information are input into an image text data quality enhancement model to obtain an output result of the image text data quality enhancement model; the image text data quality enhancement model is used to match the image and the initial description information to obtain evaluation information, and when the evaluation information indicates that the initial description information has a low matching degree with the image, the initial description information is corrected based on the image to obtain enhanced description information of the image, and the enhanced description information and the evaluation information are used as the output result; when the evaluation information indicates that the initial description information has a high matching degree with the image, the evaluation information is used as the output result; based on the output result of the image text data quality enhancement model, the training data of a multimodal large language model (MLLM) model is determined; wherein, when the output result is the evaluation information, the image and the initial description information of the image are determined as the training data of the MLLM model; when the output result includes the evaluation information and the enhanced description information of the image, the image and the enhanced description information of the image are determined as the training data of the MLLM model.
[0006] In a possible implementation manner of the first aspect, the image text data quality model is composed of a visual encoder, a multi-layer perceptron and a large language model.
[0007] In a possible implementation of the first aspect, the evaluation information includes overall information of the image and the description information, quality information of the description information, matching information between the image and the description information, object detailed information, semantic understanding information, description information, or at least one of chart description information.
[0008] In a possible implementation manner of the first aspect, the image text data quality enhancement model matches the image and the initial description information to obtain the evaluation information in the following manner:
[0009] The image text data quality enhancement model extracts feature information of the image and feature information of the initial description information, and matches the feature information of the image and feature information of the initial description information to obtain the evaluation information.
[0010] In a possible implementation of the first aspect, the image text data quality enhancement model modifies the initial description information based on the image in the following manner to obtain enhanced description information of the image, and uses the enhanced description information and the evaluation information as the output result:
[0011] The image text data quality enhancement model modifies the initial description information based on the feature information of the image to obtain enhanced description information of the image, wherein the enhanced description information of the image describes the feature information of the image;
[0012] The enhanced description information and the evaluation information are used as the output result.
[0013] In a possible implementation of the first aspect, the image text data quality enhancement model is a model trained based on an initial sample and a comparison sample; the image in the comparison sample is the same as the image in the initial sample, and the matching degree between the description information and the image in the comparison sample is different from the matching degree between the description information and the image in the initial sample; wherein, when the matching degree between the image and the description information in the initial sample is high, the matching degree between the description information and the image in the comparison sample is low, and when the matching degree between the image and the description information in the initial sample is low, the matching degree between the description information in the comparison sample is high.
[0014] In a second aspect, an embodiment of the present application provides a method for training an image text data quality enhancement model, comprising: obtaining a first-stage training sample;
[0015] The initial model is trained based on the first-stage training samples to update the initial model to an image-text feature alignment model; the image-text feature alignment model is used to identify feature information of the image and feature information of the description information in the input information, and match the feature information of the image with the feature information of the description information;
[0016] Acquire a second-stage training sample, wherein the second-stage training sample includes an initial sample and a comparison sample; the image in the comparison sample is the same as the image in the initial sample, and the matching degree between the description information in the comparison sample and the image is different from the matching degree between the description information in the initial sample and the image; wherein, when the matching degree between the image and the description information in the initial sample is high, the matching degree between the description information in the comparison sample and the image is low, and when the matching degree between the image and the description information in the initial sample is low, the matching degree between the description information in the comparison sample and the image is high;
[0017] The image-text feature alignment model is trained based on the second-stage training samples to update the image-text feature alignment model to an image-text data quality enhancement model, so that when the degree of match between the input image and the initial description information is low, the image-text data quality enhancement model corrects the initial description information based on the image to obtain enhanced description information of the image with a high degree of match with the image, so that the image and the enhanced description information of the image are determined as training data for the MLLM model.
[0018] In a possible implementation manner of the second aspect, obtaining the second-stage training sample includes:
[0019] Acquire an initial sample; the initial sample includes an image, description information of the image, and evaluation information of the description information;
[0020] A comparison sample is obtained based on the image in the initial sample, the description information of the image, and the evaluation information of the description information.
[0021] In a third aspect, an embodiment of the present application provides a data processing device, including: an acquisition unit, configured to acquire an image and initial description information of the image;
[0022] a processing unit, configured to input the image and the initial description information into an image text data quality enhancement model to obtain an output result of the image text data quality enhancement model; the image text data quality enhancement model is configured to match the image and the initial description information to obtain evaluation information, and when the evaluation information indicates that the initial description information has a low matching degree with the image, the initial description information is corrected based on the image to obtain enhanced description information of the image, and the enhanced description information and the evaluation information are used as the output result; when the evaluation information indicates that the initial description information has a high matching degree with the image, the evaluation information is used as the output result;
[0023] The processing unit is also used to determine the training data of the multimodal large language model (MLLM) model based on the output result of the image text data quality enhancement model; wherein, when the output result is the evaluation information, the image and the initial description information of the image are determined as the training data of the MLLM model; when the output result includes the evaluation information and the enhanced description information of the image, the image and the enhanced description information of the image are determined as the training data of the MLLM model.
[0024] In a fourth aspect, an embodiment of the present application provides a training device for an image text data quality enhancement model, comprising:
[0025] An acquisition unit, used to acquire training samples in the first stage;
[0026] A processing unit, configured to train the initial model based on the first-stage training samples to update the initial model to an image-text feature alignment model; the image-text feature alignment model is configured to identify feature information of the image and feature information of the description information in the input information, and to align the feature information of the image with the feature information of the description information;
[0027] The acquisition unit is further used to acquire a second-stage training sample, wherein the second-stage training sample includes an initial sample and a comparison sample; the image in the comparison sample is the same as the image in the initial sample, and the matching degree between the description information in the comparison sample and the image is different from the matching degree between the description information in the initial sample and the image; wherein, when the matching degree between the image and the description information in the initial sample is high, the matching degree between the description information in the comparison sample and the image is low, and when the matching degree between the image and the description information in the initial sample is low, the matching degree between the description information in the comparison sample and the image is high;
[0028] The processing unit is also used to train the image-text feature alignment model based on the second-stage training samples to update the image-text feature alignment model to an image-text data quality enhancement model, so that when the image-text data quality enhancement model has a low matching degree between the input image and the initial description information, the initial description information is corrected based on the image to obtain enhanced description information of the image with a high matching degree with the image, so as to determine the image and the enhanced description information of the image as training data of the MLLM model.
[0029] In a fifth aspect, an embodiment of the present application provides an electronic device, comprising a memory for storing computer program instructions and a processor for executing the program instructions, wherein when the computer program instructions are executed by the processor, the electronic device is triggered to execute the method described in any one of the first aspect or the method described in any one of the second aspect.
[0030] In a sixth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium includes a stored program, wherein when the program is running, the device where the computer-readable storage medium is located is controlled to execute any method described in the first aspect or any method described in the second aspect.
[0031] Adopt the scheme provided by the embodiment of the present application, obtain the image and the initial description information of the image; input the image and the initial description information into the image text data quality enhancement model to obtain the output result of the image text data quality enhancement model; wherein the image text data quality enhancement model is used to match the image and the initial description information to obtain the evaluation information, and when the evaluation information indicates that the initial description information has a low matching degree with the image, the initial description information is corrected based on the image to obtain the enhanced description information of the image, and the enhanced description information and the evaluation information are used as the output results. When the evaluation information indicates that the description information has a high matching degree with the image, the evaluation information is used as the output result; based on the output result of the image text data quality enhancement model, the training data of the MLLM model is determined. wherein, when the output result is the evaluation information, the image and the initial description information of the image are determined as the training data of the MLLM model; when the output result is the evaluation information and the enhanced description information of the image, the image and the enhanced description information of the image are determined as the training data of the MLLM model. In this way, in the embodiment of the present application, when the degree of match between the image and the initial description information is low, the enhanced description information of the image can be regenerated by the image text data quality enhancement model to improve the degree of match between the image and the description information. In this way, when the output result of the image text data quality enhancement model is evaluation information, it means that the degree of match between the image and the initial description information is high, and the image and the initial description information can be directly used as training data for the MLLM model. When the output result of the image text data quality enhancement model is evaluation information and enhanced description information of the image, it means that the degree of match between the image and the initial description information is low, and the image text data quality enhancement model generates enhanced description information with a high degree of match with the image. At this time, the image and the enhanced description information of the image can be used as training data for the MLLM model, so that high-quality images will not be filtered out due to low matching with their initial description information, thereby improving the utilization rate of the image and reducing the training complexity of other models. BRIEF DESCRIPTION OF THE DRAWINGS
[0032] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.
[0033] Figure 1 A flowchart of a data processing method provided in an embodiment of the present application;
[0034] Figure 2a A schematic diagram of a scenario of a data processing method provided in an embodiment of the present application;
[0035] Figure 2bA schematic diagram of a scenario of a data processing method provided in an embodiment of the present application;
[0036] Figure 3 A flowchart of a method for training an image text data quality enhancement model provided in an embodiment of the present application;
[0037] Figure 4 A schematic diagram of a scenario of a method for training an image text data quality enhancement model provided in an embodiment of the present application;
[0038] Figure 5 A schematic diagram of another scenario of a training method for an image text data quality enhancement model provided in an embodiment of the present application;
[0039] Figure 6 A schematic diagram of another scenario of a training method for an image text data quality enhancement model provided in an embodiment of the present application;
[0040] Figure 7 A schematic diagram of the structure of a data processing device provided in an embodiment of the present application;
[0041] Figure 8 A schematic diagram of the structure of a training device for an image text data quality enhancement model provided in an embodiment of the present application;
[0042] Fig. 9 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0043] In order to better understand the technical solution of the present application, the embodiments of the present application are described in detail below with reference to the accompanying drawings.
[0044] It should be clear that the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present application.
[0045] The terms used in the embodiments of the present application are only for the purpose of describing specific embodiments, and are not intended to limit the present application. The singular forms "a", "said" and "the" used in the embodiments of the present application and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings.
[0046] It should be understood that the term "and / or" used in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.
[0047] Multimodal Large Language Model (MLLM) is a new research hotspot that has emerged in recent years. It uses a powerful large language model as the brain to perform multimodal tasks. The development of the MLLM model has greatly expanded the capabilities of artificial intelligence. A key factor in training the MLLM model is the quality of the images and description information in the multimodal pre-training dataset. High-quality images and description information have a high degree of matching, that is, the description information describes the image more clearly and accurately. High-quality images and description information can improve the ability of the MLLM model to integrate multiple modalities. Therefore, the images and description information that need to be selected from the initial samples with high matching between description information and images are obtained. In this way, more samples in the initial samples are filtered out, that is, a large number of high-quality images are filtered out due to the mismatch of their description information, which makes the data utilization rate low and increases the difficulty of model training.
[0048] In view of the above problems, the embodiment of the present application provides a data processing method, a training method and a device for an image description model, wherein the method specifically comprises: obtaining an image and initial description information of the image; inputting the image and the initial description information into an image text data quality enhancement model to obtain an output result of the image text data quality enhancement model; wherein the image text data quality enhancement model is used to match the image and the initial description information to obtain evaluation information, and when the evaluation information indicates that the initial description information has a low matching degree with the image, the initial description information is corrected based on the image to obtain enhanced description information of the image, and the enhanced description information and the evaluation information are used as output results. When the evaluation information indicates that the description information has a high matching degree with the image, the evaluation information is used as the output result; based on the output result of the image text data quality enhancement model, the training data of the MLLM model is determined. wherein, when the output result is the evaluation information, the image and the initial description information of the image are determined as the training data of the MLLM model; when the output result is the evaluation information and the enhanced description information of the image, the image and the enhanced description information of the image are determined as the training data of the MLLM model. In this way, in the embodiment of the present application, when the image and the initial description information have a low matching degree, the enhanced description information of the image can be regenerated by the image text data quality enhancement model to improve the matching degree between the image and the description information. In this way, when the output result of the image text data quality enhancement model is the evaluation information, it means that the image and the initial description information have a high matching degree, and the image and the initial description information can be directly used as the training data of the MLLM model. When the output result of the image text data quality enhancement model is the evaluation information and the enhanced description information of the image, it means that the image and the initial description information have a low matching degree, and the image text data quality enhancement model generates enhanced description information with a high matching degree with the image. At this time, the image and the enhanced description information of the image can be used as the training data of the MLLM model, so that the high-quality image will not be filtered out due to the low matching degree with its initial description information, thereby improving the utilization rate of the image and reducing the training complexity of other models. The following is a detailed description.
[0049] See also Figure 1 , is a flow chart of a data processing method provided in an embodiment of the present application. Figure 1 As shown, the method includes:
[0050] Step S101: Obtain an image and initial description information of the image.
[0051] In an embodiment of the present application, in order to improve the matching degree between the image and the image description information, the data processing device may first obtain the image and the initial description information of the image. In some embodiments, the data processing device may obtain the image and the initial description information of the image from the Internet. In other embodiments, the data processing device may obtain the image and the initial description information of the image from other data sets, for example, from the CC12M (Conceptual 12M) data set or the CC3M (Conceptual Captions) data set, or the LAION-400M data set. Of course, it can also be obtained from other data sets, and the embodiment of the present application is not limited to this.
[0052] Step S102: input the image and initial description information into the image text data quality enhancement model to obtain the output result of the image text data quality enhancement model.
[0053] Among them, the image text data quality enhancement model is used to match the image and the initial description information to obtain evaluation information. When the evaluation information indicates that the initial description information and the image have a low match, the initial description information is corrected based on the image to obtain enhanced description information of the image, and the enhanced description information and the evaluation information are used as output results; when the evaluation information indicates that the initial description information and the image have a high match, the evaluation information is used as the output result.
[0054] In an embodiment of the present application, an image text data quality enhancement model can be pre-trained, and the image text data quality enhancement model is used to match the input image and the initial description information to obtain evaluation information. The evaluation information is used to indicate the matching degree of the input image and the initial description information. When the evaluation information indicates that the matching degree between the image and the initial description information is low, it means that the description of the image by the initial description information is inaccurate. At this time, the image text data quality enhancement model can correct the initial description information according to the image to obtain the enhanced description information of the image. Among them, the enhanced description information has a higher matching degree with the image. Based on this, after obtaining the image and the initial description information, the data processing device can use the image and the initial description information of the image as the input of the image text data quality enhancement model, and input the image and the initial description information of the image into the image text data quality enhancement model. After receiving the image and the initial description information, the image text data quality enhancement model can extract features from the image, detect whether the feature information of the image is contained in the initial description information, so as to determine whether the image and the initial description information are semantically consistent, thereby generating evaluation information. The image text data quality enhancement model can judge the matching degree between the image and the initial description information based on the evaluation information. When the evaluation information indicates that the matching degree between the image and the initial description information is low, the image text data quality enhancement model corrects the initial description information based on the image to obtain enhanced description information of the image. The enhanced description information of the image is used to describe the feature information of the image, and the matching degree between the enhanced description information and the image is higher than the matching degree between the initial description information and the image. The image text data quality enhancement model can output the enhanced description information and the evaluation information of the image as output results. When the evaluation information indicates that the matching degree between the image and the initial description information is high, it means that the initial description information accurately describes the image. At this time, the image text data quality enhancement model can directly output the evaluation information as an output result. The data processing device can obtain the output result output by the image text data quality enhancement model, so that the image and the description information of the image can be obtained, and the matching degree between the description information and the image is high.
[0055] That is, in the embodiment of the present application, when the degree of match between the image and the initial description information is low, the enhanced description information of the image can be regenerated through the image text data quality enhancement model to improve the degree of match between the image and the description information, so that the image and its enhanced description information will not be filtered out due to the low matching problem during subsequent use. For example, the image and its enhanced description information can be used as training data for other models, thereby improving the utilization rate of the image and reducing the training complexity of other models.
[0056] As a possible implementation method, the image text data quality enhancement model matches the image and the initial description information to obtain the evaluation information in the following manner: the image text data quality enhancement model extracts the feature information of the image and the feature information of the initial description information, and matches the feature information of the image and the feature information of the initial description information to obtain the evaluation information.
[0057] In the embodiment of the present application, when the image and initial description information are received at the input end, the image text data quality enhancement model can extract the feature information of the image and the feature information of the initial description information, for example, identify the content contained in the image to extract the feature information of the image, extract the text description content in the initial description information, and obtain the feature information of the initial description information. The feature information of the image is matched with the feature information of the initial description information, and the image text data quality enhancement model can match the feature information of the image with the feature information of the initial description information from multiple preset aspects to obtain evaluation information.
[0058] In some embodiments, the image text data quality enhancement model can be composed of a visual encoder, a multi-layer perceptron and a large language model. Among them, the visual encoder can extract feature information of the input image to obtain feature information of the image. That is, the visual encoder can extract key visual features in the image and obtain feature information so as to provide a judgment basis for the subsequent matching between the image and the description information. In some embodiments, the above-mentioned visual encoder can be a SigLIP model. Of course, the visual encoder can also be implemented by other models, for example, a CLIP (Contrastive Language-Image Pre-training, multimodal pre-training) model, or other models, and the embodiments of the present application are not limited to this.
[0059] The Large Language Model (LLM) uses Qwen-2-7B as the language model, which is responsible for processing text information, that is, the description information of the image, and combines it with the output of the visual encoder to evaluate the semantic consistency between the image and the description information.
[0060] The multi-layer perceptron is an intermediate layer connecting the output of the visual encoder and the large language model. It is used to fuse the visual features of the image with the speech features of the description information and improve the comprehensive features for the final matching evaluation.
[0061] In this way, after receiving the image, the image text data quality enhancement model can extract features of the image through the visual encoder to obtain feature information of the image. The multi-layer perceptron can extract speech features from the initial description information, and fuse the feature information of the image and the feature information of the initial description information to provide the fused feature information to the large language model. The large language model can judge the semantic consistency between the image and the initial description information in multiple preset aspects based on the fusion features of the feature information of the image and the feature information of the initial description information, thereby obtaining evaluation information for evaluating the matching degree of the image and the initial description information in multiple preset aspects.
[0062] In some embodiments, in order to improve the accuracy of the evaluation information, the evaluation information output by the image text data quality enhancement model includes at least one of the overall information of the image and the description information, the quality information of the description information, the matching information between the image and the description information, the object detailed information, the semantic understanding information, the description information or the chart description information. In some embodiments, in order to more accurately describe the evaluation information, the evaluation information output by the image text data quality enhancement model can explain the situation of each information it contains through the evaluation score and the explanation of the evaluation score. For example, the evaluation information can include the evaluation score and explanation of the overall situation of the image and the description information, the evaluation score and explanation of the quality of the description information, the evaluation score and explanation of the matching degree between the image and the description information, the evaluation score and explanation of the object detailed information, the evaluation score and explanation of the semantic understanding, and the evaluation score and explanation of the description information or the chart description.
[0063] As a possible implementation, the image text data quality enhancement model can determine the degree of match between the image and the initial description information based on the evaluation information. For example, when the evaluation information includes the overall situation of the image and the description information, the degree of match between the image and the initial description information can be determined based on the score of the overall situation. A matching threshold is pre-set in the image text data quality enhancement model. When the score of the overall situation is greater than the matching threshold, it means that the image and the initial information have a high degree of match. When the score of the overall situation is not greater than the matching threshold, it means that the image and the initial information have a low degree of match. When the image text data quality enhancement model determines that the image and the initial information have a low degree of match, it needs to correct the initial description information based on the image to obtain enhanced description information of the image.
[0064] Of course, the image text data quality enhancement model can also use other information in other evaluation information to determine the degree of match between the image and the initial description information. The above-mentioned process of determining the degree of match between the image and the initial description information based on the score of the overall situation is parameterized and will not be repeated here.
[0065] It should be understood that the above-mentioned matching degree threshold can be preset by the user according to actual needs, or it can be set by the image text data quality enhancement model during the training process, and the embodiments of the present application are not limited to this.
[0066] As a possible implementation method, the above-mentioned image text data quality enhancement model modifies the initial description information based on the image in the following way to obtain enhanced description information of the image, and uses the enhanced description information and evaluation information as output results:
[0067] The image text data quality enhancement model modifies the initial description information based on the feature information of the image to obtain the enhanced description information of the image, and takes the enhanced description information and evaluation information as the output results.
[0068] The enhanced description information of the image describes the feature information of the image.
[0069] That is, when the image text data quality enhancement model determines that the image has a low match with the initial description information, it needs to regenerate the description information of the image. At this time, the image text data quality enhancement model can correct the initial description information according to the feature information of the image. For example, it can delete the information irrelevant to the feature information of the image in the initial description information, add the feature information of the image to the initial description information, and correct the initial description information. The corrected initial description information is used as the enhanced description information of the image. The image text data quality enhancement model outputs the enhanced description information and evaluation information of the image as the output result to the data processing device.
[0070] In some embodiments, when the data processing device inputs the initial description information to the image text data quality enhancement model, the initial description information can be used as the content of the input instruction, and the input instruction and the image can be used as the input information of the image text data quality enhancement model. For example, the input instruction is: to grade the description information, and the description information is: xxxxx; for images with low overall scores, the visual elements of the image are summarized to generate enhanced description information of the image. In some embodiments, the input instruction can be input by the user, and the content of the input instruction can be different according to the user's instruction input habits, but the input instruction should contain the initial description information and the relevant information that requires the image text data quality enhancement model to generate enhanced description information of the image when the image and the initial description information match poorly. After obtaining the input instruction and the image, the data processing device can use the input instruction and the image as the input information of the image text data quality enhancement model and input them to the image text data quality enhancement model. After receiving the image, the image text data quality enhancement model can analyze the visual features of the image to obtain the feature information of the image, and parse the input instruction to obtain the initial description information, parse the feature information of the initial description information, match the feature information of the image with the feature information of the initial description information, and obtain the evaluation information. When the evaluation information indicates that the image has a low match with the initial description information, for example, the score of the overall situation in the evaluation information is lower than the matching threshold, the image text data quality enhancement model modifies the initial description information based on the feature information of the image to obtain enhanced description information of the image, and outputs the enhanced description information and the evaluation information as output results to the data processing device. When the evaluation information indicates that the image has a high match with the initial description information, for example, the score of the overall situation in the evaluation information is greater than the matching threshold, the image text data quality enhancement model outputs the evaluation information as an output result to the data processing device.
[0071] Exemplarily, the data processing device obtains image a, initial description information of image a, image b, and initial description information of image b. Assume that image a is as follows Figure 2a As shown, image b is Figure 2bAs shown. The initial description information of image a is "On October 29, 2016, at Stadium A, player a (left) of xxxx Football Club helped his teammate player b in the match between xxx Football Club and bbb Football Club." The initial description information of image b is "Beachwood Cardigan". The data processing device can generate input instruction a based on the initial description information of image a. Assume that the generated input instruction a is: The initial description information is graded: [On October 29, 2016, on October 29, 2016, at Stadium A, player a (left) of xxxx Football Club helped his teammate player b in the match between xxx Football Club and bbb Football Club. ]. For images with lower scores, key visual elements are summarized and description information is regenerated. The data processing device uses the input instruction a and image a as input information to the image text data quality enhancement model. The image text data quality enhancement model can extract features from image a to obtain feature information of image a. The image text data quality enhancement model parses the input instruction a to obtain the initial description information carried in the input instruction a. Feature extraction is performed on the initial description information to obtain feature information of the initial description information, and the feature information of the image and the feature information of the initial description information are matched to obtain evaluation information. The evaluation information obtained by the image text data quality enhancement model is as follows:
[0072]
[0073]
[0074] The image text data quality enhancement model can determine that the matching degree between the image and the initial description information is high based on the overall situation, so the image text data quality enhancement model can directly output the evaluation information.
[0075] The data processing device may generate an input instruction b according to the initial description information of the image b. Assume that the generated input instruction b is: grading the initial description information: [Beachwood Cardigan]. For images with lower scores, summarize the key visual elements and regenerate the description information.
[0076] The data processing device uses the input instruction b and image b as input information of the image text data quality enhancement model, and inputs them into the image text data quality enhancement model. The image text data quality enhancement model can extract features from image b to obtain feature information of image b. The image text data quality enhancement model parses the input instruction b to obtain the initial description information carried in the input instruction b. Feature extraction is performed on the initial description information to obtain feature information of the initial description information, and the feature information of the image and the feature information of the initial description information are matched to obtain evaluation information. The evaluation information obtained by the image text data quality enhancement model is as follows:
[0077]
[0078] The image text data quality enhancement model can determine that the matching degree between the image and the initial description information is low based on the overall situation. Therefore, the image text data quality enhancement model can modify the initial description information based on the feature information of image b to obtain enhanced description information of image b. The enhanced description information of image b is: a model wears a black polka dot cardigan, a white T-shirt and white pants. The image text data quality enhancement model can output the enhanced description information and evaluation information of image b as output results.
[0079] As a possible implementation method, the image text data quality enhancement model is a model trained based on an initial sample and a comparison sample; the image in the comparison sample is the same as the image in the initial sample, and the matching degree between the description information in the comparison sample and the image is different from the matching degree between the description information in the initial sample and the image.
[0080] Among them, when the matching degree between the image and the description information in the initial sample is high, the matching degree between the description information and the image in the comparison sample is low; when the matching degree between the image and the description information in the initial sample is low, the matching degree between the description information and the image in the comparison sample is high.
[0081] In order to improve the accuracy of the image text data quality enhancement model and the consistency of the evaluation information, initial samples and comparison samples are required when training the image text data quality enhancement model. The comparison sample is generated based on the initial sample. That is, the image in the comparison sample is the same as the image in the initial sample, but the image description information of the comparison sample is different from the image description information in the initial sample, and the matching degree is also different. When the matching degree between the image and the description information in the initial sample is high, the matching degree between the image and the description information in the comparison sample is low; when the matching degree between the image and the description information in the initial sample is low, the matching degree between the description information in the comparison sample and the image is high. In this way, there is a synergistic relationship between the use of comparison samples and the generation of enhanced description information, which can improve the accuracy of the image text data quality enhancement model.
[0082] Step S103: Determine the training data of the MLLM model based on the output result of the image text data quality enhancement model.
[0083] Among them, when the output result is evaluation information, the image and the initial description information of the image are determined as training data of the MLLM model; when the output result includes evaluation information and enhanced description information of the image, the image and the enhanced description information of the image are determined as training data of the MLLM model.
[0084] In an embodiment of the present application, after obtaining the output result of the image text data quality enhancement model, the data processing device can determine the training data of the MLLM model according to the output result, that is, it can determine whether to determine the image and the initial description information of the image as the training data of the MLLM model. When the output result of the image text data quality enhancement model is the evaluation information, it can be determined through the evaluation information that the matching degree between the image and the initial description information is high. At this time, the data processing device can directly determine the image and the initial description information of the image as the training data of the MLLM model. When the output result of the image text data quality enhancement model is the evaluation information and the enhanced description information of the image, it means that the matching degree between the image and the initial description information of the image is low. At this time, the image text data quality enhancement model regenerates the enhanced description information of the image. The enhanced description information of the image contains the feature information of the image, and the matching degree between the image and the image is high. At this time, the data processing device can directly use the image and the enhanced description information of the image as the training data of the MLLM model. In this way, for the acquired image, in the embodiments of the present application, the image text data quality enhancement model can be used to acquire descriptive information with a high degree of matching of the image, so that the image and the descriptive information with a high degree of matching can be used as training data for the MLLM model. The acquired image can be effectively utilized, and the situation in which high-quality images are filtered out due to the low match between them and their descriptive information is avoided, which greatly reduces the difficulty of acquiring training data for the MLLM model, improves the training efficiency of the MLLM model and reduces the complexity of its training.
[0085] That is to say, the data processing device can dynamically evaluate and improve the matching degree between the input image and the description information through the image text data quality enhancement model. That is, when the matching degree between the input image and the initial description information is low, the image text data quality enhancement model can rewrite the description information, and generate enhanced description information with a high matching degree with the image based on the input image and the initial description information, thereby improving the quality of the image and its description information. The data processing method adopted by the embodiment of the present application can improve the data quality of the overall training sample while retaining the data amount of the training sample to the maximum extent, reducing the difficulty of model training.
[0086] refer to Figure 3, is a flow chart of a training method for an image text data quality enhancement model provided in an embodiment of the present application. Figure 3 As shown, the method includes:
[0087] Step S301: Obtain first-stage training samples.
[0088] In the embodiment of the present application, when training the image text data quality enhancement model, training samples are required, and the first stage training samples can be obtained at this time. In some embodiments, the training device can obtain the first stage training samples from the Internet. Alternatively, in other embodiments, the LAION-400M data set is obtained to maintain consistency and avoid potential differences caused by multiple data sources.
[0089] In order to improve the diversity of the first-stage training samples, the training device may obtain some first-stage training samples from the LAION-400M dataset and obtain other first-stage training samples from the Cauldron dataset. For example, 260k data may be randomly selected from the LAION-400M dataset and 690k data may be randomly selected from the Cauldron dataset to form the first-stage training samples.
[0090] Step S302: training the initial model based on the first-stage training samples to update the initial model to an image-text feature alignment model.
[0091] Among them, the image-text feature alignment model is used to identify the feature information of the image and the feature information of the description information in the input information, and match the feature information of the image with the feature information of the description information.
[0092] In the embodiment of the present application, since the image text data quality enhancement model needs to evaluate the matching degree of the image and the description information and needs to generate enhanced description information of the image when the evaluation information indicates that the matching degree of the image and the description information is low. Therefore, in order to improve the training accuracy of the image text data quality enhancement model, the training of the image text data quality enhancement model can be divided into two stages in the embodiment of the present application. In the first stage, an image text feature alignment model can be trained through the first stage training samples. The image text feature alignment model is used to identify the feature information of the image and the feature information of the description information in the input information, and match the feature information of the image with the feature information of the description information. That is, after the training device obtains the first stage training samples, the first stage training samples can be used to train the initial model, and the parameters of the initial model can be adjusted during the training process to obtain the image text feature alignment model. The image text feature alignment model can understand the content of the image and the description information, and identify the feature information of the image and the feature information of the description information, and match the feature information of the image with the feature information of the description information.
[0093] In some embodiments, the initial model may be composed of a visual encoder, a multi-layer perceptron, and a large language model.
[0094] Step S303: Obtain second-stage training samples.
[0095] Among them, the second stage training samples include initial samples and comparison samples; the image in the comparison sample is the same as the image in the initial sample, and the matching degree between the description information in the comparison sample and the image is different from the matching degree between the description information in the initial sample and the image; among them, when the matching degree between the image and the description information in the initial sample is high, the matching degree between the description information in the comparison sample and the image is low, and when the matching degree between the image and the description information in the initial sample is low, the matching degree between the description information in the comparison sample and the image is high.
[0096] In an embodiment of the present application, the training of the image text data quality enhancement model is divided into two stages. The first stage is to train the initial model as an image text feature alignment model, which is used to identify the feature information of the image and the feature information of the description information, and match the feature information of the image with the feature information of the description information. In the second stage, the image text feature alignment model can be trained as an image text data quality enhancement model. Based on this, after completing the training of the first stage and obtaining the image text feature alignment model, the second stage training is required. In order to improve the training accuracy of the image text data quality enhancement model, it is usually necessary to re-acquire the training samples for the second stage model training, that is, to obtain the second stage training samples. In order to further improve the training accuracy of the image text data quality enhancement model, the second stage training samples can include initial samples and comparison samples generated based on the initial samples.
[0097] In some embodiments, obtaining the second-stage training samples includes:
[0098] Obtain an initial sample, wherein the initial sample includes an image, description information of the image, and evaluation information of the description information.
[0099] Based on the images in the initial sample, the description information of the images, and the evaluation information of the description information, a comparison sample is obtained.
[0100] That is, in order to more accurately train the image text data quality enhancement model, the initial sample obtained includes an image, image description information, and evaluation information of the description information. Among them, the image and the image description information in the initial sample can be obtained by referring to the method of obtaining the first-stage training sample in the above step S301. After obtaining the image and the image description information, the training device can obtain the evaluation information of the description information through the GPT-4o model. The initial sample is obtained in the above manner. After obtaining the initial sample, a comparison sample can be generated according to the initial sample. In some embodiments, in order to reduce the difficulty of generating a comparison sample, a comparison sample can be generated by the GPT-4o model. When the evaluation information of the initial sample indicates that the image and the description information match poorly, a first instruction and an image in the initial sample can be input to the GPT-4o model, wherein the first instruction is used to instruct the GPT-4o model to generate description information with a high degree of matching with the image. When the evaluation information of the initial sample indicates that the image and the description information match highly, a second instruction and an image in the initial sample can be input to the GPT-4o model, wherein the second instruction is used to instruct the GPT-4o model to generate description information with a low degree of matching with the image.
[0101] Assume that P1 represents the first instruction, P2 represents the second instruction, I represents the image, C represents the description information of the image, and S(I, C) represents the matching degree between the image and the description information. The newly generated description information is represented by C', and the matching degree between the image and the newly generated description information is represented by S'. Then
[0102] C' high = P1(I, C), where S(I, C) is low, S' high = S(I, C' high),
[0103] C' low = P2(I, C), where S(I, C) is high, S' low = S(I, C' low).
[0104] It should be understood that a high C′ indicates a rewritten description information with a high degree of match with image I, a low S(I, C) indicates a low degree of match between the image and the original description information, and a high S′ indicates a high degree of match between image I and the newly generated description information. P1(I, C) indicates that the GPT-4o model is instructed to rewrite the description information in image I and description information C with a low degree of match to obtain description information with a high degree of match; P2(I, C) indicates that the GPT-4o model is instructed to rewrite the description information in image I and description information C with a high degree of match to obtain description information with a low degree of match; a low C′ indicates a rewritten description information with a low degree of match with image I; a high S(I, C) indicates a high degree of match between the image and the original description information, and a low S′ indicates a low degree of match between image I and the newly generated description information. In this way, the second-stage training samples including comparison samples and initial samples can be constructed in the above manner.
[0105] In some embodiments, assuming that the initial sample Draw = {(I1, C1), (I2, C2)}; assuming that S(I1, C1) is high, then C' is low and S' is low; assuming that S(I2, C2) is low, then C' is high and S' is high. Then, the second-stage training sample D finally constructed includes {image text input: target output}:
[0106]
[0107] For example, assuming that the initial sample contains 260k data, the generated comparison sample is also 260k data, and the second-stage training sample is 520k data.
[0108] In some embodiments, in order to ensure the diversity of the second-stage training samples and make the image text data quality enhancement model training more accurate, the second-stage training samples include not only the initial samples and the comparison samples, but also other samples obtained from other data sets such as the Cauldron data set. The initial samples, comparison samples and other samples are mixed to obtain the second-stage training samples.
[0109] Step S304: train the image-text feature alignment model based on the second-stage training samples to update the image-text feature alignment model to an image-text data quality enhancement model, so that when the degree of match between the input image and the initial description information is low, the image-text data quality enhancement model corrects the initial description information based on the image to obtain enhanced description information of the image with a high degree of match with the image, so as to determine the image and the enhanced description information of the image as training data for the MLLM model.
[0110] In an embodiment of the present application, after obtaining the second-stage training samples, the image-text feature alignment model can be trained using the second-stage training samples. During the training process, the parameters of the image-text feature alignment model can be adjusted to obtain an image-text data quality enhancement model. The image-text data quality enhancement model can perform feature matching on the input image and description information to obtain evaluation information. When the evaluation information indicates that the matching degree between the image and the description information is low, the input description information is corrected according to the feature information of the image to obtain enhanced description information of the image. The enhanced description information contains the feature information of the image, and the matching degree with the image is high. In this way, the matching degree between the input image and the initial description information can be evaluated by the image-text data quality enhancement model. When the matching degree between the input image and the initial description information is low, enhanced description information with a high matching degree with the image is generated, so that the image and the enhanced description information of the image can be determined as the training data of the MLLM model, reducing the possibility that the image description information cannot be used as the training data of the MLLM model due to the matching degree between the image and the image, improving the utilization rate of the image, and reducing the difficulty of obtaining the training data of the MLLM model.
[0111] In some embodiments, the evaluation information output by the image text data quality enhancement model includes at least one of the overall information of the image and the description information, the quality information of the description information, the matching information between the image and the description information, the object detailed information, the semantic understanding information, the description information or the chart description information. In some embodiments, in order to more accurately describe the evaluation information, the evaluation information output by the image text data quality enhancement model can explain the situation of each information it contains through the evaluation score and the explanation of the evaluation score. For example, the evaluation information can include the evaluation score and explanation of the overall situation of the image and the description information, the evaluation score and explanation of the quality of the description information, the evaluation score and explanation of the matching degree between the image and the description information, the evaluation score and explanation of the object detailed information, the evaluation score and explanation of the semantic understanding, and the evaluation score and explanation of the description information or the chart description.
[0112] In some embodiments, when the image-text feature alignment model is trained based on the second-stage training samples, the image description information can be added to the input instruction, and the input instruction and the image can be used as the input information of the model to train the model. In this way, the image-text data quality enhancement model obtained after the training is completed can output evaluation information of the matching degree between the image and the initial description information based on the input image and the input instruction containing the initial description information. And when the evaluation information indicates that the matching degree between the image and the initial description information is low, the image-text data quality enhancement model will output the rewritten enhanced description information with a higher matching degree with the image.
[0113] In some embodiments, in order to improve the training accuracy of the image text data quality enhancement model, the number of data with low matching degree between images and description information can be increased in the second stage training samples, thereby enhancing the image text data quality enhancement model's ability to evaluate and improve the removal of samples with poor matching quality between images and description information.
[0114] In this way, the image text data quality enhancement model can output the rewritten enhanced description information with a higher degree of match with the image when the evaluation information indicates that the image and the initial description information have a low degree of match. Therefore, the image text data quality enhancement model has an adaptive description information re-function, so that each image can obtain description information with a high degree of match with the image. In this way, the image text data quality enhancement model can dynamically improve the matching quality between the image and the description information, and can also ensure that when the image and the description information are used for other model training, each collected image can be used, thereby improving the utilization rate of the image data and reducing the difficulty and complexity of model training.
[0115] In some embodiments, the image text data quality enhancement model in the embodiment of the present application can be represented by the AITQE model. For the convenience of explanation, the following description takes the AITQE model as an example of the image text data quality enhancement model in the embodiment of the present application. In order to evaluate the AITQE model, we use it to filter or enhance data to train the MLLM. Since the performance of the MLLM reflects the quality of the training data and the effectiveness of the AITQE model in the embodiment of the present application, it is crucial to ensure stable training and evaluation, that is, if two MLLMs are trained under exactly the same settings, the evaluation results should also be close.
[0116] Preliminary experiments: The training strategy of MLLM is mostly two-stage training after LLaVA (i.e., only the pre-alignment stage of the projection layer and the fine-tuning stage of the LLM backbone are trained once each). Assuming that 558k data of the original LLaVA are used as the first-stage training data and 665k data are used as the second-stage training data, a preliminary experiment of training the LLaVA model is conducted using the same training settings as LLaVA. Then, the second-stage model is evaluated on five benchmarks: GQA, OKVQA, TextVQA, VQAv2, and SEED-Bench-2. The training-evaluation process was run twice with exactly the same settings, and the results are shown in Table 1. The average evaluation scores of the five benchmarks differ by 2.60, which indicates that it may be unstable if this training strategy and evaluation process is used. Therefore, it is crucial to adopt a more stable method.
[0117] Table 1: Evaluation results of the LLaVA second-stage model
[0118] Second stage model average value First training 57.73 Second training 60.33
[0119] Training strategy and evaluation settings: In the embodiments of the present application, a strategy for stabilizing MLLM training is determined. For the architecture of MLLM, the MLP layers in the SigLIP, Qwen-2-7B, and AITQE models are used in the embodiments of the present application. That is, the initial model is composed of the MLP layers in the SigLIP, Qwen-2-7B, and AITQE models. In the training phase of MLLM, in the embodiments of the present application, 558k samples of LLaVA are mixed with instruction data from the Cauldron dataset, and in the process of training the model in the first stage, all parameters of the model are adjusted so that the model in the first stage can be used for evaluation. Another advantage of this training strategy is that it eliminates the second stage of training, reducing the potential impact of two-stage training.
[0120] In order to evaluate the accuracy, multiple evaluation benchmarks can be used for evaluation, such as the SEED-Bench-2 benchmark, the MME (Multimodal Model Evaluation) benchmark, the AMBER benchmark, the OKVQA model, the VQAv2 model, the DocVQA model, the TextVQA model, and the Textcaps model. For the AMBER benchmark, evaluation is performed in terms of attributes, existence, relations, and generated vision. For the MME benchmark, four hallucination-related tests (existence, count, location, and color) are performed, and the score range is set to [0,100]. For the VQAv2 model, due to the large amount of tests, 1,000 were extracted for testing. For the Textcaps model, the CIDEr score is multiplied by 100 to calculate the average score.
[0121] Under the above evaluation settings, another pre-experiment was conducted in the first stage with 558k training samples of LLaVA in the Cauldron dataset, all parameters in the model were trained, and the evaluation was performed using the above evaluation data. The results are shown in Table 2. It can be seen from Table 2 that the difference between the two training evaluations is much smaller, which indicates that the training and evaluation are more stable.
[0122] Table 2: Evaluation results of the first stage LLaVA model
[0123] First stage model average value First training 61.45 Second training 60.72
[0124] In order to more clearly illustrate the function of the image text data quality enhancement model, the image text data quality enhancement model is compared with other models as follows.
[0125] Table 3 below shows a comparison of random sampling, MLM filter and the image text data quality enhancement model in the embodiment of the present application. For reference, the model trained using the strategy described in the embodiment of the present application can be used using the training samples of the above 558k LLaVa model (the last row in the table).
[0126] Table 3: Comparison of MLLMs trained on 256K and 558K samples filtered from a fixed 2M data pool. MME stands for Hallucination Related Test of MME, with scores ranging from 0 to 100. Abbreviations: CTQ (Description Quality), ITM (Image to Description Match), ODF (Object Detail Implementation), SU (Semantic Understanding).
[0127] Table 3
[0128]
[0129]
[0130] AITQE is effective in filtering data for high-quality images and their description information. In the 256K setting, the scores for random sampling were relatively low, highlighting the need for methods to improve the quality of training data. MLM filters with four standards showed varying degrees of improvement, with an average improvement of 1.64. The AITQE model in the embodiment of the present application showed the most promising performance, achieving the highest overall average score of 65.21. This is an improvement of 7.16 points over random sampling and 5.52 points over the average of the MLM filters, indicating its potential as a robust and general filtering method. Compared with the MLLM trained on LLaVA-558K, it is worth noting that the AITQE filtering training model achieved excellent results in several benchmark tests despite using less than half the number of training samples. This shows that the AITQE filtering method in the embodiment of the present application can effectively identify high-quality images and description information for MLLM training.
[0131] Lowering the threshold to include more data degrades model performance. In the 558k setting, we observe that increasing the training data from 256K random samples to 558k improves the average score of the trained MLLM by 2.02 points. This improvement is mainly attributed to the increase in the amount of training data. While AITQE filtering shows better results than random selection, this gain (+1.82) is not as significant compared to the 256K scenario (+7.16). It is worth noting that the average score of AITQE in the 558k setting is lower than the average score in the 256k setting (-3.32). This degradation is due to the fact that the selection process prioritizes samples with higher scores, and the 558K dataset inevitably includes the top-scoring 256K samples as well as other data with lower scores. As a result, the overall average quality of the 558K dataset decreases. Compared to the MLM filter, the average performance of AITQE (61.89) is comparable to that of the MLM filter (61.84). AITQE also achieves a slightly higher average (+0.8) compared to LLaVA-558K. This suggests that AITQE provides a consistently effective filtering method. These results suggest that increasing the amount of data by filtering a fixed pool will inevitably include lower quality samples, which may offset the benefits of a larger dataset. This highlights the critical balance between data quantity and quality in MLLM training, especially when data resources are limited.
[0132] AITQE can be used as an enhancement tool to expand high-quality description data. Table 4 shows that AITQE is a quality enhancer for both image and description information of randomly sampled data.
[0133]
[0134]
[0135] As we scale up the size of our training data, the effectiveness of AITQE as a data quality enhancer is evident. In addition to its role as a filter for identifying data quality, AITQE’s more important function is its ability to enhance multimodal training data. We analyze the performance gains for different dataset sizes (from 256K to 12M random samples), as detailed in Table 4 and Figure 4 .
[0136] AITQE significantly improves model performance on 256K samples, with an average score increase of 4.29 points. Substantial improvements are observed on tasks such as SEED2, AMBER, and most notably Textcaps. However, it is worth noting that performance on DocVQA and TextVQA decreases slightly at this scale. When we scale up to 558K samples, random sampling shows improvement over the 256K baseline, but the AITQE-processed data maintains a clear advantage, with an average score increase of 4.05, demonstrating AITQE's consistent advantage.
[0137] The benefits of AITQE become more pronounced at the 2M sample scale. Here, we observe that AITQE not only maintains its performance advantage, but also shows significant improvement. AITQE achieves an average score of 65.35 on augmented data, a substantial improvement of 4.50 points. Notably, performance on Textcaps continues to improve, reaching 77.04 points. At the largest scale of 12M samples, AITQE demonstrates its robustness and scalability. AITQE achieves an impressive average score of 66.90 on filtered data, maintaining a significant increase of 4.70 points. The scale shows improvements on most benchmarks, especially in SEED2, VQAv2, DocVQA, and Textcaps.
[0138] The above process can demonstrate the scalability and robustness of the AITQE model in the embodiment of the present application. As the amount of training data increases from 256K to 12M samples, AITQE consistently generates higher quality data sets, resulting in excellent MLLM performance. The consistent performance improvement at various scales highlights the effectiveness of AITQE in improving data quality, ultimately resulting in a more powerful and more general multimodal language model.
[0139] In order to further verify the effectiveness of AITQE, we first conducted a comparative experiment between AITQE and ShareCaptioner (shared description information). The results of applying AITQE and ShareCaptioner to the 558K dataset are shown in Table 5.
[0140] Table 5: Comparison between 558k training samples and ShareCaptioner
[0141]
[0142] As can be seen from Table 5, AITQE significantly outperforms ShareCaptioner in overall performance, with an average score of 64.12 for AITQE and 58.15 for ShareCaptioner, a substantial improvement of 5.97 points. AITQE shows particular strengths in most benchmarks. The most significant difference is observed in Textcaps, where AITQE outperforms Share Captioner by 44.76 points, highlighting AITQE’s unique strength in image description information.
[0143] The impact of the comparison samples and description information rewriting strategies on the performance of the AITQE model is analyzed in Table 6. The model is used to filter 256K data, and Table 6 shows the result scores of each training MLLM.
[0144] Table 6 shows the performance comparison of MLLM using 256K training samples, which are filtered by intermediate models using different training strategies.
[0145]
[0146]
[0147] Synergistic effect of combined strategies in building AITQE model: The effectiveness of combining contrast sampling and description rewriting techniques in improving model performance can be demonstrated in Table 6. The average score of the basic scorer of the GPT model collected in the first round was 59.88. When contrast sampling was applied alone, the average score obtained dropped slightly to 59.81. The application of rewriting description information as an independent strategy resulted in a significant drop in the average score to 57.70.
[0148] Although these strategies have their own shortcomings, their combination produces significant results. The AITQE model provided in the embodiment of the present application is trained by integrating the training data of the comparison samples and the rewriting enhancement of the description information, and obtained the highest average score of 65.21. This is 5.33 points higher than the basic scorer. The AITQE model shows consistent performance in most benchmarks, with particularly significant enhancements in Textcaps.
[0149] For better explanation, Figure 5 and Figure 6 The score distribution in the GPT training data set and the scores of the training samples of the embodiments of the present application can be characterized.
[0150] exist Figure 5(a) in Figure 1 shows the distribution of GPT scores. The scores of the training samples corresponding to the basic scorer are non-uniformly distributed, but there is a prominent single peak near score 7. In contrast, the AITQE training dataset contains contrast samples, making the score distribution of the training samples more diverse. This method introduces a large number of low- and high-scoring samples.
[0151] The main purpose of citing comparative samples in the embodiments of the present application is to balance data distribution and reduce potential bias. In this way, the diversity of training samples can be increased, thereby improving the ability of the image text data quality enhancement model to distinguish between images and description information with different matching degrees. In addition, skin-beautifying images are paired with description information with high matching degrees and description information with low matching degrees, reducing the potential bias that the model may have on specific image features.
[0152] Figure 5 (b) shows the score comparison between the AITQE model with comparison samples added to the training data and the original GPT. Figure 5 (c) shows the score comparison between the AITQE model without adding comparison samples to the training data and the original GPT. By comparison, the model with added comparison samples shows a more concentrated score distribution, characterized by a smaller standard deviation and a more obvious peak. This tighter clustering of score differences indicates that the AITQE model's predictions are closer to the GPT scores. This highlights the effectiveness of adding comparison samples during training, which improves the consistency and accuracy of scores.
[0153] exist Figure 6 It is shown that the AITQE model trained with contrast samples and the AITQE model without contrast samples produce more enhanced description information. This is due to the synergistic relationship between the use of contrast samples and the regeneration of description information. The combination of these two strategies has been shown to be an effective way to improve the model's ability to critically evaluate and improve image description information. This synergy not only improves the model's discriminative ability, but also improves its ability to generate description information.
[0154] In summary, the image text data quality enhancement model obtained after training in the embodiment of the present application can match the image and the description information to obtain evaluation information, and when the evaluation information indicates that the description information has a low match with the image, it generates enhanced description information for the image to improve the match between the image and the description information. In this way, the quality of the multimodal training data can be dynamically improved while maintaining the amount of multimodal training data. The image text data quality enhancement model can maximize the use of the visual information in the valuable multimodal training data, thereby improving the utilization rate and scalability of the multimodal training data.
[0155] See also Figure 7, which is a schematic diagram of the structure of a data processing device provided in an embodiment of the present application. Figure 7 As shown, the device comprises:
[0156] The acquisition unit 701 is used to acquire an image and initial description information of the image.
[0157] The processing unit 702 is used to input the image and the initial description information into the image text data quality enhancement model to obtain the output result of the image text data quality enhancement model.
[0158] Among them, the image text data quality enhancement model is used to match the image and the initial description information to obtain evaluation information. When the evaluation information indicates that the initial description information and the image have a low match, the initial description information is corrected based on the image to obtain enhanced description information of the image, and the enhanced description information and the evaluation information are used as output results; when the evaluation information indicates that the initial description information and the image have a high match, the evaluation information is used as the output result.
[0159] The processing unit 702 is also used to determine the training data of the multimodal large language model MLLM model based on the output result of the image text data quality enhancement model.
[0160] Among them, when the output result is the evaluation information, the image and the initial description information of the image are determined as the training data of the MLLM model; when the output result includes the evaluation information and the enhanced description information of the image, the image and the enhanced description information of the image are determined as the training data of the MLLM model.
[0161] As a possible implementation method, the image-text data quality model is composed of a visual encoder, a multi-layer perceptron and a large language model.
[0162] As a possible implementation manner, the evaluation information includes overall information of the image and description information, quality information of the description information, matching information between the image and description information, object detailed information, semantic understanding information, description information or at least one of chart explanation information.
[0163] As a possible implementation method, the image text data quality enhancement model matches the image and the initial description information to obtain the evaluation information in the following way:
[0164] The image text data quality enhancement model extracts the feature information of the image and the feature information of the initial description information, and matches the feature information of the image and the feature information of the initial description information to obtain evaluation information.
[0165] As a possible implementation method, the image text data quality enhancement model modifies the initial description information based on the image in the following manner to obtain enhanced description information of the image, and uses the enhanced description information and evaluation information as output results:
[0166] The image text data quality enhancement model modifies the initial description information based on the feature information of the image to obtain the enhanced description information of the image, and takes the enhanced description information and evaluation information as the output results.
[0167] The enhanced description information of the image describes the feature information of the image.
[0168] As a possible implementation method, the image text data quality enhancement model is a model trained based on an initial sample and a comparison sample; the image in the comparison sample is the same as the image in the initial sample, and the matching degree between the description information in the comparison sample and the image is different from the matching degree between the description information in the initial sample and the image; wherein, when the matching degree between the image and the description information in the initial sample is high, the matching degree between the description information in the comparison sample and the image is low, and when the matching degree between the image and the description information in the initial sample is low, the matching degree between the description information in the comparison sample and the image is high.
[0169] See also Figure 8 , is a schematic diagram of the structure of a training device for an image text data quality enhancement model provided in an embodiment of the present application. Figure 8 As shown, the device includes
[0170] The acquisition unit 801 is used to acquire first-stage training samples.
[0171] The processing unit 802 is used to train the initial model based on the first-stage training samples to update the initial model to an image-text feature alignment model.
[0172] The image-text feature alignment model is used to identify the feature information of the image and the feature information of the description information in the input information, and align the feature information of the image and the feature information of the description information.
[0173] The acquisition unit 801 is also used to obtain the second stage training samples.
[0174] Among them, the second stage training samples include initial samples and comparison samples; the image in the comparison sample is the same as the image in the initial sample, and the matching degree between the description information in the comparison sample and the image is different from the matching degree between the description information in the initial sample and the image; among them, when the matching degree between the image and the description information in the initial sample is high, the matching degree between the description information in the comparison sample and the image is low, and when the matching degree between the image and the description information in the initial sample is low, the matching degree between the description information in the comparison sample and the image is high.
[0175] The processing unit 802 is also used to train the image-text feature alignment model based on the second-stage training samples to update the image-text feature alignment model to an image-text data quality enhancement model, so that when the degree of match between the input image and the initial description information is low, the image-text data quality enhancement model corrects the initial description information based on the image to obtain enhanced description information of the image with a high degree of match with the image, so as to determine the image and the enhanced description information of the image as training data of the MLLM model.
[0176] As a possible implementation method, the acquisition unit 801 is specifically used to obtain an initial sample; the initial sample includes an image, description information of the image, and evaluation information of the description information; based on the image in the initial sample, the description information of the image, and the evaluation information of the description information, a comparison sample is obtained.
[0177] Corresponding to the above embodiments, the present application also provides an electronic device. Fig. 9 A schematic diagram of the structure of an electronic device provided in an embodiment of the present invention, the electronic device 900 may include: a processor 901, a memory 902 and a communication unit 903. These components communicate through one or more buses, and those skilled in the art can understand that the structure of the electronic device shown in the figure does not constitute a limitation on the embodiment of the present invention, it can be a bus structure or a star structure, and can also include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.
[0178] The communication unit 903 is used to establish a communication channel so that the electronic device can communicate with other devices, receive user data sent by other devices or send user data to other devices.
[0179] The processor 901 is the control center of the electronic device. It uses various interfaces and lines to connect various parts of the entire electronic device. It runs or executes software programs, instructions, and / or modules stored in the memory 902, and calls data stored in the memory to perform various functions of the electronic device and / or process data. The processor can be composed of an integrated circuit (IC), for example, it can be composed of a single packaged IC, or it can be composed of multiple packaged ICs with the same or different functions. For example, the processor 901 can only include a central processing unit (CPU). In an embodiment of the present invention, the CPU can be a single computing core or multiple computing cores.
[0180] The memory 902 is used to store the execution instructions of the processor 901. The memory 902 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.
[0181] When the execution instructions in the memory 902 are executed by the processor 901 , the electronic device 900 is enabled to execute part or all of the steps in the above embodiments.
[0182] In a specific implementation, the present invention further provides a computer storage medium, wherein the computer storage medium may store a program, and when the program is executed, the program may include some or all of the steps in the above-mentioned embodiments provided by the present invention. The storage medium may be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.
[0183] In a specific implementation, the present invention further provides a computer program product, wherein the computer program product includes executable instructions, and when the executable instructions are executed on a computer, the computer executes part or all of the steps in the above embodiments provided by the present invention.
[0184] Those skilled in the art can clearly understand that the technology in the embodiments of the present invention can be implemented by means of software plus a necessary general hardware platform. Based on this understanding, the technical solution in the embodiments of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which can be stored in a storage medium such as ROM / RAM, a disk, an optical disk, etc., and includes a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present invention or some parts of the embodiments.
[0185] In this specification, the same or similar parts between the various embodiments can be referred to each other. In particular, for the device embodiment and the terminal embodiment, since they are basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the description in the method embodiment.
Claims
1. A data processing method, characterized in that: include: Obtaining an image and initial description information of the image; The image and the initial description information are input into an image text data quality enhancement model to obtain an output result of the image text data quality enhancement model; the image text data quality enhancement model is used to match the image and the initial description information to obtain evaluation information, and when the evaluation information indicates that the initial description information has a low matching degree with the image, the initial description information is corrected based on the image to obtain enhanced description information of the image, and the enhanced description information and the evaluation information are used as the output result; When the evaluation information indicates that the initial description information has a high degree of match with the image, the evaluation information is used as an output result; Based on the output result of the image text data quality enhancement model, the training data of the multimodal large language model (MLLM) model is determined; wherein, when the output result is the evaluation information, the image and the initial description information of the image are determined as the training data of the MLLM model; when the output result includes the evaluation information and the enhanced description information of the image, the image and the enhanced description information of the image are determined as the training data of the MLLM model.
2. The method according to claim 1, characterized in that The image text data quality model is composed of a visual encoder, a multi-layer perceptron and a large language model.
3. The method according to claim 1, characterized in that The evaluation information includes at least one of overall information of the image and the description information, quality information of the description information, matching information between the image and the description information, object detailed information, semantic understanding information, description information or chart explanation information.
4. The method according to claim 1, characterized in that: The image text data quality enhancement model matches the image and the initial description information to obtain evaluation information in the following manner: The image text data quality enhancement model extracts feature information of the image and feature information of the initial description information, and matches the feature information of the image and feature information of the initial description information to obtain the evaluation information.
5. The method according to claim 1, characterized in that The image text data quality enhancement model modifies the initial description information based on the image in the following manner to obtain enhanced description information of the image, and uses the enhanced description information and the evaluation information as the output result: The image text data quality enhancement model modifies the initial description information based on the feature information of the image to obtain enhanced description information of the image, wherein the enhanced description information of the image describes the feature information of the image; The enhanced description information and the evaluation information are used as the output result.
6. The method according to claim 1, characterized in that The image text data quality enhancement model is a model obtained by training based on an initial sample and a comparison sample; the image in the comparison sample is the same as the image in the initial sample, and the matching degree between the description information and the image in the comparison sample is different from the matching degree between the description information and the image in the initial sample; wherein, when the matching degree between the image and the description information in the initial sample is high, the matching degree between the description information and the image in the comparison sample is low, and when the matching degree between the image and the description information in the initial sample is low, the matching degree between the description information and the image in the comparison sample is high.
7. A training method for an image text data quality enhancement model, characterized in that: include: Get the first stage training samples; The initial model is trained based on the first-stage training samples to update the initial model to an image-text feature alignment model; the image-text feature alignment model is used to identify feature information of the image and feature information of the description information in the input information, and match the feature information of the image with the feature information of the description information; Acquire a second-stage training sample, wherein the second-stage training sample includes an initial sample and a comparison sample; the image in the comparison sample is the same as the image in the initial sample, and the matching degree between the description information in the comparison sample and the image is different from the matching degree between the description information in the initial sample and the image; wherein, when the matching degree between the image and the description information in the initial sample is high, the matching degree between the description information in the comparison sample and the image is low, and when the matching degree between the image and the description information in the initial sample is low, the matching degree between the description information in the comparison sample and the image is high; The image-text feature alignment model is trained based on the second-stage training samples to update the image-text feature alignment model to an image-text data quality enhancement model, so that when the degree of match between the input image and the initial description information is low, the image-text data quality enhancement model corrects the initial description information based on the image to obtain enhanced description information of the image with a high degree of match with the image, so as to determine the image and the enhanced description information of the image as training data for the MLLM model.
8. The method according to claim 7, characterized in that The obtaining of the second stage training samples comprises: Acquire an initial sample; the initial sample includes an image, description information of the image, and evaluation information of the description information; A comparison sample is obtained based on the image in the initial sample, the description information of the image, and the evaluation information of the description information.
9. A data processing device, characterized in that: include: An acquisition unit, used for acquiring an image and initial description information of the image; A processing unit is used to input the image and the initial description information into an image text data quality enhancement model to obtain an output result of the image text data quality enhancement model; the image text data quality enhancement model is used to match the image and the initial description information to obtain evaluation information, and when the evaluation information indicates that the initial description information has a low matching degree with the image, the initial description information is corrected based on the image to obtain enhanced description information of the image, and the enhanced description information and the evaluation information are used as the output result; When the evaluation information indicates that the initial description information has a high degree of match with the image, the evaluation information is used as an output result; The processing unit is also used to determine the training data of the multimodal large language model (MLLM) model based on the output result of the image text data quality enhancement model; wherein, when the output result is the evaluation information, the image and the initial description information of the image are determined as the training data of the MLLM model; when the output result includes the evaluation information and the enhanced description information of the image, the image and the enhanced description information of the image are determined as the training data of the MLLM model.
10. A training device for an image text data quality enhancement model, characterized in that: include: An acquisition unit, used to acquire training samples in the first stage; A processing unit, configured to train the initial model based on the first-stage training samples to update the initial model to an image-text feature alignment model; the image-text feature alignment model is configured to identify feature information of the image and feature information of the description information in the input information, and to align the feature information of the image with the feature information of the description information; The acquisition unit is further used to acquire a second-stage training sample, wherein the second-stage training sample includes an initial sample and a comparison sample; the image in the comparison sample is the same as the image in the initial sample, and the matching degree between the description information in the comparison sample and the image is different from the matching degree between the description information in the initial sample and the image; wherein, when the matching degree between the image and the description information in the initial sample is high, the matching degree between the description information in the comparison sample and the image is low, and when the matching degree between the image and the description information in the initial sample is low, the matching degree between the description information in the comparison sample and the image is high; The processing unit is also used to train the image-text feature alignment model based on the second-stage training samples to update the image-text feature alignment model to an image-text data quality enhancement model, so that when the image-text data quality enhancement model has a low matching degree between the input image and the initial description information, the initial description information is corrected based on the image to obtain enhanced description information of the image with a high matching degree with the image, so as to determine the image and the enhanced description information of the image as training data of the MLLM model.
11. An electronic device, characterized in that: The invention comprises a memory for storing computer program instructions and a processor for executing the program instructions, wherein when the computer program instructions are executed by the processor, the electronic device executes the method according to any one of claims 1 to 6 or the method according to claim 7 or 8.
12. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored program, wherein when the program is executed, the device where the computer-readable storage medium is located is controlled to execute the method according to any one of claims 1 to 6 or the method according to claim 7 or 8.
Citation Information
Patent Citations
Embryo evaluation method and device based on multi-modal data
CN116434841A
Image generation method and data processing method for image generation
CN117409109A
Cited By
Newborn fundus image description generation method and imaging method
CN119992242A
Neonatal fundus image description generation method and imaging method
CN119992242B