Data cleaning method, device, electronic device and storage medium

Abstract

The present invention provides a data cleaning method, device, electronic device, and storage medium. The method includes: obtaining a target dataset; determining category constraints based on the categories included in the target dataset; and dividing the target dataset into multiple sub-datasets; performing the following steps for any sub-dataset: using the sub-dataset as the sub-dataset to be cleaned and the other sub-datasets as training sets; training a pre-visual language model based on the training set; inputting the images and category constraints in the sub-dataset to be cleaned into the pre-visual language model to obtain pseudo-labels for the images in the sub-dataset to be cleaned; inputting the images, labels, and pseudo-labels in the sub-dataset to be cleaned as triplets into a post-visual language model to output a determination result for the images in the sub-dataset to be cleaned; and performing a cleaning operation on the sub-dataset to be cleaned based on the determination result. The present invention can effectively identify problems such as labeling errors, uneven multi-category labeling, and the absence of a clear category main body.

Description

Technical Field

[0001] The present invention relates to the technical field of image data processing, and in particular to a data cleaning method, device, electronic device and storage medium. Background Art

[0002] A visual language model (VLM) uses multimodal techniques to combine visual and textual inputs or outputs. With the emergence of large-scale language models based on the self-attention mechanism and deep learning Transformer architecture, VLMs have rapidly developed and demonstrated excellent performance on a wide range of visual tasks. Currently, mainstream VLMs generally use convolutional neural networks or vision transformers to encode input images to achieve image perception. If the model has the corresponding generation capabilities, a corresponding decoding network is also used to achieve the desired task.

[0003] Object recognition based on image classification is a fundamental task in computer vision and a fundamental topic in artificial intelligence. Improving model performance in image classification requires more high-quality datasets. However, existing public datasets (especially early ones) often suffer from numerous class labeling errors, inconsistent multi-category labeling, and the absence of distinct class entities. Furthermore, existing data labeling tasks are often performed manually, which is relatively costly. Summary of the Invention

[0004] Embodiments of the present invention provide a data cleaning method, device, electronic device, and storage medium to address the problems of incorrect category labeling, incomplete labeling, and lack of obvious category entities in currently available public data sets.

[0005] In a first aspect, an embodiment of the present invention provides a data cleaning method, comprising:

[0006] Obtain a target dataset; wherein the target dataset includes images and labels corresponding to the images;

[0007] Determine the category constraints based on the categories included in the target dataset; divide the target dataset into multiple sub-datasets; and perform the following steps for any sub-dataset:

[0008] This sub-dataset is used as the sub-dataset to be cleaned, and the other sub-datasets are used as training sets; the pre-visual language model is trained based on the training set;

[0009] Input the images and category constraints in the sub-dataset to be cleaned into the pre-visual language model to obtain pseudo labels for the images in the sub-dataset to be cleaned;

[0010] The images, labels and pseudo-labels in the sub-dataset to be cleaned are input as triplets into the post-visual language model, and the judgment results of the images in the sub-dataset to be cleaned are output; based on the judgment results, the sub-dataset to be cleaned is cleaned; wherein the judgment results are used to indicate whether the images in the sub-dataset to be cleaned match their corresponding labels and pseudo-labels.

[0011] In one possible implementation, the images and category constraints in the sub-dataset to be cleaned are input into a pre-processed visual language model to obtain pseudo labels for the images in the sub-dataset to be cleaned, including:

[0012] The images and category constraints in the sub-dataset to be cleaned are input into a pre-processed visual language model. The pre-processed visual language model is used to extract representation information of the images in the sub-dataset to be cleaned, and the representation information is used as the first image feature code. Based on the first image feature code and the category constraints, pseudo labels of the images in the sub-dataset to be cleaned are obtained.

[0013] In one possible implementation, the front-end visual language model includes a Vision-Transformer encoder;

[0014] The Vision-Transformer encoder is used to receive images in the sub-dataset to be cleaned, map the images included in the sub-dataset to be cleaned to a preset feature space, obtain representation information of the images included in the sub-dataset to be cleaned, and encode the representation information of the images as the first image feature.

[0015] In one possible implementation, the front visual language model includes a first bidirectional Transformer encoder, a second bidirectional Transformer encoder, a third bidirectional Transformer encoder, and a bidirectional Transformer decoder; wherein weights are shared among the first bidirectional Transformer encoder, the second bidirectional Transformer encoder, the third bidirectional Transformer encoder, and the bidirectional Transformer decoder;

[0016] According to the first image feature encoding and category constraints, the pseudo labels of the images in the sub-dataset to be cleaned are obtained, including:

[0017] Inputting the first image feature code and the image query code into a first bidirectional Transformer encoder, and obtaining a first original feature code through the first bidirectional Transformer encoder;

[0018] Input the category constraint into the second bidirectional Transformer encoder, and generate the category text feature encoding through the second bidirectional Transformer encoder;

[0019] Inputting the first original feature code and the category text query code into a third bidirectional Transformer encoder, and obtaining a distilled first feature code through the third bidirectional Transformer encoder and the category text feature code;

[0020] The first feature code after distillation is input into the bidirectional Transformer decoder, and the text feature corresponding to the first feature code after distillation is obtained through the bidirectional Transformer decoder;

[0021] The text features are used as pseudo labels for the images in the sub-dataset to be cleaned.

[0022] In one possible implementation, the images, labels, and pseudo-labels in the sub-dataset to be cleaned are input as triplets into a post-visual language model, and a judgment result of the images in the sub-dataset to be cleaned is output; based on the judgment result, a cleaning operation is performed on the sub-dataset to be cleaned, including:

[0023] The images, labels, and pseudo-labels in the sub-dataset to be cleaned are input as triplets into the post-visual language model; the post-visual language model is enabled to identify the category of the objects contained in the input image based on the category constraint, determine the matching relationship between the images in the sub-dataset to be cleaned and the labels and pseudo-labels according to the preset text prompts, and output the determination result of the images in the sub-dataset to be cleaned;

[0024] If the image in the sub-dataset to be cleaned matches the label but does not match the pseudo-label, then the label of the image is determined to be correct;

[0025] If the image in the sub-dataset to be cleaned matches the label and the pseudo-label, it is determined that there are multiple correct labels for the image, and the image is re-labeled according to the label and pseudo-label corresponding to the image; wherein the label is different from the pseudo-label;

[0026] If the image in the sub-dataset to be cleaned does not match the label but matches the pseudo label, the label of the image is determined to be wrong, and the image is re-labeled according to the pseudo label corresponding to the image.

[0027] In one possible implementation, training a pre-processed visual language model based on a training set includes:

[0028] Input the image in the training set into the front visual language model to obtain the second image feature encoding, and input the category constraint into the front visual language model to obtain the category label text encoding;

[0029] Obtaining a second original feature code according to the second image feature code;

[0030] Obtain a distilled second feature code based on the second original feature code and the category label text code;

[0031] Dimension compression is performed on the category label text encoding and the second feature encoding after distillation to obtain the category label text encoding features and the second feature encoding features after distillation;

[0032] Compute contrast loss and text-image matching loss based on the category label text encoding features and the distilled second feature encoding features;

[0033] Obtain a pseudo label text corresponding to the distilled second feature code according to the distilled second feature code;

[0034] Calculate language model loss based on pseudo-labeled text;

[0035] Determine a first loss function based on contrast loss and language model loss; when the first loss function converges, obtain a trained pre-visual language model;

[0036] Alternatively, a second loss function is determined based on contrast loss, text-image matching loss, and language model loss; when the second loss function converges, a trained pre-visual language model is obtained.

[0037] In one possible implementation, after inputting the images, labels, and pseudo-labels in the sub-dataset to be cleaned as triplets into a post-visual language model and outputting a judgment result of the images in the sub-dataset to be cleaned, and performing a cleaning operation on the sub-dataset to be cleaned according to the judgment result, the method further includes:

[0038] Determine whether there is a sub-dataset that has not been cleaned. If so, use the sub-dataset as the sub-dataset to be cleaned, and return to use the other sub-datasets as training sets; train the pre-visual language model based on the training set until there is no sub-dataset that has not been cleaned.

[0039] In a second aspect, an embodiment of the present invention provides a data cleaning device, comprising:

[0040] An acquisition module is used to acquire a target data set; wherein the target data set includes images and labels corresponding to the images;

[0041] The determination module is used to determine the category constraints according to the categories included in the target dataset; and divide the target dataset into multiple sub-datasets; and perform the following steps for any sub-dataset:

[0042] The training module is used to use the sub-dataset as the sub-dataset to be cleaned and the other sub-datasets as the training set; the pre-visual language model is trained based on the training set;

[0043] The pseudo-label generation module is used to input the images and category constraints in the sub-dataset to be cleaned into the front-end visual language model to obtain pseudo labels for the images in the sub-dataset to be cleaned;

[0044] The analysis module is used to input the images, labels and pseudo-labels in the sub-dataset to be cleaned as triplets into the post-visual language model, and output the judgment results of the images in the sub-dataset to be cleaned; based on the judgment results, the sub-dataset to be cleaned is cleaned; wherein the judgment results are used to indicate whether the images in the sub-dataset to be cleaned match their corresponding labels and pseudo-labels.

[0045] In a third aspect, an embodiment of the present invention provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the steps of the method described in the first aspect or any possible implementation of the first aspect are implemented.

[0046] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the steps of the method described in the first aspect or any possible implementation of the first aspect.

[0047] Embodiments of the present invention provide a data cleaning method, apparatus, electronic device, and storage medium. By dividing a dataset to be cleaned into multiple sub-datasets and training a corresponding pre-trained visual language model for each sub-dataset, this method can prevent a unified model from generating memory and affecting the final data cleaning results. For each sub-dataset to be cleaned, matching is performed based on the images, labels, and pseudo-labels in that sub-dataset. This method can effectively identify labeling errors, inconsistent multi-category labeling, and the absence of a clear category main body. Furthermore, it eliminates the need for manual identification, reducing labor costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments or related technical descriptions. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0049] Figure 1 This is a flow chart of an implementation of a data cleaning method provided by an embodiment of the present invention;

[0050] Figure 2 This is a system architecture diagram of the data cleaning method provided by an embodiment of the present invention;

[0051] Figure 3This is a diagram of a pre-visual language model framework of a data cleaning method provided by an embodiment of the present invention;

[0052] Figure 4 is a flow chart of an implementation of a data cleaning method provided by another embodiment of the present invention;

[0053] Figure 5 Schematic diagram of the structure of a data cleaning device provided by an embodiment of the present invention;

[0054] Figure 6 is a schematic diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0055] In the following description, specific details such as particular system structures and techniques are provided for purposes of illustration, not limitation, to facilitate a thorough understanding of the embodiments of the present invention. However, it will be apparent to those skilled in the art that the present invention may be practiced in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted so as not to obscure the description of the present invention with unnecessary detail.

[0056] In order to make the purpose, technical solutions and advantages of the present invention more clear, specific embodiments will be described below with reference to the accompanying drawings.

[0057] Figure 1 This is a flow chart of the implementation of the data cleaning method provided by the embodiment of the present invention; Figure 1 As shown:

[0058] Step 110: Obtain a target dataset; wherein the target dataset includes images and labels corresponding to the images.

[0059] In this embodiment, the target dataset can be a classic dataset widely used in the fields of computer vision and machine learning, a common dataset for image classification tasks, a dataset for image classification tasks, a large-scale image database, a common dataset for object recognition and segmentation, a small dataset for image classification tasks, a large-scale dataset for urban scene understanding, or a dataset including facial images, etc., without limitation. The target dataset can include multiple images and labels corresponding to each image.

[0060] Step 120: Determine category constraints based on the categories included in the target dataset; divide the target dataset into multiple sub-datasets; and execute the steps involved in step 130 for any sub-dataset. Step 130 includes steps 131, 132, and 133.

[0061] In this embodiment, the category constraints can be viewed as a collection of categories annotated in the target dataset, and are used to limit the model's recognition scope during data cleaning. For example, the target dataset may include four categories: people, cats, dogs, and cars. However, some images in the target dataset also include other elements, such as boats and trees, which are not labeled in the target dataset. Therefore, during data cleaning, category constraints can be used to limit the object categories included in the images recognized by the model during data cleaning.

[0062] The traditional data cleaning method is to analyze and clean all the image information in the data set at the same time. However, as the model is used, it will generate memory, which will affect the data cleaning effect. Therefore, this embodiment divides the target data set into multiple sub-datasets, analyzes each sub-dataset separately, and trains a corresponding pre-visual language model for each sub-dataset. This can avoid the unified model from generating memory and affecting the final data cleaning results. Specifically, the analysis method is as follows: Step 131 to Step 133:

[0063] Step 131: Use the sub-dataset as the sub-dataset to be cleaned, and the other sub-datasets as training sets; and train the pre-visual language model based on the training sets.

[0064] Exemplarily, the target dataset can be a VLCS dataset, which is a dataset commonly used for domain generalization. The target dataset can be divided into four domains, that is, into four sub-datasets A, B, C, and D. A can be the sub-dataset to be cleaned, and B, C, and D are training sets for training the front-end visual language model. During training, any two sub-datasets of B, C, and D can be used as training sets, and the remaining one can be used as a test set. When the preset loss function in the front-end visual language model converges, the training of the front-end visual language model is completed, and a trained front-end visual language model is obtained.

[0065] Step 132: Input the images and category constraints in the sub-dataset to be cleaned into the pre-visual language model to obtain pseudo labels for the images in the sub-dataset to be cleaned.

[0066] Figure 2 This is a system architecture diagram of the data cleaning method provided by the embodiment of the present invention; Figure 2 This embodiment is described. For example, the pre-visual language model may include a pre-feature extraction module, a feature distillation module, and a pre-text generation module. Each sub-dataset includes multiple images to be detected; taking an image as an example, when the image and category constraints in the sub-dataset to be cleaned are input into the pre-visual language model, the pseudo-label corresponding to the image in the sub-dataset to be cleaned can be obtained through the various modules in the pre-visual language model under the constraints of the category constraints.

[0067] Specifically, in some embodiments, step 132 may include:

[0068] The images and category constraints in the sub-dataset to be cleaned are input into a pre-processed visual language model. The pre-processed visual language model is used to extract representation information of the images in the sub-dataset to be cleaned, and the representation information is used as the first image feature code. Based on the first image feature code and the category constraints, pseudo labels of the images in the sub-dataset to be cleaned are obtained.

[0069] In this embodiment, after the images and category constraints in the sub-dataset to be cleaned are input into the front-end visual language model, the front-end visual language model performs the following operations:

[0070] The pre-processing feature extraction module extracts representational information for each image in the sub-dataset to be cleaned, subject to pre-set category constraints. This information is then encoded as the first image feature of each image and fed into the feature distillation module. Under the category constraints, the feature distillation module, in conjunction with the pre-processing text generation module, processes the first image feature encodings into pseudo labels for the image.

[0071] Figure 3 This is a diagram of the front-end visual language model framework of the data cleaning method provided by the embodiment of the present invention, referring to Figure 3 In some embodiments, the front-end visual language model includes a Vision-Transformer encoder.

[0072] The Vision-Transformer encoder is used to receive images in the sub-dataset to be cleaned, map the images included in the sub-dataset to be cleaned to a preset feature space, obtain representation information of the images included in the sub-dataset to be cleaned, and encode the representation information of the images as the first image feature.

[0073] The following combination Figure 3 This embodiment is described. In this embodiment, the front visual language model can be a model improved based on BLIP2, which can include a Vision-Transformer encoder. After the image and category constraints in the sub-dataset to be cleaned are input into the front visual language model, the front visual language model preprocesses the input image, and then the Vision-Transformer encoder extracts the representation information of the image based on the received image, and then outputs the representation information as the first image coding feature of the image.

[0074] Specifically, in some embodiments, the front-end visual language model includes a first bidirectional Transformer encoder, a second bidirectional Transformer encoder, a third bidirectional Transformer encoder, and a bidirectional Transformer decoder, wherein weights are shared among the first bidirectional Transformer encoder, the second bidirectional Transformer encoder, the third bidirectional Transformer encoder, and the bidirectional Transformer decoder.

[0075] The above-mentioned process of obtaining pseudo labels of images in the sub-dataset to be cleaned based on the first image feature encoding and the category constraint condition may include:

[0076] The first image feature code and the image query code are input into a first bidirectional Transformer encoder, and a first original feature code is obtained through the first bidirectional Transformer encoder.

[0077] The category constraint is input into the second bidirectional Transformer encoder, and the category text feature encoding is generated by the second bidirectional Transformer encoder.

[0078] The first original feature code and the category text query code are input into the third bidirectional Transformer encoder, and the distilled first feature code is obtained through the third bidirectional Transformer encoder and the category text feature code.

[0079] The first feature code after distillation is input into the bidirectional Transformer decoder, and the text feature corresponding to the first feature code after distillation is obtained through the bidirectional Transformer decoder.

[0080] The text features are used as pseudo labels for the images in the sub-dataset to be cleaned.

[0081] like Figure 3 As shown, Figure 3 The solid arrows in the figure represent the direction of data transmission, and the dotted arrows represent the weight sharing between the bidirectional Transformer encoders. In this embodiment, the front-end visual language model includes three bidirectional Transformer encoders and one bidirectional Transformer decoder. Pseudo-labels can be obtained by coordinating the bidirectional Transformer encoders. Specifically:

[0082] First, when the Vision-Transformer encoder outputs the first image feature code to the first bidirectional Transformer encoder, the front visual language model controls the image query code to be input into the first bidirectional Transformer encoder. The first bidirectional Transformer encoder calculates and outputs the first original feature code of the image based on the first image feature code, the image query code, and the image attention layer. Among them, the image query code is a parameter built into the front visual language model. It is a frozen parameter and does not participate in training. The image attention layer is a preset parameter built into the front visual language model, indicating whether mutual query is allowed. If allowed, it is 1.

[0083] Correspondingly, the expression of the first original feature code can be:

[0084] I f3ats =BERT(Q img ,I emb ,Att img )

[0085] Among them, I feats Q is the first original feature code; img Encoding image queries in the Vision-Transformer encoder; I emb Encode the first image feature; Att img is the image attention layer in the Vision-Transformer encoder.

[0086] Secondly, to avoid model recognition errors during the data cleaning process, this embodiment needs to input the category constraint conditions into the second bidirectional Transformer encoder. The second bidirectional Transformer encoder encodes the category constraint conditions and outputs the category text feature code.

[0087] In order to meet the input requirements of the BERT network (i.e., the bidirectional Transformer encoder), the first original feature encoding needs to be re-dimensionalized. For example, based on the network size of the Vision-Transformer encoder selected above, the re-mapping dimension input can be set to 768 and the output dimension can be set to 1408. This mapping can be implemented by a linear network or a multi-layer network.

[0088] The second bidirectional Transformer encoder then outputs the first raw feature code and transmits it to the third bidirectional Transformer encoder. The third bidirectional Transformer encoder outputs a processed first raw feature code based on the input first raw feature code and the category text query code. The processed first raw feature code and the category text feature code are distilled to produce a distilled first feature code. The category text query code is a trainable parameter built into the pre-trained visual language model and is obtained through a training process.

[0089] Correspondingly, the expression of the first feature code after distillation can be:

[0090] I distill =BERT(Q distill ,I feats ,Att distill )

[0091] Among them, Q distill Encodes the image query in the third bidirectional Transformer encoder; Att distill is the image attention layer in the third bidirectional Transformer encoder.

[0092] Finally, the third bidirectional Transformer encoder inputs the distilled first feature code into the bidirectional Transformer decoder, which decodes it and obtains and outputs the text feature corresponding to the distilled first feature code, which is the pseudo label corresponding to the image.

[0093] Step 133: Input the images, labels, and pseudo-labels in the sub-dataset to be cleaned as triplets into the post-visual language model, and output the judgment result of the images in the sub-dataset to be cleaned; based on the judgment result, perform a cleaning operation on the sub-dataset to be cleaned; wherein the judgment result is used to indicate whether the images in the sub-dataset to be cleaned match their corresponding labels and pseudo-labels.

[0094] In this embodiment, referring to Figure 2 After obtaining the pseudo-label corresponding to the image to be detected (any image in the sub-dataset to be cleaned), the image, its corresponding label, and its corresponding pseudo-label are input as a triplet into the post-visual language model. Under the constraints of the category constraints, the post-visual language model can output the judgment result of the image to be detected based on its preset text prompts. This determines whether the image to be detected has label errors or incomplete labels, so that the corresponding cleaning operation can be performed on the image to be detected based on the judgment result.

[0095] In some embodiments, step 133 may include:

[0096] The images, labels and pseudo-labels in the sub-dataset to be cleaned are input as triplets into the post-visual language model; the post-visual language model is enabled to identify the category of the objects contained in the input image based on the category constraint, and to judge the matching relationship between the images in the sub-dataset to be cleaned and the labels and pseudo-labels according to the preset text prompts, and output the judgment results of the images in the sub-dataset to be cleaned.

[0097] If the image in the sub-dataset to be cleaned matches the label but does not match the pseudo label, then the label of the image is determined to be correct.

[0098] If the image in the sub-dataset to be cleaned matches the label and the pseudo-label, it is determined that there are multiple correct labels for the image, and the image is re-labeled according to the label and pseudo-label corresponding to the image; wherein the label is different from the pseudo-label.

[0099] If the image in the sub-dataset to be cleaned does not match the label but matches the pseudo label, the label of the image is determined to be wrong, and the image is re-labeled according to the pseudo label corresponding to the image.

[0100] In this embodiment, the post-visual language model is any one of the open source models, which includes multiple preset text prompts, and the text prompt selection module can select the corresponding text prompt according to the input image, and make corresponding selections based on the text prompts, thereby being able to identify the category of the object included in the input image and determine whether the category of the object included in the image matches its corresponding label or pseudo-label.

[0101] When the image is input into the post-visual language model, the text prompt selection module selects the corresponding text prompt. The model responds to the text prompt, and the text prompt selection module selects a new text prompt based on the response. This cycle continues until the final judgment result is output.

[0102] If the image in the sub-dataset to be cleaned matches the label but does not match the pseudo-label, it is determined that the label of the image is correct and no further processing is required for the image.

[0103] If the image in the sub-dataset to be cleaned matches the label and the pseudo-label, and the label is different from the pseudo-label, it is determined that the image has multiple correct labels. Then it is necessary to add the pseudo-label corresponding to the image to the label of the image to obtain the new label of the image, so that the new label of the image includes the label and the pseudo-label of the image.

[0104] If the image in the sub-dataset to be cleaned does not match the label but matches the pseudo-label, it is determined that the label of the image is wrong. In this case, the label corresponding to the image needs to be deleted, and the pseudo-label corresponding to the image is used as the new label of the image to re-label the image.

[0105] For example, if only people appear in the image, the label is person and the pseudo label is car, it means that the label of the image is correct and there is no need to process the image.

[0106] If a person and a vehicle appear in an image at the same time, the label is person and the pseudo label is vehicle, it means that there are multiple correct labels for the image, so the content of the pseudo label needs to be added to the label content.

[0107] If only people appear in the image, the label is car, and the pseudo label is person, then it is determined that the label of the image is wrong. The label of the image needs to be deleted, and then the pseudo label is used as the new label to re-label the image.

[0108] If only trees appear in the image, but the label is a car and the pseudo-label is a person, then the image is determined to be incorrect and needs to be deleted from the target dataset.

[0109] In some embodiments, the post-visual language model may also output the determination process when outputting the determination result.

[0110] For example, Figure 2 As shown in the figure, assume that the label corresponding to the image is "person" and the pseudo label is "car". "Q" represents the text prompt content selected by the text prompt selection module; "A" represents the answer to the text prompt. The judgment process can be:

[0111] Q: “What can you see in this picture?”

[0112] A: "This picture shows two cars and some people, including two girls, one holding the other's shoulders."

[0113] Q: “This picture is labeled person. Do you think it’s reasonable?”

[0114] A: "This image is labeled 'person', which describes the person in the image and reasonably expresses the content."

[0115] Q: “Do you think it’s reasonable for a not-so-smart model to think this image should be a car?”

[0116] A: "The category of this image should be vehicle, and the description of the vehicle in the picture reasonably expresses this content."

[0117] Q: “For this picture, which one do you think is more reasonable, person or car, or are both reasonable?”

[0118] A: "For this image, both person and car are reasonable because there are both vehicles and people in the image."

[0119] In some embodiments, in step 133, after inputting the images, labels, and pseudo-labels in the sub-dataset to be cleaned as triplets into the post-visual language model and outputting the judgment result of the images in the sub-dataset to be cleaned, the following steps may be further performed after performing the cleaning operation on the sub-dataset to be cleaned based on the judgment result:

[0120] Determine whether there is a sub-dataset that has not been cleaned. If so, use the sub-dataset as the sub-dataset to be cleaned, and return to use the other sub-datasets as training sets; train the pre-visual language model based on the training set until there is no sub-dataset that has not been cleaned.

[0121] In this embodiment, after the current sub-dataset to be cleaned completes data cleaning, it can be determined whether there is a sub-dataset that has not been cleaned. If so, the sub-dataset that has not been cleaned is used as a new sub-dataset to be cleaned, and the process jumps to step 130, and steps 131 to 133 in step 130 are executed again until all sub-datasets are cleaned.

[0122] In some embodiments, training a pre-processed visual language model based on a training set may include:

[0123] Input the image in the training set into the front visual language model to obtain the second image feature encoding, and input the category constraint into the front visual language model to obtain the category label text encoding;

[0124] Obtaining a second original feature code according to the second image feature code;

[0125] Obtain a distilled second feature code based on the second original feature code and the category label text code;

[0126] Dimension compression is performed on the category label text encoding and the second feature encoding after distillation to obtain the category label text encoding features and the second feature encoding features after distillation;

[0127] Compute contrast loss and text-image matching loss based on the category label text encoding features and the distilled second feature encoding features;

[0128] Obtain a pseudo label text corresponding to the distilled second feature code according to the distilled second feature code;

[0129] Calculate language model loss based on pseudo-labeled text;

[0130] Determine a first loss function based on contrast loss and language model loss; when the first loss function converges, obtain a trained pre-visual language model;

[0131] Alternatively, a second loss function is determined based on contrast loss, text-image matching loss, and language model loss; when the second loss function converges, a trained pre-visual language model is obtained.

[0132] In this embodiment, the calculation process of the second original feature code, the second feature code after distillation and other data can refer to the above-mentioned related embodiments and will not be described in detail again.

[0133] In this embodiment, when the loss function converges, it indicates that the training of the pre-visual language model is complete, and the obtained model can be used for subsequent data cleaning. There are two types of loss functions, and the pre-visual language models obtained based on the two loss functions are the same in use.

[0134] Among them, the first loss function can be: Loss1=Loss itc +Loss lm

[0135] Where, Loss itc Loss is the contrast loss; lm is the language model loss.

[0136] Loss itc It is a contrast loss based on cosine similarity. The comparison form is matrix expansion multiplication followed by selecting the text encoding feature with the largest similarity as the prediction, and the loss calculation form is the cross entropy loss function.

[0137] The second loss function can be: Loss2=Loss itc +Loss itm +Loss lm

[0138] Where, Loss itm is the text-image matching loss. It can be calculated by selecting a pair of positive and negative text samples, that is, performing a binary classification match on each of the positive and negative text samples based on the distilled second feature encoding (or the distilled second feature encoding feature). The loss is calculated as the cross-entropy loss function.

[0139] Figure 4 This is a flow chart of a data cleaning method according to another embodiment of the present invention. Figure 4 As shown:

[0140] In this embodiment, a selected dataset is first obtained. Based on this dataset, corresponding category text constraints are determined. The dataset is then divided into N sub-datasets. One of the uncleaned sub-datasets is selected as the sub-dataset to be cleaned. One of the remaining sub-datasets is selected as the test set, and the others are used as training sets. The pre-trained model is trained until a preset loss function converges, resulting in a fully trained pre-trained model.

[0141] The images in the sub-dataset to be cleaned are then fed into the pre-trained model for visual feature extraction (representational information extraction). After alignment and distillation of the extracted visual features, pseudo-labels (in this case, in text form) are obtained for the images in the sub-dataset to be cleaned.

[0142] Afterwards, the images in the sub-dataset to be cleaned are input into the post-model together with the images, the labels corresponding to the images, and the pseudo labels corresponding to the images to generate the final judgment results of the images in the sub-dataset to be cleaned, and data cleaning is performed on the images in the sub-dataset to be cleaned based on the results.

[0143] Finally, determine whether there are any uncleaned sub-datasets in the dataset. If so, retrain the model and clean it. Otherwise, end.

[0144] In summary, by dividing the dataset to be cleaned into multiple sub-datasets and training a corresponding pre-trained visual language model for each sub-dataset, this embodiment of the present invention can prevent a unified model from incurring memory and affecting the final data cleaning results. By matching each sub-dataset to be cleaned based on the images, labels, and pseudo-labels within that sub-dataset, the method can effectively identify labeling errors, inconsistent multi-category labeling, and the absence of a clear category main body. Furthermore, it eliminates the need for manual identification, reducing labor costs.

[0145] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0146] The following are device embodiments of the present invention. For details not fully described therein, reference may be made to the corresponding method embodiments described above.

[0147] Figure 5 A schematic diagram of the structure of a data cleaning device provided by an embodiment of the present invention is shown. For ease of explanation, only the parts related to the embodiment of the present invention are shown, which are described in detail as follows:

[0148] like Figure 5 As shown, the data cleaning device 5 includes: an acquisition module, which is used to acquire a target data set; wherein the target data set includes images and labels corresponding to the images;

[0149] The determination module 51 is configured to determine category constraints based on the categories included in the target dataset; divide the target dataset into multiple sub-datasets; and perform the following steps on any sub-dataset:

[0150] The training module 52 is configured to use the sub-dataset as the sub-dataset to be cleaned and the other sub-datasets as training sets; and train the pre-visual language model based on the training sets;

[0151] A pseudo label generation module 53 is used to input the images and category constraints in the sub-dataset to be cleaned into the pre-visual language model to obtain pseudo labels for the images in the sub-dataset to be cleaned;

[0152] The analysis module 54 is used to input the images, labels, and pseudo-labels in the sub-dataset to be cleaned as triplets into the post-visual language model, and output the judgment result of the images in the sub-dataset to be cleaned; based on the judgment result, the sub-dataset to be cleaned is cleaned; wherein the judgment result is used to indicate whether the images in the sub-dataset to be cleaned match their corresponding labels and pseudo-labels.

[0153] In a possible implementation, the pseudo-label generating module 53 is specifically configured to:

[0154] The images and category constraints in the sub-dataset to be cleaned are input into a pre-processed visual language model. The pre-processed visual language model is used to extract representation information of the images in the sub-dataset to be cleaned, and the representation information is used as the first image feature code. Based on the first image feature code and the category constraints, pseudo labels of the images in the sub-dataset to be cleaned are obtained.

[0155] In one possible implementation, the front-end visual language model includes a Vision-Transformer encoder;

[0156] The Vision-Transformer encoder is used to receive images in the sub-dataset to be cleaned, map the images included in the sub-dataset to be cleaned to a preset feature space, obtain representation information of the images included in the sub-dataset to be cleaned, and encode the representation information of the images as the first image feature.

[0157] In one possible implementation, the front visual language model includes a first bidirectional Transformer encoder, a second bidirectional Transformer encoder, a third bidirectional Transformer encoder, and a bidirectional Transformer decoder;

[0158] The pseudo-label generation module 53 is specifically used for:

[0159] Inputting the first image feature code and the image query code into a first bidirectional Transformer encoder, and obtaining a first original feature code through the first bidirectional Transformer encoder;

[0160] Input the category constraint into the second bidirectional Transformer encoder, and generate the category text feature encoding through the second bidirectional Transformer encoder;

[0161] Inputting the first original feature code and the category text query code into a third bidirectional Transformer encoder, and obtaining a distilled first feature code through the third bidirectional Transformer encoder and the category text feature code;

[0162] The first feature code after distillation is input into the bidirectional Transformer decoder, and the text feature corresponding to the first feature code after distillation is obtained through the bidirectional Transformer decoder;

[0163] The text features are used as pseudo labels for the images in the sub-dataset to be cleaned.

[0164] In a possible implementation, the analysis module 54 is specifically configured to:

[0165] The images, labels, and pseudo-labels in the sub-dataset to be cleaned are input as triplets into the post-visual language model; the post-visual language model is enabled to identify the category of the objects contained in the input image based on the category constraint, determine the matching relationship between the images in the sub-dataset to be cleaned and the labels and pseudo-labels according to the preset text prompts, and output the determination result of the images in the sub-dataset to be cleaned;

[0166] If the image in the sub-dataset to be cleaned matches the label but does not match the pseudo-label, then the label of the image is determined to be correct;

[0167] If the image in the sub-dataset to be cleaned matches the label and the pseudo-label, it is determined that there are multiple correct labels for the image, and the image is re-labeled according to the label and pseudo-label corresponding to the image; wherein the label is different from the pseudo-label;

[0168] If the image in the sub-dataset to be cleaned does not match the label but matches the pseudo label, the label of the image is determined to be wrong, and the image is re-labeled according to the pseudo label corresponding to the image.

[0169] In one possible implementation, the training module is specifically used to:

[0170] Input the image in the training set into the front visual language model to obtain the second image feature encoding, and input the category constraint into the front visual language model to obtain the category label text encoding;

[0171] Obtaining a second original feature code according to the second image feature code;

[0172] Obtain a distilled second feature code based on the second original feature code and the category label text code;

[0173] Dimension compression is performed on the category label text encoding and the second feature encoding after distillation to obtain the category label text encoding features and the second feature encoding features after distillation;

[0174] Compute contrast loss and text-image matching loss based on the category label text encoding features and the distilled second feature encoding features;

[0175] Obtain a pseudo label text corresponding to the distilled second feature code according to the distilled second feature code;

[0176] Calculate language model loss based on pseudo-labeled text;

[0177] Determine a first loss function based on contrast loss and language model loss; when the first loss function converges, obtain a trained pre-visual language model;

[0178] Alternatively, a second loss function is determined based on contrast loss, text-image matching loss, and language model loss; when the second loss function converges, a trained pre-visual language model is obtained.

[0179] In a possible implementation, the device further includes a judgment module 55, which is specifically configured to:

[0180] Determine whether there is a sub-dataset that has not been cleaned. If so, use the sub-dataset as the sub-dataset to be cleaned, and return to use the other sub-datasets as training sets; train the pre-visual language model based on the training set until there is no sub-dataset that has not been cleaned.

[0181] Figure 6 Schematic diagram of an electronic device provided by an embodiment of the present invention. Figure 6 As shown, the electronic device 6 of this embodiment includes: a processor 60, a memory 61, and a computer program 62 stored in the memory 61 and executable on the processor 60. When the processor 60 executes the computer program 62, the steps in the above-mentioned data cleaning method embodiments are implemented, for example Figure 1Alternatively, when the processor 60 executes the computer program 62, the functions of the modules / units in the above-mentioned device embodiments are realized, for example Figure 5 The functions of each module are shown.

[0182] Exemplarily, the computer program 62 may be divided into one or more modules / units, which are stored in the memory 61 and executed by the processor 60 to implement the present invention. The one or more modules / units may be a series of computer program instruction segments capable of implementing specific functions, which are used to describe the execution process of the computer program 62 in the electronic device 6. For example, the computer program 62 may be divided into Figure 5 The modules shown.

[0183] The electronic device 6 can be a computing device such as a desktop computer, a notebook, a PDA, a cloud server, etc. The electronic device 6 can include, but is not limited to, a processor 60 and a memory 61. It can be understood by those skilled in the art that Figure 6 It is only an example of the electronic device 6 and does not constitute a limitation of the electronic device 6. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the electronic device may also include input and output devices, network access devices, buses, etc.

[0184] The processor 60 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.

[0185] The memory 61 can be an internal storage unit of the electronic device 6, such as a hard disk or memory of the electronic device 6. The memory 61 can also be an external storage device of the electronic device 6, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the electronic device 6. Furthermore, the memory 61 can also include both an internal storage unit of the electronic device 6 and an external storage device. The memory 61 is used to store the computer program and other programs and data required by the electronic device. The memory 61 can also be used to temporarily store data that has been output or is about to be output.

[0186] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0187] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant description of other embodiments.

[0188] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present invention.

[0189] In the embodiments provided by the present invention, it should be understood that the disclosed devices / electronic devices and methods can be implemented in other ways. For example, the device / electronic device embodiments described above are merely schematic. For example, the division of the modules or units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0190] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0191] In addition, the functional units in the various embodiments of the present invention may be integrated into a single processing unit, each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0192] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention implements all or part of the process in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by the processor, it can implement the steps of the above-mentioned various data cleaning method embodiments. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device that can carry the computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal and software distribution medium, etc.

[0193] The embodiments described above are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention, and should all be included in the scope of protection of the present invention.

Claims

1. A data cleaning method, characterized in that: include: Obtain a target data set; wherein the target data set includes images and labels corresponding to the images; Determine category constraints based on the categories included in the target dataset; divide the target dataset into multiple sub-datasets; and perform the following steps on any one of the sub-datasets: The sub-dataset is used as the sub-dataset to be cleaned, and the other sub-datasets are used as training sets; a pre-visual language model is trained based on the training sets; Inputting the images in the sub-dataset to be cleaned and the category constraints into the front-end visual language model to obtain pseudo labels for the images in the sub-dataset to be cleaned; Inputting the images, labels, and pseudo-labels in the sub-dataset to be cleaned as triplets into a post-visual language model, and outputting a determination result of the images in the sub-dataset to be cleaned; performing a cleaning operation on the sub-dataset to be cleaned based on the determination result; wherein the determination result is used to indicate whether the images in the sub-dataset to be cleaned match their corresponding labels and pseudo-labels; The step of inputting the images in the sub-dataset to be cleaned and the category constraint into the pre-visual language model to obtain pseudo labels for the images in the sub-dataset to be cleaned includes: Inputting the images in the sub-dataset to be cleaned and the category constraint into the pre-processed visual language model, extracting representation information of the images in the sub-dataset to be cleaned using the pre-processed visual language model, and encoding the representation information as a first image feature; and obtaining pseudo labels for the images in the sub-dataset to be cleaned based on the first image feature code and the category constraint; The method includes inputting the images, labels, and pseudo-labels in the sub-dataset to be cleaned as triplets into a post-visual language model, outputting a determination result of the images in the sub-dataset to be cleaned, and performing a cleaning operation on the sub-dataset to be cleaned according to the determination result, including: Inputting the images, labels, and pseudo-labels in the sub-dataset to be cleaned as triplets into a post-visual language model; enabling the post-visual language model to identify the category of the objects contained in the input image based on the category constraint, determine the matching relationship between the images in the sub-dataset to be cleaned and the labels and pseudo-labels according to preset text prompts, and output the determination result of the images in the sub-dataset to be cleaned; If the image in the sub-dataset to be cleaned matches the label but does not match the pseudo-label, it is determined that the label of the image is correct; If the image in the sub-dataset to be cleaned matches the label and the pseudo-label, it is determined that the image has multiple correct labels, and the image is re-labeled according to the label and the pseudo-label corresponding to the image; wherein the label is different from the pseudo-label; If the image in the sub-dataset to be cleaned does not match the label but matches the pseudo label, it is determined that the label of the image is wrong, and the image is re-labeled according to the pseudo label corresponding to the image.

2. The data cleaning method according to claim 1, characterized in that: The front visual language model includes a Vision-Transformer encoder; The Vision-Transformer encoder is used to receive the image in the sub-dataset to be cleaned, map the image included in the sub-dataset to be cleaned to a preset feature space, obtain representation information of the image included in the sub-dataset to be cleaned, and encode the representation information of the image as the first image feature.

3. The data cleaning method according to claim 1, wherein: The front visual language model includes a first bidirectional Transformer encoder, a second bidirectional Transformer encoder, a third bidirectional Transformer encoder and a bidirectional Transformer decoder; wherein the first bidirectional Transformer encoder, the second bidirectional Transformer encoder, the third bidirectional Transformer encoder and the bidirectional Transformer decoder share weights; Obtaining pseudo labels for images in the sub-dataset to be cleaned according to the first image feature code and the category constraint includes: Inputting the first image feature code and the image query code into the first bidirectional Transformer encoder, and obtaining a first original feature code through the first bidirectional Transformer encoder; Inputting the category constraint into the second bidirectional Transformer encoder, and generating a category text feature code through the second bidirectional Transformer encoder; Inputting the first original feature code and the category text query code into the third bidirectional Transformer encoder, and obtaining a distilled first feature code through the third bidirectional Transformer encoder and the category text feature code; Inputting the distilled first feature code into the bidirectional Transformer decoder, and obtaining text features corresponding to the distilled first feature code through the bidirectional Transformer decoder; The text features are used as pseudo labels for the images in the sub-dataset to be cleaned.

4. The data cleaning method according to claim 1, wherein: The training of the front-end visual language model based on the training set includes: Inputting the image in the training set into a front-end visual language model to obtain a second image feature code, and inputting the category constraint into the front-end visual language model to obtain a category label text code; Obtaining a second original feature code according to the second image feature code; Obtaining a distilled second feature code according to the second original feature code and the category label text code; Dimensionally compressing the category label text encoding and the distilled second feature encoding to obtain category label text encoding features and distilled second feature encoding features; Calculating contrast loss and text-image matching loss based on the category label text encoding feature and the distilled second feature encoding feature; Obtaining a pseudo label text corresponding to the distilled second feature code according to the distilled second feature code; Calculating a language model loss based on the pseudo-labeled text; Determining a first loss function based on the contrast loss and the language model loss; when the first loss function converges, obtaining a trained pre-visual language model; Alternatively, a second loss function is determined based on the contrast loss, the text-image matching loss, and the language model loss; when the second loss function converges, a trained pre-visual language model is obtained.

5. The data cleaning method according to claim 1, wherein: Inputting the images, labels and pseudo labels in the sub-dataset to be cleaned as triples into the post-visual language model, and outputting the determination results of the images in the sub-dataset to be cleaned; After performing a cleaning operation on the sub-dataset to be cleaned according to the determination result, the method further includes: Determine whether there is a sub-dataset that has not been cleaned. If so, use the sub-dataset as the sub-dataset to be cleaned and return the other sub-datasets as the training set; The step of training a front-end visual language model based on the training set until no uncleaned sub-dataset exists.

6. A data cleaning device, characterized in that: include: An acquisition module is used to acquire a target data set; wherein the target data set includes images and labels corresponding to the images; A determination module is configured to determine category constraints based on the categories included in the target dataset; divide the target dataset into multiple sub-datasets; and perform the following steps on any one of the sub-datasets: A training module is used to use the sub-dataset as the sub-dataset to be cleaned and the other sub-datasets as training sets; and train the pre-visual language model based on the training sets; a pseudo-label generation module, configured to input the images in the sub-dataset to be cleaned and the category constraints into the front-end visual language model to obtain pseudo-labels for the images in the sub-dataset to be cleaned; an analysis module, configured to input the images, labels, and pseudo-labels in the sub-dataset to be cleaned as triplets into a post-visual language model, and output a determination result of the images in the sub-dataset to be cleaned; and perform a cleaning operation on the sub-dataset to be cleaned based on the determination result; wherein the determination result is used to indicate whether the images in the sub-dataset to be cleaned and their corresponding labels and pseudo-labels match; The pseudo-label generation module is specifically configured to: Inputting the images in the sub-dataset to be cleaned and the category constraint into the pre-processed visual language model, extracting representation information of the images in the sub-dataset to be cleaned using the pre-processed visual language model, and encoding the representation information as a first image feature; and obtaining pseudo labels for the images in the sub-dataset to be cleaned based on the first image feature code and the category constraint; The analysis module is specifically used to: Inputting the images, labels, and pseudo-labels in the sub-dataset to be cleaned as triplets into a post-visual language model; enabling the post-visual language model to identify the category of the objects contained in the input image based on the category constraint, determine the matching relationship between the images in the sub-dataset to be cleaned and the labels and pseudo-labels according to preset text prompts, and output the determination result of the images in the sub-dataset to be cleaned; If the image in the sub-dataset to be cleaned matches the label but does not match the pseudo-label, it is determined that the label of the image is correct; If the image in the sub-dataset to be cleaned matches the label and the pseudo-label, it is determined that the image has multiple correct labels, and the image is re-labeled according to the label and the pseudo-label corresponding to the image; wherein the label is different from the pseudo-label; If the image in the sub-dataset to be cleaned does not match the label but matches the pseudo label, it is determined that the label of the image is wrong, and the image is re-labeled according to the pseudo label corresponding to the image.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 5 are implemented.

8. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

Images