Data cleaning method and device based on text local and global similarity filtering

Abstract

The invention provides a data cleaning method and device based on text local and global similarity filtering, and solves the problems that in the prior art, only global information is considered, and a CLIP model is poor in long text compatibility. According to the scheme, the similarity between the complete text and the image content and the similarity between each sentence and the image content in the text are considered, and the illusion content in the generated text can be effectively filtered out. Meanwhile, after the text content is split into the short sentences, the text content is just compatible with the CLIP, and the short text can be well processed. Firstly, a visual encoder and a text encoder in a CLIP model are used for encoding an image and a split text respectively, and global image features and local text features are obtained. Then calculating the similarity between the image feature and each text feature; the sentences with the similarity lower than a threshold value are filtered out of the text, the remaining sentences continue to form a complete text, and finally generated, high-quality and low-illusion image-text pair data are obtained.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a data cleaning method and device based on local and global text similarity filtering. Background Art

[0002] Due to the lack of data for real annotations and high-quality images and texts, many researchers now use existing multimodal large language models to construct large-scale high-quality image and text datasets. However, the content generated by these models generally has hallucination problems (for example, generating content that is not in the image). Directly using these generated data to train models will seriously affect the performance of these models. Therefore, after generating text content, the multimodal model CLIP is used to calculate the similarity between images or some rules to filter low-quality samples, and finally retain high-quality samples for model training.

[0003] Existing data cleaning methods have the following disadvantages: 1) Existing data cleaning methods often calculate the similarity of the entire text and image, while the generated hallucination content sometimes only occupies a small part of the text. This way of calculating similarity can easily ignore the hallucination content; 2) A lot of the generated text content is long text content, while the CLIP model used for similarity estimation often supports short texts. Directly using CLIP to calculate similarity will cause the model to only focus on a part of the valid content in the generated text. Summary of the invention

[0004] The present invention provides a data cleaning method and device based on local and global similarity filtering of texts, so as to solve the problem that the prior art cannot effectively filter the hallucination content generated by the model and effectively filter the long texts.

[0005] In a first aspect, the present invention provides a data cleaning method based on local and global text similarity filtering, which specifically comprises the following steps: Step S1, obtaining an image A and a long text B used to describe the content of the image A; Step S2, preprocessing the long text B to form a preprocessed long text C; Step S3: split the long text C into a text D consisting of N short texts = {d0, d1, ..., d N}; Step S4: input image A, long text C and text D into the CLIP model, and extract features of image A through the CLIP model to form image features E and text features F={f0,f1,...,f N ,f N+1}; Step S5: normalize the image feature E and the text feature F to form a normalized image feature G and a normalized text feature H = {h0, h1, ..., h N ,h N+1}; Step S6: Using the normalized image feature G and the normalized text feature H, a similarity I={i0,i1,...,i N ,i N+1}; Step S7: Filter each text according to the similarity I of each text to form M short texts J={j0,j1,...,j M}, and form the long text K through J.

[0006] Where N and M are both positive integers; d0, d1, ..., d N Any of them represents a short text.

[0007] Among them, f0,f1,...,f N Represents N short texts d0, d1, ..., d N Text features of N+1 Represents the text features of the long text C.

[0008] Among them, h0,h1,...,h N Respectively represent f0,f1,...,f N Text features after normalization; h N+1 Indicates f N+1 The text features after normalization. Among them, i0,i1,...,i N Represents N short texts d0, d1, ..., d N Normalized text features h0,h1,...,h N and the similarity between the normalized image feature G (in this application, it may also be referred to as local similarity); i N+1 Represents the similarity between the long text C and the normalized image features (in this application, it can also be called global similarity).

[0009] Preferably, in step S2, the preprocessing includes performing deduplication processing and repeated prefix removal processing on the content of the long text B based on rules to form a preprocessed long text C.

[0010] Preferably, in step S4, the CLIP model includes an image encoder and a text encoder; wherein the image feature E is formed by the image A and the image encoder of the CLIP model; and the text feature F is formed by the long text C, the text D and the text encoder of the CLIP model.

[0011] Preferably, in step S5, the normalization process includes L2 normalization process, and the specific formula is as follows: in, x It indicates the feature that needs to be normalized, including one of the image feature and the text feature; d indicates the length of the feature that needs to be normalized.

[0012] Preferably, in step S6, the similarity I of each text in the image A and the text C is formed by performing a dot product between the normalized image feature G and the normalized text feature H, which is specifically expressed as follows: Preferably, in step S7, each text is filtered according to the similarity I of each text to form M short texts J={j0,j1,...,j M}, and form a long text K through J, which specifically includes the following steps: Step S701: Use global similarity to convert similarity i N+1 The long text B that is smaller than the threshold T1 is filtered out; Step S702: Use local similarity to convert similarities i0,i1,...,i N The short texts smaller than the threshold T2 are filtered out to form M short texts J={j0,j1,...,j M}; Step S703: The M short texts j0, j1, ..., j M They are concatenated to form a long text K.

[0013] Among them, T1 and T2 are both positive numbers.

[0014] In a second aspect, the present invention also provides a data cleaning device based on local and global text similarity filtering, which specifically includes the following modules: A data acquisition module, used for acquiring an image A and a long text B used for describing the content of the image A; A preprocessing module, used for preprocessing the long text B to form a preprocessed long text C; The text splitting module is used to split the long text C into a text D consisting of N short texts = {d0, d1, ..., d N}; The feature extraction module is used to input the image A, the long text C and the text D into the CLIP model, and extract the features of the image A through the CLIP model to form the image feature E and the text feature F={f0,f1,...,f N ,f N+1}; A normalization processing module is used to normalize the image feature E and the text feature F to form a normalized image feature G and a normalized text feature H={h0,h1,...,h N ,h N+1}; A similarity calculation module is used to form a similarity I={i0,i1,...,i N ,i N+1}; The filtering and text generation module is used to filter each text according to the similarity I of each text, and form M short texts J={j0,j1,...,j M}, and form the long text K through J.

[0015] Where N and M are both positive integers; d0, d1, ..., d N Any of them represents a short text.

[0016] Among them, f0,f1,...,f N Represents N short texts d0, d1, ..., d N Text features of N+1 Represents the text features of the long text C.

[0017] Among them, h0,h1,...,h N Respectively represent f0,f1,...,f N Text features after normalization; h N+1 Indicates f N+1 Text features after normalization.

[0018] Among them, i0,i1,...,i N Respectively represent the local similarity; i N+1 Represents the global similarity.

[0019] Preferably, in the preprocessing module, the preprocessing includes performing deduplication processing and repeated prefix removal processing on the content of the long text B based on rules to form a preprocessed long text C.

[0020] Preferably, in the feature extraction module, the CLIP model includes an image encoder and a text encoder; wherein the image feature E is formed by the image A and the image encoder of the CLIP model; and the text feature F is formed by the long text C, the text D and the text encoder of the CLIP model.

[0021] Preferably, in the normalization processing module, the normalization processing includes L2 normalization processing, and the specific formula is as follows: in, x It indicates the feature that needs to be normalized, including one of the image feature and the text feature; d indicates the length of the feature that needs to be normalized.

[0022] Preferably, in the similarity calculation module, the similarity I of each text in the image A and the text C is formed by performing a dot product between the normalized image feature G and the normalized text feature H, which is specifically expressed as follows: Preferably, the filtering processing and text generation module specifically includes the following submodules: The first processing submodule is used to convert the similarity i N+1 The long text B that is smaller than the threshold T1 is filtered out; The second processing submodule converts the similarities i0,i1,...,i N The short texts smaller than the threshold T2 are filtered out to form M short texts J={j0,j1,...,j M}; The third processing submodule converts the M short texts j0, j1, ..., j M They are concatenated to form a long text K.

[0023] Among them, T1 and T2 are both positive numbers.

[0024] In a third aspect, the present invention further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a data cleaning method based on local and global text similarity filtering as described in any one of the first aspects of the present application.

[0025] In a fourth aspect, the present invention further provides an electronic device, comprising: a memory storing a computer program; and a processor communicatively connected to the memory, for executing a data cleaning method based on local and global text similarity filtering as described in any one of the first aspects of the present application when the computer program is called.

[0026] Compared with the prior art, the present invention has the following obvious outstanding substantive features and significant advantages: The present invention proposes a data cleaning method and device based on local and global text similarity filtering to solve the problem that only global information is considered in the prior art and the CLIP model has poor compatibility with long texts. The scheme considers the similarity between the complete text and the image content and the similarity between each sentence in the text and the image content, and can effectively filter out the hallucination content in the generated text. At the same time, after the text content is split into short sentences, it is also compatible with the ability of CLIP to process short texts relatively well. First, the visual encoder and the text encoder in the CLIP model are used to encode the image and the split text respectively to obtain global image features and local text features. Then the similarity between the image features and each text feature is calculated. Sentences with similarity lower than the threshold are filtered out from the text, and the remaining sentences continue to form a complete text, and finally the generated, high-quality, low-hallucination image-text pair data is obtained. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] The accompanying drawings, which constitute part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention. In the accompanying drawings: Figure 1 It is a flow chart of a data cleaning method based on local and global text similarity filtering according to a preferred embodiment of the present invention.

[0028] Figure 2 This is an example diagram of a data cleaning method based on local and global text similarity filtering according to a preferred embodiment of the present invention.

[0029] Figure 3 It is a structural schematic diagram of a data cleaning device based on local and global text similarity filtering according to a preferred embodiment of the present invention. DETAILED DESCRIPTION

[0030] The present invention provides a data cleaning method and device based on local and global text similarity filtering. In order to make the purpose, technical solution and effect of the present invention clearer and more specific, the present invention is further described in detail with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0031] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0032] Embodiment 1: like Figure 1-Figure 2 As shown, the data cleaning method based on local and global text similarity filtering described in this embodiment specifically includes the following steps: Step S1: Obtain image A and long text B used to describe the content of image A.

[0033] Step S2: preprocess the long text B to form a preprocessed long text C.

[0034] Optionally, the preprocessing includes performing deduplication processing and repeated prefix removal processing on the content of the long text B based on rules to form a preprocessed long text C.

[0035] Step S3: split the long text C into a text D consisting of N short texts = {d0, d1, ..., d N}; where N is a positive integer; d0, d1, ..., d N Any of them represents a short text.

[0036] Step S4: input image A, long text C and text D into the CLIP model, and extract features of image A through the CLIP model to form image features E and text features F={f0,f1,...,f N ,f N+1}; where f0,f1,...,f N Represents N short texts d0, d1, ..., d N Text features of N+1 Represents the text features of the long text C.

[0037] Optionally, in step S4, the CLIP model includes an image encoder and a text encoder; wherein the image feature E is formed by the image A and the image encoder of the CLIP model; and the text feature F is formed by the long text C, the text D and the text encoder of the CLIP model.

[0038] Step S5: normalize the image feature E and the text feature F to form a normalized image feature G and a normalized text feature H = {h0, h1, ..., h N ,h N+1}; where h0,h1,...,h N Respectively represent f0,f1,...,f N Text features after normalization; h N+1 Indicates f N+1 Text features after normalization.

[0039] Optionally, in step S5, the normalization process includes L2 normalization process, and the specific formula is as follows: in, x It indicates the feature that needs to be normalized, including one of the image feature and the text feature; d indicates the length of the feature that needs to be normalized.

[0040] Step S6: Using the normalized image feature G and the normalized text feature H, a similarity I={i0,i1,...,i N ,i N+1}; where i0,i1,...,i N Both represent local similarity; i N+1 Represents the global similarity.

[0041] Optionally, in step S6, a dot product is performed between the normalized image feature G and the normalized text feature H to form a similarity I between each text in the image A and the text C, which is specifically expressed as follows: .

[0042] Step S7: Filter each text according to the similarity I of each text to form M short texts J={j0,j1,...,j M}, and form a long text K through J; where M is a positive integer.

[0043] Optionally, step S7 specifically includes the following steps: Step S701: Use global similarity to convert similarity i N+1 The long text B that is smaller than the threshold T1 is filtered out; Step S702: Use local similarity to convert similarities i0,i1,...,i NThe short texts smaller than the threshold T2 are filtered out to form M short texts J={j0,j1,...,j M}; Step S703: The M short texts j0, j1, ..., j M They are concatenated to form a long text K.

[0044] Among them, T1 and T2 are both positive numbers.

[0045] like Figure 2 The image shown in the upper left corner is Image. Figure 2 The upper right corner is a description of the image through a large language model. The specific description is "This picture shows a suspension bridge, which spans across the valley. There are towering towers on both sides of the bridge, and many steel cables are hung between the towers. These steel cables support the bridge deck, on which vehicles are running, and below the bridge are dense trees and vegetation...". The above description is divided into three sentences according to the period. The first sentence is "This picture shows a suspension bridge, which spans across the valley.", the second sentence is "There are towering towers on both sides of the bridge, and many steel cables are hung between the towers.", and the third sentence is "These steel cables support the bridge deck, on which vehicles are running, and below the bridge are dense trees and vegetation...". The image Image and the above three sentences are respectively input into the image encoder and text encoder of the CLIP model for feature extraction. The feature similarity between the extracted image features and the text features is calculated, and double filtering is performed through local similarity and global similarity according to the calculated similarity to obtain the final filtered text, that is, the cleaned text "This picture shows a suspension bridge, the bridge body spanning between the valleys. There are towering towers on both sides of the bridge, and many steel cables are suspended between the towers. These steel cables support the bridge deck." It can be seen that the sentence "There are vehicles driving on the bridge deck" is the hallucination text generated by the large language model based on the image Image. At the same time, the content after the sentence "There are vehicles driving on the bridge deck" "Under the bridge are dense trees and vegetation..." is also deleted. Therefore, the method described in this embodiment is used to filter the hallucination text content of the large language model.

[0046] Embodiment 2: like Figure 3 As shown, the data cleaning device based on local and global text similarity filtering described in this embodiment specifically includes the following modules: The data acquisition module is used to acquire an image A and a long text B used to describe the content of the image A.

[0047] The preprocessing module is used to preprocess the long text B to form a preprocessed long text C.

[0048] Optionally, in the preprocessing module, the preprocessing includes performing deduplication processing and repeated prefix removal processing on the content of the long text B based on rules to form a preprocessed long text C.

[0049] The text splitting module is used to split the long text C into a text D consisting of N short texts = {d0, d1, ..., d N}; where N is a positive integer; d0, d1, ..., d N Any of them represents a short text.

[0050] The feature extraction module is used to input the image A, the long text C and the text D into the CLIP model, and extract the features of the image A through the CLIP model to form the image feature E and the text feature F={f0,f1,...,f N ,f N+1}; where f0,f1,...,f N Represents N short texts d0, d1, ..., d N Text features of N+1 Represents the text features of the long text C.

[0051] Optionally, in the feature extraction module, the CLIP model includes an image encoder and a text encoder; wherein the image feature E is formed by the image A and the image encoder of the CLIP model; and the text feature F is formed by the long text C, the text D and the text encoder of the CLIP model.

[0052] A normalization processing module is used to normalize the image feature E and the text feature F to form a normalized image feature G and a normalized text feature H={h0,h1,...,h N ,h N+1}; where h0,h1,...,h N Respectively represent f0,f1,...,f N Text features after normalization; h N+1 Indicates f N+1 Text features after normalization.

[0053] Optionally, in the normalization processing module, the normalization processing includes L2 normalization processing, and the specific formula is as follows: in, x It indicates the feature that needs to be normalized, including one of the image feature and the text feature; d indicates the length of the feature that needs to be normalized.

[0054] A similarity calculation module is used to form a similarity I={i0,i1,...,i N ,i N+1}; where i0,i1,...,i N Respectively represent the local similarity; i N+1 Represents the global similarity.

[0055] Optionally, by performing a dot product between the normalized image feature G and the normalized text feature H, a similarity I of each text in the image A and the text C is formed, which is specifically expressed as follows: .

[0056] The filtering and text generation module is used to filter each text according to the similarity I of each text, and form M short texts J={j0,j1,...,j M}, and form a long text K through J; where M is a positive integer.

[0057] Optionally, the filtering processing and text generation module specifically includes the following submodules: The first processing submodule is used to convert the similarity i N+1 Long text B that is less than a threshold value T1 is filtered out; where T1 is a positive number.

[0058] The second processing submodule converts the similarities i0,i1,...,i N The short texts smaller than the threshold T2 are filtered out to form M short texts J={j0,j1,...,j M}; where T2 is a positive number.

[0059] The third processing submodule converts the M short texts j0, j1, ..., j M They are concatenated to form a long text K.

[0060] The specific embodiments of the present invention are described in detail above, but they are only examples, and the present invention is not limited to the specific embodiments described above. For those skilled in the art, any equivalent modifications and substitutions made to the present invention are also within the scope of the present invention. Therefore, the equalization changes and modifications made without departing from the spirit and scope of the present invention should be included in the scope of the present invention.

Claims

1. A data cleaning method based on local and global text similarity filtering, characterized in that: The specific steps include: Step S1, obtaining an image A and a long text B used to describe the content of the image A; Step S2, preprocessing the long text B to form a preprocessed long text C; Step S3: split the long text C into a text D consisting of N short texts = {d0, d1, ..., d N }; Step S4: input image A, long text C and text D into the CLIP model, and extract features of image A through the CLIP model to form image features E and text features F={f0,f1,...,f N ,f N+1 }; Step S5: normalize the image feature E and the text feature F to form a normalized image feature G and a normalized text feature H = {h0, h1, ..., h N ,h N+1 }; Step S6: Using the normalized image feature G and the normalized text feature H, a similarity I={i0,i1,...,i N ,i N+1 }; Step S7: Filter each text according to the similarity I of each text to form M short texts J={j0,j1,...,j M }, and form a long text K through J; Where N and M are both positive integers; d0, d1, ..., d N Any of the represents a short text; Among them, f0,f1,...,f N Represents N short texts d0, d1, ..., d N Text features of N+1 Represents the text features of the long text C; Among them, h0,h1,...,h N Respectively represent f0,f1,...,f N Text features after normalization; h N+1 Indicates f N+1 Text features after normalization; Among them, i0,i1,...,i N Represents N short texts d0, d1, ..., d N Normalized text features h0,h1,...,h N The similarity between the normalized image feature G is called local similarity; N+1 It represents the similarity between the long text C and the normalized image features, which is called global similarity.

2. A method for data cleaning based on local and global text similarity filtering according to claim 1, characterized in that: In step S2, the preprocessing includes performing deduplication processing and repeated prefix removal processing on the content of the long text B based on rules to form a preprocessed long text C.

3. A method for data cleaning based on local and global text similarity filtering according to claim 1, characterized in that: In step S4, the CLIP model includes an image encoder and a text encoder; wherein the image feature E is formed by the image A and the image encoder of the CLIP model; and the text feature F is formed by the long text C, the text D and the text encoder of the CLIP model.

4. A method for data cleaning based on local and global text similarity filtering according to claim 1, characterized in that: In step S5, the normalization process includes L2 normalization process, and the specific formula is as follows: in, x It indicates the feature that needs to be normalized, including one of the image feature and the text feature; d indicates the length of the feature that needs to be normalized.

5. The method for data cleaning based on local and global text similarity filtering according to claim 1, characterized in that: In step S6, the similarity I of each text in the image A and the text C is formed by performing a dot product between the normalized image feature G and the normalized text feature H, which is specifically expressed as follows: 。 6. A method for data cleaning based on local and global text similarity filtering according to claim 1, characterized in that: In step S7, each text is filtered according to the similarity I of each text to form M short texts J={j0,j1,...,j M }, and form a long text K through J, which specifically includes the following steps: Step S701: Use global similarity to convert similarity i N+1 The long text B that is smaller than the threshold T1 is filtered out; Step S702: Use local similarity to convert similarities i0,i1,...,i N The short texts smaller than the threshold T2 are filtered out to form M short texts J={j0,j1,...,j M }; Step S703: The M short texts j0, j1, ..., j M Splice them together to form a long text K; Among them, T1 and T2 are both positive numbers.

7. A data cleaning device based on local and global text similarity filtering, characterized in that: The modules include: A data acquisition module, used for acquiring an image A and a long text B used for describing the content of the image A; A preprocessing module, used for preprocessing the long text B to form a preprocessed long text C; The text splitting module is used to split the long text C into a text D consisting of N short texts = {d0, d1, ..., d N }; The feature extraction module is used to input the image A, the long text C and the text D into the CLIP model, and extract the features of the image A through the CLIP model to form the image feature E and the text feature F={f0,f1,...,f N ,f N+1 }; A normalization processing module is used to normalize the image feature E and the text feature F to form a normalized image feature G and a normalized text feature H={h0,h1,...,h N ,h N+1 }; A similarity calculation module is used to form a similarity I={i0,i1,...,i N ,i N+1 }; The filtering and text generation module is used to filter each text according to the similarity I of each text, and form M short texts J={j0,j1,...,j M }, and form a long text K through J; Where N and M are both positive integers; d0, d1, ..., d N Any of the represents a short text; Among them, f0,f1,...,f N Represents N short texts d0, d1, ..., d N Text features of N+1 Represents the text features of the long text C; Among them, h0,h1,...,h N Respectively represent f0,f1,...,f N Text features after normalization; h N+1 Indicates f N+1 Text features after normalization; Among them, i0,i1,...,i N Represents N short texts d0, d1, ..., d N Normalized text features h0,h1,...,h N The similarity between the normalized image feature G is called local similarity; N+1 It represents the similarity between the long text C and the normalized image features, which is called global similarity.

8. The device for data cleaning based on local and global text similarity filtering according to claim 7, characterized in that: The filtering processing and text generation module specifically includes the following submodules: The first processing submodule is used to convert the similarity i N+1 The long text B that is smaller than the threshold T1 is filtered out; The second processing submodule converts the similarities i0,i1,...,i N The short texts smaller than the threshold T2 are filtered out to form M short texts J={j0,j1,...,j M }; The third processing submodule converts the M short texts j0, j1, ..., j M Splice them together to form a long text K; Among them, T1 and T2 are both positive numbers.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method for data cleaning based on local and global text similarity filtering as described in any one of claims 1 to 6 is implemented.

10. An electronic device, characterized in that: The invention comprises a memory, a processor and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, a method for data cleaning based on local and global text similarity filtering as described in any one of claims 1 to 6 is implemented.

Images