A multimodal data filtering method and system based on contrastive learning and multi-granular semantic alignment

Abstract

This invention relates to a multimodal data screening method and system based on contrastive learning and multi-granular semantic alignment. The method includes: receiving raw multimodal data samples; calculating the cosine similarity between the visual feature vectors and text feature vectors of each sample to obtain a global consistency score; extracting key regions corresponding to cue words in the visual data, outputting multiple key region candidate boxes, extracting entity nouns in the text, and calculating the correspondence between key region candidate boxes and entity nouns to obtain a local region alignment score; performing part-of-speech tagging on the text, extracting detection targets, constructing an attention heatmap based on the word vectors of both, and calculating the overlap of attention points as a fine-grained score; fusing the global consistency score, local region alignment score, and fine-grained score to obtain a quality confidence score, and retaining or deleting corresponding samples based on the quality confidence score to complete the screening. Compared with existing technologies, this invention significantly reduces the risk of large model illusions and improves data quality.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence, and in particular to a method and system for screening multimodal data based on contrastive learning and multi-granular semantic alignment. Background Technology

[0002] With the rapid development of multimodal large models (MLLMs, such as GPT-5, Gemini 2.5, Sora2, etc.), the requirements for data quality in model training have shifted from "quantity" to "quality." Currently, multimodal training data (mainly image-text and video-text pairs) is primarily obtained from web scraping, which suffers from severe noise problems.

[0003] Existing data cleaning techniques primarily rely on global feature matching from models such as CLIP (Contrastive Language-Image Pre-training). However, this coarse-grained "global matching" has significant drawbacks: Hallucination Origin: Global matching can only determine "there is a cat in the picture and a cat in the text," but it cannot identify subtle attribute inconsistencies such as "there is a yellow cat in the text, but a white cat in the picture." This data noise is one of the main reasons why large models produce "illusions."

[0004] A one-size-fits-all threshold: Traditional methods usually set a fixed cosine similarity threshold for filtering, which can easily lead to the accidental deletion of "hard positives" (i.e., data with obscure meaning but correct matching) or the omission of "hard negatives" (i.e., dirty data that looks similar but has incorrect details).

[0005] High labor costs: To solve the above problems, a large amount of manual review is often required, which is costly and inefficient and cannot meet the cleaning needs of petabyte-level data.

[0006] In summary, to address the aforementioned issues, it is necessary to propose an automated data evaluation method that considers not only the "whole" but also the "part" and "details" to quickly construct high-quality datasets. Summary of the Invention

[0007] The purpose of this invention is to overcome the shortcomings of the existing technology and provide a multimodal data filtering method and system based on contrastive learning and multi-granular semantic alignment.

[0008] The objective of this invention can be achieved through the following technical solutions: A multimodal data filtering method based on contrastive learning and multi-granular semantic alignment, the method comprising: Receive raw multimodal data samples, the samples including visual-text data pairs; Visual feature vectors and text feature vectors of the samples are extracted using a visual encoder and a text encoder, respectively; the cosine similarity of the visual feature vectors and text feature vectors of each sample is calculated to obtain a global consistency score. Using the text feature vector as prompt words, the key regions corresponding to the prompt words in the visual data are extracted using an open vocabulary object detection algorithm, and multiple key region candidate boxes are output; entity nouns in the text are extracted using a pre-trained large language model, and the correspondence between the key region candidate boxes and entity nouns is calculated to obtain a local region alignment score. Natural language processing algorithms are used to perform part-of-speech tagging on the text, and words with part-of-speech tags of adjectives and verbs are extracted as detection targets. A fine-grained cross-attention algorithm is used to construct an attention heatmap based on visual feature vectors and word vectors of detection targets. Key region candidate boxes corresponding to nouns modified by detection targets are obtained, and the overlap of attention landing points is calculated as a fine-grained score. The global consistency score, local region alignment score, and fine-grained score are fused through an adaptive weighted network to obtain a quality confidence score. Based on the quality confidence score, the corresponding samples are retained or deleted to complete the screening.

[0009] Furthermore, the visual encoder adopts the ViT-H / 14 architecture to extract the visual global feature vector of the last layer and the visual feature map of the intermediate layer of the image or video frame. The text encoder uses RoBERTa-Large to extract sentence-level global text vectors and token-level text word vector sequences from the text.

[0010] Furthermore, the visual global feature vector and text global vector are used to calculate cosine similarity during the process of obtaining local region alignment scores; the visual feature map and text word vector sequence of the intermediate layer are used to construct the attention heatmap.

[0011] Furthermore, after obtaining the global consistency score, the mean and variance of the similarity of the current batch of samples are calculated based on the cosine similarity. A dynamic lower limit is obtained based on the mean and variance of the cosine similarity. If the cosine similarity of any sample is less than the dynamic lower limit and the image entropy of the sample is lower than the preset minimum value, it is marked as garbage data and discarded.

[0012] Furthermore, after extracting the key regions of the target corresponding to the text in the image using the open vocabulary target detection algorithm, the confidence scores of the candidate boxes corresponding to each key region are also output. Based on the preset candidate box confidence threshold, the key region candidate boxes with confidence scores higher than the preset candidate box confidence threshold are retained, and the key region candidate boxes with confidence scores lower than the preset candidate box confidence threshold are deleted.

[0013] Furthermore, the process of calculating the correspondence between the candidate bounding boxes of the key regions and entity names to obtain the local region alignment score includes: Based on the coordinates of the candidate boxes of each key region, ROI image fragments are cropped from the visual data of the original samples and input into the visual encoder to obtain ROI feature vectors; The entity nouns extracted by the pre-trained large language model are input into the text encoder to obtain entity word feature vectors; Calculate the cosine similarity between the feature vectors of each ROI and the feature vectors of the entity words, obtain multiple sets of matching pair scores, and obtain the final local region alignment score by weighted averaging of all matching pair scores.

[0014] Furthermore, the natural language processing algorithms include NLTK and Stanford CoreNLP.

[0015] Furthermore, the process of constructing the attention heatmap based on the visual feature vector and the word vector of the detection target using a fine-grained cross-attention algorithm includes: Using the visual feature vectors as Key and Value vectors, and the word vectors of the detected target as Query vectors, the attention heatmap is calculated and obtained; the calculation expression for the attention heatmap is: in, For attention heatmap, For the query vector, For the key vector, Key vector The feature dimensions.

[0016] Furthermore, during the process of retaining or deleting corresponding samples based on the quality confidence score, based on the preset upper and lower limits of the quality confidence score threshold, boundary fuzzy samples with quality confidence scores between the upper and lower limits of the quality confidence score threshold are selected, and the boundary fuzzy samples are used to optimize the adaptive weighted network and fine-grained cross-attention algorithm.

[0017] A multimodal data filtering system based on contrastive learning and multi-granular semantic alignment, the system comprising: Data receiving module: Receives raw multimodal data samples, including visual-text data pairs; Global consistency scoring module: The visual feature vector and text feature vector of the sample are extracted using a visual encoder and a text encoder, respectively; the cosine similarity of the visual feature vector and text feature vector of each sample is calculated to obtain the global consistency score. Local region alignment scoring module: Using the text feature vector as prompt words, the module uses an open vocabulary object detection algorithm to extract key regions corresponding to the prompt words in the visual data and outputs multiple key region candidate boxes; it uses a pre-trained large language model to extract entity nouns in the text and calculates the correspondence between the key region candidate boxes and entity nouns to obtain a local region alignment score. Fine-grained scoring module: Uses natural language processing algorithms to perform part-of-speech tagging on the text, extracting words with part-of-speech tags of adjectives and verbs as detection targets; adopts a fine-grained cross-attention algorithm to construct an attention heatmap based on visual feature vectors and word vectors of detection targets; obtains candidate boxes of key regions corresponding to nouns modified by detection targets, and calculates the overlap of attention landing points as fine-grained scoring; Screening module: The global consistency score, local region alignment score and fine-grained score are fused through an adaptive weighted network to obtain the quality confidence score. Based on the quality confidence score, the corresponding samples are retained or deleted to complete the screening.

[0018] Compared with the prior art, the beneficial effects of the present invention include: 1. This invention effectively identifies and removes noisy data that is semantically related but contains erroneous details, such as mismatched colors, incorrect quantities, and misaligned spatial / action relationships, from the data source. This allows for the rapid selection of high-quality datasets for training while reducing the possibility of large multimodal models learning incorrect associations during training, directly improving the authenticity and accuracy of the content generated by the model and alleviating the illusion problem of large models at its root. In this invention, a multi-granularity (global-local-detail) alignment evaluation is introduced, and evaluation scores in three dimensions are calculated and weighted and fused to achieve three-dimensional data quality identification of global, local, and detailed aspects, enabling precise data screening.

[0019] 2. The local evaluation layer of this invention achieves accurate matching between key regions of interest (ROI) in images and text entities through open vocabulary target detection and entity extraction from large language models. It can identify the problem of missing / mismatched entities where a certain entity is extracted from the text but there is no corresponding region in the image.

[0020] 3. The detailed evaluation layer of this invention innovatively adopts a fine-grained cross-attention mechanism, which combines part-of-speech tagging to extract adjectives and verbs. By constructing an attention heatmap and calculating the overlap of attention points, it accurately verifies error-prone attribute binding issues such as color, quantity, and spatial / action relationships, filling the gap in detailed verification of traditional technologies.

[0021] 4. This invention introduces an active learning strategy. During the final screening, a small number of difficult samples in the boundary ambiguity area that the system cannot accurately determine are collected. Boundary samples are then optimized for the adaptive weighted network and fine-grained cross-attention algorithm, thereby continuously improving the boundary determination capability of the scoring model.

[0022] 5. The global evaluation layer of this invention abandons the traditional screening method of fixed cosine similarity threshold, and adopts dynamic statistical truncation method to dynamically set the screening lower limit according to the score distribution of the current batch of samples, adapting to the characteristics of different batches of data, and avoiding the accidental deletion of high-quality samples with obscure semantics due to fixed threshold. Attached Figure Description

[0023] Figure 1 This is a flowchart of the method of the present invention. Detailed Implementation

[0024] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0025] Example 1 This embodiment discloses a multimodal data filtering method based on contrastive learning and multi-granularity semantic alignment, the method as follows: Figure 1 As shown, steps S1-S5 are included, and the specific descriptions of each step are as follows: Step S1: Receive raw multimodal data samples, which include visual-text data pairs. Specifically, the visual data includes image data and video data.

[0026] To accurately identify low-quality data, this method evaluates sample quality across three dimensions: global, local, and detailed, and then weights and merges the specific evaluation scores from these three dimensions.

[0027] Step S2: Extract the visual feature vector and text feature vector of the sample using the visual encoder and text encoder respectively; calculate the cosine similarity of the visual feature vector and text feature vector of each sample to obtain the global consistency score.

[0028] Specifically, the VisualEncoder employs an improved ViT-H / 14 architecture. It not only extracts the visual global feature vector from the last layer of an image or video frame, but also... (Used for global initial screening), it also extracts visual feature maps from intermediate layers (such as the 9th and 12th layers). This preserves the spatial location information of the image, providing a basis for detailed verification in the subsequent step S4.

[0029] The text encoder uses RoBERTa-Large to extract sentence-level global text vectors. And token-level text word vector sequences , used for fine-grained pairs of entities and attributes.

[0030] Step S2 is the global evaluation layer, which performs quality evaluation on the global feature vector.

[0031] In the process of obtaining local region alignment scores, cosine similarity is calculated using visual global feature vectors and text global vectors. With text global vector The expression for cosine similarity is: in, The global consistency score is calculated using cosine similarity.

[0032] After obtaining the global consistency score, the mean and variance of the similarity of the current batch of samples are calculated based on cosine similarity. and variance Get the dynamic lower bound If the cosine similarity of any sample is less than the dynamic lower bound, then... If the image entropy (a measure of image information content) of the sample is lower than a preset minimum value (such as a pure black / pure white image), it is marked as garbage data and discarded.

[0033] Step S3: Using text feature vectors as prompt words, the key regions corresponding to the prompt words in the visual data are extracted using an open vocabulary object detection algorithm, and multiple key region candidate boxes are output; entity nouns in the text are extracted using a pre-trained large language model, and the correspondence between key region candidate boxes and entity nouns is calculated to obtain local region alignment scores.

[0034] Step S3 is the local evaluation layer, which performs quality evaluation on local feature vectors.

[0035] After extracting the key regions of the target corresponding to the text in the image using the open vocabulary target detection algorithm, the confidence scores of the candidate boxes corresponding to each key region are also output. Based on the preset candidate box confidence threshold, the candidate boxes of the key regions with confidence scores higher than the preset candidate box confidence threshold are retained, and the candidate boxes of the key regions with confidence scores lower than the preset candidate box confidence threshold are deleted.

[0036] A key region (ROI) is a semantically independent object region that occupies a prominent position in an image, typically marked with a rectangular bounding box. This invention utilizes open-vocabulary object detection algorithms (such as Grounding DINO) or YOLO-World, using text as a prompt, to extract key regions of the target object in the image. The specific process includes: Input the text description as a prompt word into the detector; The detector outputs N candidate boxes. and their corresponding confidence levels ; Set a confidence threshold (e.g., 0.4) to retain high-confidence boxes as ROIs.

[0037] Entity nouns in text can be extracted using a pre-trained large language model. For example, with the prompt "Please extract the core entity nouns (Subject, Object) in this sentence and output a JSON list", the input is "A yellow cat is sitting on the sofa", and the output is ["cat", "sofa"].

[0038] The process of calculating the correspondence between candidate bounding boxes of key regions and entity names to obtain local region alignment scores includes: Based on the coordinates of candidate bounding boxes for each key region, image fragments of the Region of Interest (ROI) are cropped from the visual data of the original samples and input into the visual encoder to obtain the ROI feature vectors. ; The entity nouns extracted from the pre-trained large language model are input into the text encoder to obtain entity word feature vectors. ; Calculate the cosine similarity between the feature vectors of each ROI and the feature vectors of entity words, and obtain scores for multiple matching pairs. The final local region alignment score is obtained by calculating a weighted average of the scores of all matching pairs.

[0039] The cosine similarity between the feature vectors of each ROI and the feature vectors of entity words is calculated, and the expression for calculating the matching pair score is as follows: .

[0040] Traditional IoU is unusable (because it only contains text and no coordinates). This invention uses region-text embedding distance to calculate a matching score between the cropped ROI image features and the corresponding entity word vectors. If the text mentions "cat" but no "cat" ROI is detected in the image, or if the detected region does not match the features of "cat," the score of this layer drops significantly.

[0041] Step S4: Use natural language processing algorithms to perform part-of-speech tagging on the text and extract words with part-of-speech tags of adjectives and verbs as detection targets; use a fine-grained cross-attention algorithm to construct an attention heatmap based on visual feature vectors and word vectors of detection targets; obtain candidate boxes of key regions corresponding to nouns modified by detection targets, and calculate the overlap of attention landing points as a fine-grained score.

[0042] Step S4 is the detailed evaluation layer, which performs quality assessment on details and aims to resolve the problem of incorrect binding of color, quantity, and spatial relationship (orientation).

[0043] The attention heatmap is constructed using intermediate layer visual feature maps and text word vector sequences.

[0044] Natural language processing algorithms include NLTK and Stanford CoreNLP, with NLTK or Stanford CoreNLP used for part-of-speech tagging (POS tagging). Words tagged with "JJ" (adjective) and "VB" (verb) are extracted as detection targets.

[0045] A fine-grained cross-attention mechanism is employed to check for consistency in color, quantity, and spatial relationships (e.g., "cat on the table" vs. "cat under the table"). The process of constructing an attention heatmap based on visual feature vectors and word vectors of the detected target using a fine-grained cross-attention algorithm includes: Using visual feature vectors as key and value vectors, and word vectors of the detected target as query vectors, an attention heatmap is calculated. The expression for calculating the attention heatmap is: in, For attention heatmap, For the query vector, For the key vector, Key vector The feature dimensions.

[0046] Obtain candidate bounding boxes for key regions corresponding to the nouns modifying the target, and calculate the overlap of attention points as a fine-grained score, specifically: Get the ROI mask corresponding to the noun modified by the adjective or action verb (such as "cat"). ; Calculate the overlap of attention points: If "yellow" focuses its attention primarily on the area of ​​the "cat," it will receive a high score; if "yellow" focuses on the background or other objects, it will be judged as an attribute error (misalignment).

[0047] Step S5: The global consistency score, local region alignment score, and fine-grained score are fused through an adaptive weighted network to obtain the quality confidence score. Based on the quality confidence score, the corresponding samples are retained or deleted to complete the screening.

[0048] The process of fusion through adaptive weighted networks is as follows: Construct a lightweight MLP (Multilayer Perceptron) gating network.

[0049] Input: [Global score, Local score, Detail score, Image entropy, Text length].

[0050] Mechanism: MLP automatically outputs the weights [α, β, γ] of three scores based on the distribution characteristics of the input data (e.g., low image entropy represents a simple graph, and high entropy represents a complex graph).

[0051] Results: For simple graphs, the system automatically focuses on the global score; for complex scene graphs, the system automatically focuses on the detail score and calculates the final quality confidence score.

[0052] In the process of retaining or deleting corresponding samples based on quality confidence, the system also filters out boundary fuzzy samples whose quality confidence is between the upper and lower limits of the quality confidence threshold based on the preset upper and lower limits of the quality confidence threshold. The boundary fuzzy samples are then used to optimize the boundary samples of the adaptive weighted network and the fine-grained cross-attention algorithm.

[0053] Specifically, the data is divided into three categories based on confidence level: High confidence region (reserved): Directly enter the high-quality training set.

[0054] Low confidence zone (discard): judged as noise and filtered directly.

[0055] Uncertainty / Boundary Samples: These are data that the system is unsure about, and they are used for boundary sample optimization.

[0056] A small number of samples from areas with blurred boundaries are pushed to human annotators, or a teacher model with larger parameters (such as GPT-4V) is used for automatic model evaluation and classification, and then included in the training set for optimization training.

[0057] By optimizing the training set and using the classification results (whether they are positive or negative samples) as supervision signals, only the projection layers of the adaptive MLP network and the fine-grained cross-attention algorithm are fine-tuned.

[0058] By continuously learning from these challenging samples, the scoring model's ability to determine boundaries will become stronger and stronger, and over time, the proportion requiring human intervention will decrease exponentially.

[0059] Example 2 This embodiment, based on Embodiment 1 above, discloses a multimodal data filtering system based on contrastive learning and multi-granular semantic alignment. The system includes: Data receiving module: Receives raw multimodal data samples, including visual-text data pairs; Global consistency scoring module: Extracts visual feature vectors and text feature vectors of samples using a visual encoder and a text encoder respectively; calculates the cosine similarity of visual feature vectors and text feature vectors of each sample to obtain a global consistency score; Local region alignment scoring module: Using text feature vectors as prompt words, it uses an open vocabulary object detection algorithm to extract key regions corresponding to the prompt words in the visual data and outputs multiple key region candidate boxes; it uses a pre-trained large language model to extract entity nouns in the text and calculates the correspondence between key region candidate boxes and entity nouns to obtain local region alignment scores; Fine-grained scoring module: Uses natural language processing algorithms to perform part-of-speech tagging on the text, extracting words with part-of-speech tags of adjectives and verbs as detection targets; adopts a fine-grained cross-attention algorithm to construct an attention heatmap based on visual feature vectors and word vectors of detection targets; obtains candidate boxes of key regions corresponding to nouns modified by detection targets, and calculates the overlap of attention landing points as fine-grained scoring; The screening module integrates global consistency scores, local region alignment scores, and fine-grained scores through an adaptive weighted network to obtain quality confidence scores. Based on the quality confidence scores, corresponding samples are retained or deleted to complete the screening.

[0060] For details regarding the above modules, please refer to the relevant descriptions and effects in Example 1 for further understanding.

[0061] Example 3 Based on Embodiment 1, this embodiment provides an electronic device, including: one or more processors and a memory, wherein the memory stores one or more programs, the one or more programs including instructions for executing the aforementioned multimodal data filtering method based on contrastive learning and multigranular semantic alignment.

[0062] At the hardware level, the electronic device includes a processor, internal bus, network interface, memory, and non-volatile memory, and may also include other hardware required for business operations. The processor reads the corresponding computer program from the non-volatile memory into memory and then runs it to implement the aforementioned multimodal data filtering method based on contrastive learning and multi-granularity semantic alignment. Of course, in addition to software implementation, this invention does not exclude other implementation methods, such as logic devices or a combination of hardware and software, etc. That is to say, the execution entity of the following processing flow is not limited to individual logic units, but can also be hardware or logic devices.

[0063] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0064] Computer-readable media include both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0065] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A multimodal data filtering method based on contrastive learning and multi-granularity semantic alignment, characterized in that, The method includes: Receive raw multimodal data samples, the samples including visual-text data pairs; Visual feature vectors and text feature vectors of the samples are extracted using a visual encoder and a text encoder, respectively; the cosine similarity of the visual feature vectors and text feature vectors of each sample is calculated to obtain a global consistency score. Using the text feature vector as prompt words, the key regions corresponding to the prompt words in the visual data are extracted using an open vocabulary object detection algorithm, and multiple key region candidate boxes are output; entity nouns in the text are extracted using a pre-trained large language model, and the correspondence between the key region candidate boxes and entity nouns is calculated to obtain a local region alignment score. Natural language processing algorithms are used to perform part-of-speech tagging on the text, and words with part-of-speech tags of adjectives and verbs are extracted as detection targets. A fine-grained cross-attention algorithm is used to construct an attention heatmap based on visual feature vectors and word vectors of detection targets. Key region candidate boxes corresponding to nouns modified by detection targets are obtained, and the overlap of attention landing points is calculated as a fine-grained score. The global consistency score, local region alignment score, and fine-grained score are fused through an adaptive weighted network to obtain a quality confidence score. Based on the quality confidence score, the corresponding samples are retained or deleted to complete the screening.

2. The multimodal data filtering method based on contrastive learning and multi-granularity semantic alignment according to claim 1, characterized in that, The visual encoder adopts the ViT-H / 14 architecture to extract the visual global feature vector of the last layer and the visual feature map of the middle layer of the image or video frame. The text encoder uses RoBERTa-Large to extract sentence-level global text vectors and token-level text word vector sequences from the text.

3. The multimodal data filtering method based on contrastive learning and multi-granularity semantic alignment according to claim 2, characterized in that, In the process of obtaining the local region alignment score, the visual global feature vector and the text global vector are used to calculate the cosine similarity; in the process of constructing the attention heatmap, the visual feature map of the intermediate layer and the text word vector sequence are used for construction.

4. The multimodal data filtering method based on contrastive learning and multi-granularity semantic alignment according to claim 1, characterized in that, After obtaining the global consistency score, the mean and variance of the similarity of the current batch of samples are calculated based on the cosine similarity. A dynamic lower limit is obtained based on the mean and variance of the cosine similarity. If the cosine similarity of any sample is less than the dynamic lower limit and the image entropy of the sample is lower than the preset minimum value, it is marked as garbage data and discarded.

5. The multimodal data filtering method based on contrastive learning and multi-granularity semantic alignment according to claim 1, characterized in that, After extracting the key regions of the target corresponding to the text in the image using the open vocabulary target detection algorithm, the confidence scores of the candidate boxes corresponding to each key region are also output. Based on the preset candidate box confidence threshold, the candidate boxes of the key regions with confidence scores higher than the preset candidate box confidence threshold are retained, and the candidate boxes of the key regions with confidence scores lower than the preset candidate box confidence threshold are deleted.

6. The multimodal data filtering method based on contrastive learning and multi-granularity semantic alignment according to claim 1, characterized in that, The process of calculating the correspondence between the candidate bounding boxes of the key regions and entity names, and obtaining the local region alignment score, includes: Based on the coordinates of the candidate boxes of each key region, ROI image fragments are cropped from the visual data of the original samples and input into the visual encoder to obtain ROI feature vectors; The entity nouns extracted by the pre-trained large language model are input into the text encoder to obtain entity word feature vectors; Calculate the cosine similarity between the feature vectors of each ROI and the feature vectors of the entity words, obtain multiple sets of matching pair scores, and obtain the final local region alignment score by weighted averaging of all matching pair scores.

7. The multimodal data filtering method based on contrastive learning and multi-granularity semantic alignment according to claim 1, characterized in that, The natural language processing algorithms include NLTK and Stanford CoreNLP.

8. A multimodal data filtering method based on contrastive learning and multi-granularity semantic alignment according to claim 1, characterized in that, The process of constructing the attention heatmap based on visual feature vectors and word vectors of the detected target using a fine-grained cross-attention algorithm includes: Using the visual feature vectors as Key and Value vectors, and the word vectors of the detected target as Query vectors, the attention heatmap is calculated and obtained; the calculation expression for the attention heatmap is: in, For attention heatmap, For the query vector, For the key vector, Key vector The feature dimensions.

9. A multimodal data filtering method based on contrastive learning and multi-granularity semantic alignment according to claim 1, characterized in that, In the process of retaining or deleting corresponding samples based on the quality confidence, the system also filters out boundary fuzzy samples whose quality confidence is between the upper and lower limits of the quality confidence threshold based on the preset upper and lower limits of the quality confidence threshold. The boundary fuzzy samples are then used to optimize the boundary samples of the adaptive weighted network and the fine-grained cross-attention algorithm.

10. A multimodal data filtering system based on contrastive learning and multi-granularity semantic alignment, characterized in that, The system includes: Data receiving module: Receives raw multimodal data samples, including visual-text data pairs; Global consistency scoring module: The visual feature vector and text feature vector of the sample are extracted using a visual encoder and a text encoder, respectively; the cosine similarity of the visual feature vector and text feature vector of each sample is calculated to obtain the global consistency score. Local region alignment scoring module: Using the text feature vector as prompt words, the module uses an open vocabulary object detection algorithm to extract key regions corresponding to the prompt words in the visual data and outputs multiple key region candidate boxes; it uses a pre-trained large language model to extract entity nouns in the text and calculates the correspondence between the key region candidate boxes and entity nouns to obtain a local region alignment score. Fine-grained scoring module: Uses natural language processing algorithms to perform part-of-speech tagging on the text, extracting words with part-of-speech tags of adjectives and verbs as detection targets; adopts a fine-grained cross-attention algorithm to construct an attention heatmap based on visual feature vectors and word vectors of detection targets; obtains candidate boxes of key regions corresponding to nouns modified by detection targets, and calculates the overlap of attention landing points as fine-grained scoring; Screening module: The global consistency score, local region alignment score and fine-grained score are fused through an adaptive weighted network to obtain the quality confidence score. Based on the quality confidence score, the corresponding samples are retained or deleted to complete the screening.

Images