Multimodal large model data cleaning and management method and system

Abstract

The present application relates to the technical field of data cleaning, and specifically discloses a method and system for cleaning and managing multimodal large model data. After performing basic format filtering on the original multimodal dataset, the system quantitatively evaluates the image clarity and text fluency in the multimodal dataset through a unimodal quality assessment mechanism to screen out qualified image and text data samples. Then, a semantic-level alignment assessment mechanism is further introduced to perform semantic-level interactive response analysis on each group of corresponding image samples and image text descriptions in the dataset to quantitatively evaluate the degree of semantic alignment between the image samples and the text descriptions, and screen out highly semantically aligned image-text data pairs. This method ensures that the quality of multimodal training samples meets the standards by performing multi-level cleaning and management on the multimodal dataset, and achieves high alignment at the cross-modal semantic level, thereby improving the accuracy and robustness of the multimodal large model in cross-modal understanding and generation tasks.

Description

Technical Field

[0001] The present application relates to the field of data cleaning technology, and more specifically, to a multimodal large model data cleaning and management method and system. Background Art

[0002] With the rapid development of artificial intelligence technology towards multimodal fusion, large-scale multimodal pre-trained models have become the core driving force for cross-modal understanding and generation tasks. The performance of such models is highly dependent on the quality of the training data, but in actual application scenarios, the original multimodal datasets often have significant quality defects: image data may contain low-resolution, blurred and distorted visual information, and text descriptions may contain grammatical errors, information redundancy, or semantic ambiguity. Even more serious is that there are often semantic misalignments or insufficient correlations between image and text data pairs. These data defects can cause cognitive biases in the model during feature extraction and cross-modal alignment, which in turn affects the accuracy and robustness of downstream tasks. Therefore, building an effective multimodal large model data cleaning and governance solution to screen high-quality, highly relevant training samples from massive amounts of raw data is crucial to improving the performance of multimodal large models.

[0003] Currently, for the cleaning and management of multimodal data, existing technologies are mostly limited to the independent processing of a single modality. In the image dimension, simple size filtering or format verification is usually used, and in the text dimension, it relies on regular keyword matching or length truncation. Although such methods can remove obviously invalid data, they fail to deeply evaluate the quality characteristics within each modality, and lack refined measurements of cross-modal semantic associations. Especially when processing image-text pairing data, traditional technologies often ignore the deep correspondence between visual features and semantic descriptions, resulting in low-quality data samples that do not trigger basic filtering rules continuing to contaminate the training set, ultimately causing the model to exhibit feature response offsets or semantic understanding deviations in multimodal alignment tasks.

[0004] Therefore, we look forward to an optimized multimodal large model data cleaning and management method and system. Summary of the Invention

[0005] In order to solve the above technical problems, the present application is proposed. The embodiment of the present application provides a multimodal large model data cleaning and management method and system, which performs basic format filtering on the original multimodal data set, and then quantitatively evaluates the image clarity and text fluency in the multimodal data set through a unimodal quality evaluation mechanism to screen out qualified image and text data samples. Furthermore, by introducing a semantic-level alignment evaluation mechanism, a semantic-level interactive response analysis is performed on each group of corresponding image samples and image text descriptions in the data set to quantitatively evaluate the degree of semantic alignment between the image samples and the text descriptions, thereby further screening out highly semantically aligned image-text data pairs. By performing multi-level cleaning and management on the multimodal data set, this method can ensure that the multimodal training samples are not only of qualified quality, but also highly aligned at the cross-modal semantic level, thereby effectively improving the accuracy and robustness of the multimodal large model in cross-modal understanding and generation tasks.

[0006] According to one aspect of the present application, a multimodal large model data cleaning and management method is provided, which includes:

[0007] Obtain the original multimodal dataset;

[0008] After performing initial data cleaning on the original multimodal data set, extracting a multimodal data sample to be selected therefrom, the multimodal data sample to be selected comprising image data to be selected and a text description to be selected corresponding to the image data to be selected;

[0009] Performing visual feature extraction on the image data to be selected to obtain a visual feature coding vector of the image to be selected;

[0010] Performing semantic feature extraction on the text description to be selected to obtain a semantic feature encoding vector of the text description to be selected;

[0011] Performing semantic-level fine-grained alignment coding on the visual feature coding vector of the image to be selected and the semantic feature coding vector of the text description to be selected to obtain a semantic-level fine-grained interactive response coding vector of the image to be selected and the text description;

[0012] Based on the image-text semantic-level fine-grained interaction response encoding vector to be selected, determine whether to filter the multimodal data sample to be selected.

[0013] According to another aspect of the present application, a multimodal large model data cleaning and management system is provided, which includes:

[0014] Multimodal data acquisition module, used to obtain original multimodal data sets;

[0015] a multimodal data preprocessing module, configured to perform initial data cleaning on the original multimodal data set and extract multimodal data samples to be selected therefrom, wherein the multimodal data samples to be selected include image data to be selected and text descriptions to be selected corresponding to the image data to be selected;

[0016] A visual feature extraction module, configured to extract visual features from the image data to be selected to obtain a visual feature encoding vector of the image to be selected;

[0017] A semantic feature extraction module, configured to extract semantic features from the text description to be selected to obtain a semantic feature encoding vector of the text description to be selected;

[0018] A fine-grained alignment coding module is used to perform semantic-level fine-grained alignment coding on the visual feature coding vector of the image to be selected and the semantic feature coding vector of the text description to be selected to obtain a semantic-level fine-grained interactive response coding vector of the image to be selected and the text description;

[0019] The sample screening module is used to determine whether to filter the multimodal data sample to be selected based on the semantic-level fine-grained interactive response encoding vector of the image-text to be selected.

[0020] Compared with the prior art, the multimodal large model data cleaning and management method and system provided by this application, after performing basic format filtering on the original multimodal dataset, quantitatively evaluates the image clarity and text fluency in the multimodal dataset through a unimodal quality assessment mechanism to screen out qualified image and text data samples. Furthermore, by introducing a semantic-level alignment assessment mechanism, a semantic-level interactive response analysis is performed on each group of corresponding image samples and image text descriptions in the dataset to quantitatively evaluate the degree of semantic alignment between the image samples and the text descriptions, thereby further screening out highly semantically aligned image-text data pairs. This method can ensure that the multimodal training samples not only meet the quality standards, but also achieve a high degree of alignment at the cross-modal semantic level by performing multi-level cleaning and management on the multimodal dataset, thereby effectively improving the accuracy and robustness of the multimodal large model in cross-modal understanding and generation tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The above and other purposes, features, and advantages of the present application will become more apparent through a more detailed description of the embodiments of the present application in conjunction with the accompanying drawings. The accompanying drawings are intended to provide a further understanding of the embodiments of the present application and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the present application and do not constitute a limitation of the present application. In the drawings, the same reference numerals generally represent the same components or steps.

[0022] Figure 1 The present invention provides a flowchart of a multimodal large model data cleaning and management method according to an embodiment of the present application.

[0023] Figure 2 This is a data flow diagram of the multimodal large model data cleaning and management method according to an embodiment of the present application.

[0024] Figure 3 This is a flowchart of sub-step S2 of the multimodal large model data cleaning and management method according to an embodiment of the present application.

[0025] Figure 4 This is a flowchart of sub-step S22 of the multimodal large model data cleaning and management method according to an embodiment of the present application.

[0026] Figure 5 This is a flowchart of sub-step S5 of the multimodal large model data cleaning and management method according to an embodiment of the present application.

[0027] Figure 6 This is a flowchart of sub-step S6 of the multimodal large model data cleaning and management method according to an embodiment of the present application.

[0028] Figure 7 This is a block diagram of a multimodal large model data cleaning and management system according to an embodiment of the present application. DETAILED DESCRIPTION

[0029] As used in this application and the claims, unless the context clearly indicates otherwise, the words "a," "an," "an," and / or "the" are not intended to refer to the singular but may include the plural. Generally speaking, the terms "comprises" and "include" only indicate the inclusion of the steps and elements specifically identified, and these steps and elements do not constitute an exclusive list. A method or apparatus may also include other steps or elements.

[0030] Although the present application makes various references to certain modules in the system according to embodiments of the present application, any number of different modules can be used and run on the user terminal and / or server. The modules are illustrative only, and different aspects of the system and method can use different modules.

[0031] Flowcharts are used in this application to illustrate the operations performed by the systems according to the embodiments of the present application. It should be understood that the preceding or following operations are not necessarily performed in exact order. Instead, the various steps may be processed in reverse order or simultaneously, as needed. Furthermore, other operations may be added to these processes, or one or more operations may be removed from these processes.

[0032] Below, the exemplary embodiments according to the present application will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application, and it should be understood that the present application is not limited to the exemplary embodiments described herein.

[0033] It is worth noting that in this application, all actions to obtain data are carried out in compliance with the relevant data protection laws and policies of the country where they are located and with the authorization given by the owner of the corresponding device.

[0034] In response to the technical problems described in the above background technology, this application proposes a multimodal large model data cleaning and management method, which performs basic format filtering on the original multimodal dataset, and then quantitatively evaluates the image clarity and text fluency in the multimodal dataset through a unimodal quality evaluation mechanism to screen out qualified image and text data samples. Furthermore, by introducing a semantic-level alignment evaluation mechanism, a semantic-level interactive response analysis is performed on each group of corresponding image samples and image text descriptions in the dataset to quantitatively evaluate the degree of semantic alignment between the image samples and the text descriptions, thereby further screening out highly semantically aligned image-text data pairs. This method can ensure that the multimodal training samples not only meet the quality standards, but also achieve high alignment at the cross-modal semantic level by performing multi-level cleaning and management on the multimodal dataset, thereby effectively improving the accuracy and robustness of the multimodal large model in cross-modal understanding and generation tasks.

[0035] Figure 1 The present invention provides a flowchart of a multimodal large model data cleaning and management method according to an embodiment of the present application. Figure 2 The data flow diagram of the multimodal large model data cleaning and management method according to the embodiment of the present application is shown in FIG. Figure 1 and Figure 2 As shown, the multimodal large model data cleaning and management method includes the following steps: S1, obtaining an original multimodal data set; S2, performing initial data cleaning on the original multimodal data set and extracting a multimodal data sample to be selected therefrom, wherein the multimodal data sample to be selected includes image data to be selected and a text description to be selected corresponding to the image data to be selected; S3, performing visual feature extraction on the image data to be selected to obtain a visual feature coding vector of the image to be selected; S4, performing semantic feature extraction on the text description to be selected to obtain a semantic feature coding vector of the text description to be selected; S5, performing semantic-level fine-grained alignment coding on the visual feature coding vector of the image to be selected and the semantic feature coding vector of the text description to be selected to obtain a semantic-level fine-grained interactive response coding vector of the image to be selected and the text; S6, determining whether to filter the multimodal data sample to be selected based on the semantic-level fine-grained interactive response coding vector of the image to be selected and the text.

[0036] In the above-mentioned multimodal large model data cleaning and management method, step S1 obtains the original multimodal dataset. It should be understood that the training of a multimodal large model relies on massive amounts of heterogeneous data. To construct the initial multimodal dataset, this application, based on a distributed data acquisition framework, aggregates multimodal data containing image-text pairs from public or private data sources through customized crawler tools (such as Scrapy combined with Selenium to simulate browser behavior), API interfaces (such as Flickr API, social media open platforms), or database exports (such as MySQL relational databases). Specifically, for image data, standard format files such as JPEG and PNG and associated text descriptions (including user annotations, automatically generated Alt-Text, or manually annotated titles) are collected, and metadata (such as image resolution and text encoding format) is recorded. For text data, natural language descriptions, tags, or annotations corresponding to the image are extracted and normalized through UTF-8 encoding to avoid character set conflicts. In this way, it helps to ensure the diversity and scale of data sources, provide original input for subsequent dataset cleaning and management, and preserve the potential semantic relevance of the original data.

[0037] In practice, acquiring raw multimodal datasets typically relies on a distributed, heterogeneous data collection framework. This framework aggregates data from a variety of sources, including but not limited to web crawlers, open platform API calls, and database exports. These methods efficiently capture multimodal data resources containing image-text pairs from public or private data sources. For example, customized web crawler tools (such as Scrapy combined with Selenium to simulate browser behavior) can automatically extract embedded images and their corresponding textual descriptions from web page structures. Furthermore, standardized interfaces provided by platforms like the Flickr API and social media platforms can systematically acquire image-text pairs with labels, annotations, and metadata. Furthermore, for existing internal historical data assets, database exports can be used to import image-text data stored in relational databases (such as MySQL) into a unified dataset. These diverse collection methods collectively form the technical foundation for constructing multimodal datasets, helping to ensure the breadth and depth of acquired data.

[0038] In terms of image data collection, the main focus is on obtaining files in standard formats, such as common image formats such as JPEG and PNG, and simultaneously collecting related natural language text descriptions. These text descriptions may come from user comments, automatically generated Alt-Text tags, or manually annotated titles, which provide contextual information for the image at the semantic level. At the same time, image metadata information such as resolution, aspect ratio, and camera model will be recorded during the data collection process. Although this information does not directly participate in the subsequent semantic analysis, it has important reference value in evaluating image clarity and judging whether it meets training standards. In addition, to avoid text parsing errors caused by character encoding issues, all text descriptions will be normalized using UTF-8 encoding to eliminate compatibility barriers between different character sets, thereby ensuring the stability and consistency of the text content in the subsequent feature extraction stage.

[0039] It is worth noting that the construction of the original multimodal dataset is not simply a matter of splicing images and text together, but rather requires preserving the underlying semantic association between the two. This semantic association may be explicit, such as with a clear text description below the image, or implicit, such as a certain thematic relevance between images on social platforms and the text updates posted by users. Therefore, during the data collection process, special attention should be paid to the pairing logic between images and text, and data pairs with clear semantic connections should be selected as much as possible to improve the effectiveness of subsequent semantic-level alignment evaluations. For example, on social media platforms, a travel photo is often accompanied by travel experiences or location descriptions shared by users. This type of image-text combination naturally has a high degree of semantic consistency and is suitable as a training sample for multimodal models.

[0040] Furthermore, given the high data volume requirements for multimodal large-scale model training, the construction of the original dataset must strike a balance between acquisition efficiency and data capacity. To this end, distributed data acquisition architectures are widely adopted, significantly improving the speed and stability of data acquisition by executing data capture tasks in parallel across multiple nodes. At the same time, caching mechanisms and deduplication strategies are introduced to avoid repeated collection of identical or highly similar data samples, thereby ensuring data diversity while controlling data redundancy. Furthermore, an incremental update mechanism can be incorporated to regularly capture new content from the data source, maintaining the timeliness and continuous expansion capabilities of the original dataset.

[0041] In the above-mentioned multimodal large model data cleaning and management method, the step S2 is to extract multimodal data samples to be selected from the original multimodal data set after initial data cleaning, and the multimodal data samples to be selected include image data to be selected and text descriptions to be selected corresponding to the image data to be selected. Figure 3Flowchart of sub-step S2 of the multimodal large model data cleaning and management method according to an embodiment of the present application. Figure 3 As shown, the step S2 includes the steps of: S21, performing basic filtering on the original multimodal dataset to obtain an original multimodal dataset after basic filtering; S22, performing single-modal quality assessment on the original multimodal dataset after basic filtering to obtain an original multimodal dataset after secondary filtering; S23, extracting the multimodal data samples to be selected from the original multimodal dataset after secondary filtering.

[0042] Specifically, in a specific example of the present application, the step S21 includes: removing obviously invalid data samples in the original multimodal dataset, and the obviously invalid data samples include images that cannot be loaded, texts that are too short / too long, and specific format errors. It should be understood that invalid data such as images that cannot be loaded, texts with incorrect formats or missing content will inevitably be mixed in during the acquisition, storage or transmission process. Not only will it fail to provide effective information for the training of the multimodal large model, it will also occupy computing resources, increase the training time cost, and may even cause errors in model training. Therefore, in order to preliminarily purify the original multimodal dataset, the present application is based on the principle of data validity judgment, and removes obviously invalid data samples in the original multimodal dataset to obtain the original multimodal dataset after basic filtering. Specifically, in image processing scenarios, an image loading library (such as Python's Pillow library) is used to attempt to load each image in the dataset. If an error occurs during the loading process or the basic information of the image (such as size and color mode) cannot be obtained, the image is determined to be an invalid sample that cannot be loaded and is deleted. In text processing scenarios, a reasonable length range is set for text data, such as stipulating that the text length must be between 10 and 500 words, and text samples that are too short (such as only containing 1-2 meaningless characters) or too long (over 1,000 words and redundant and confusing content) are filtered out. For specific format errors, such as the presence of garbled characters in text data (including unrecognizable special character encodings) and image formats that do not meet common standards (such as non-JPEG, PNG, BMP, etc.), such data samples are directly eliminated. In this way, obviously invalid data samples in the dataset can be quickly and effectively removed, making the original multimodal dataset after basic filtering more concise and effective, thereby reducing the complexity of subsequent data processing.

[0043] Specifically, the step S22 performs a unimodal quality assessment on the original multimodal dataset after the basic filtration to obtain the original multimodal dataset after the secondary filtration. Specifically, since the above-mentioned basic filtration can only remove obviously invalid data but cannot detect the quality of the data content, the original multimodal dataset after the basic filtration may still contain potential low-quality data samples such as blurred images, redundant text or unclear semantics. Therefore, in order to further improve the overall quality of the dataset, this application introduces a unimodal quality assessment mechanism on the basis of basic filtration, and performs quality assessment on the image data and text data in the original multimodal dataset after the basic filtration, so as to screen out image and text data samples with qualified quality. Among them, Figure 4 FIG. 1 is a flowchart of sub-step S22 of the multimodal large model data cleaning and management method according to an embodiment of the present application. Figure 4 As shown, the step S22 includes the following steps: S221, using a pre-trained model to evaluate the image clarity of each image data and the text fluency of each text description in the original multimodal dataset after basic filtering; S222, based on a comparison between the image clarity and a preset clarity threshold, filtering out the image data in the original multimodal dataset whose quality does not meet the requirements; S223, based on a comparison between the text fluency and a preset fluency threshold, filtering out the text descriptions in the original multimodal dataset after basic filtering whose quality does not meet the requirements.

[0044] Specifically, for image data, using image clarity as a quality assessment metric, a pre-trained Noise2Noise denoising network was used to denoise each image in the original multimodal dataset after basic filtering. The Structural Similarity Index (SSIM) between the input image data and the denoised output of the model was calculated to reveal the noise level of the image data and thus assess image clarity. Furthermore, the SSIM was used as a quantitative score for image clarity. Image data below a preset clarity threshold was considered low-quality image samples and eliminated, while image data above the preset clarity threshold was considered high-quality image samples and retained. For text data, text fluency is used as a quality assessment metric. A language fluency discriminator is constructed based on the RoBERTa-large model. The input text description is segmented and processed into subwords (Byte-Pair Encoding, vocabulary size 50,265). The model then obtains the predicted probability distribution of the next position of each subword unit. The perplexity of the text description is calculated based on the cross-entropy loss (a natural exponential function is applied to the cross-entropy loss function value). Lower perplexity indicates better fluency of the text description, meaning the language is more natural, coherent, and easy to understand. Furthermore, the inverse of the perplexity is used as a quantitative score of text fluency. Text data below a preset fluency threshold is considered low-quality text samples and is eliminated, while text data above the preset fluency threshold is considered high-quality text samples and is retained. This method further eliminates potential low-quality data samples, ensuring that the retained image and text data samples have a high level of quality within their respective modalities, meeting high-quality training requirements.

[0045] Specifically, the step S23 extracts the multimodal data samples to be selected from the original multimodal data set after the secondary filtration. Specifically, despite the unimodal filtration, the cross-modal semantic mismatch problem in the multimodal data set (such as the image content is irrelevant to the text description) may still pollute the data set (such as the text illustration of "dog running" is actually a static cat image), resulting in semantic understanding deviations in the large multimodal model during the training process, affecting the accuracy and generalization ability of the model. Therefore, in order to accurately locate and eliminate semantically mismatched multimodal data samples, this application takes each group of corresponding image data and text data as a multimodal data sample to be selected based on the image-text pairing relationship in the original multimodal data set after the secondary filtration, so as to focus on the cross-modal alignment problem that may exist between the two and perform semantic consistency verification.

[0046] In the above-mentioned multimodal large model data cleaning and management method, the step S3 performs visual feature extraction on the image data to be selected to obtain a visual feature encoding vector of the image to be selected. In a specific example of the present application, the step S3 includes: inputting the image data to be selected into a visual encoder based on the ViT model to perform visual feature extraction to obtain a visual feature encoding vector of the image to be selected. It should be understood that since the image data itself is a pixel matrix, it cannot be directly used for semantic consistency comparison with the text description. Therefore, in order to effectively extract the image content semantics of the image data to be selected, the present application is based on the Visual Transformer (ViT) model (ViT-B / 16 architecture, pre-trained on ImageNet-21k), which extracts visual features from the image data to be selected to capture key information in the image, such as objects, scenes, actions, etc., to convert it into a high-dimensional visual feature encoding vector of the image to be selected, providing a basis for subsequent semantic consistency comparison with the text description. Specifically, the ViT model, as an image feature extraction model based on the Transformer architecture, divides the image data to be selected into a series of non-overlapping small blocks (patches) and linearly embeds each image block into a fixed-size vector as the input sequence of the model. It uses the self-attention mechanism of the Transformer architecture to process the sequence of image block embedding vectors to capture the global dependencies between image blocks, thereby generating a deep feature representation of the image data to be selected and obtaining the visual feature encoding vector of the image to be selected. Based on this, the visual feature encoding vector of the image to be selected, as a vectorized representation of the semantic connotation of the image data to be selected, can accurately describe the core semantic content in the image data, and provide strong support for the subsequent semantic consistency verification with the text description.

[0047] In the above-mentioned multimodal large model data cleaning and management method, the step S4 extracts semantic features from the text description to be selected to obtain a semantic feature encoding vector of the text description to be selected. Similarly, the text description to be selected is a natural language text. In order to convert it into a semantic feature representation that is understandable and operable by a computer, so as to align it with the visual feature space, the present application uses a pre-trained BERT model to perform semantic analysis on the text description to be selected, and extracts the contextual semantic embedding representation of the text description to be selected, so as to obtain a semantic feature encoding vector of the text description to be selected. Specifically, the BERT model performs word segmentation on the input text description, and converts the word segmentation results into a corresponding word embedding vector sequence through a word embedding layer, and uses the self-attention mechanism of the Transformer architecture to perform deep context modeling on the word embedding vector sequence, captures the semantic relationship between words, and mines the entity relationship (such as the action-object association in "boys playing football"), action logic and other fine-grained semantic information in the text description to be selected, thereby generating a semantic feature encoding vector of the text description to be selected.

[0048] In the above-mentioned multimodal large model data cleaning and management method, the step S5 performs semantic-level fine-grained alignment encoding on the visual feature coding vector of the image to be selected and the semantic feature coding vector of the text description to be selected to obtain the semantic-level fine-grained interactive response coding vector of the image to be selected and the text. It should be understood that the present application takes into account that simple cosine similarity calculation cannot model the fine-grained semantic association pattern of cross-modal features (such as local semantic matching or implicit relationship). Therefore, in order to refine the measurement of the degree of alignment between images and texts (such as the correspondence between the "red car" in the text description and the local red area in the image), the present application proposes a cross-modal semantic-level fine-grained alignment coding method, which mines the local semantic matching relationship and implicit semantic association between the two by performing fine-grained semantic alignment interaction on the visual feature coding vector of the image to be selected and the semantic feature coding vector of the text description to be selected in the local space, and integrates the information of the global semantic space through the inference aggregation mechanism to obtain the semantic-level fine-grained interactive response coding vector of the image to be selected and the text. Among them, Figure 5 Flowchart of sub-step S5 of the multimodal large model data cleaning and management method according to an embodiment of the present application. Figure 5As shown, the step S5 includes the steps of: S51, performing local segmentation on the visual feature coding vector of the image to be selected and the semantic feature coding vector of the text description to be selected to obtain a sequence of ordered coding vectors of local visual features of the image to be selected and a sequence of ordered coding vectors of local semantic features of the text description to be selected; S52, inputting each group of corresponding ordered coding vectors of local visual features of the image to be selected and ordered coding vectors of local semantic features of the text description to be selected in the sequence of ordered coding vectors of local visual features of the image to be selected and the sequence of ordered coding vectors of local semantic features of the text description to be selected into the semantic level transfer interaction response reasoning unit to obtain a sequence of local semantic interaction response coding matrices of the image to be selected-text description; S53, performing semantic transfer coding on the sequence of local semantic interaction response coding matrices of the image to be selected-text description to obtain the semantic level fine-grained interaction response coding vector of the image to be selected-text.

[0049] Specifically, in a specific example of the present application, step S51 includes: first, performing an ordered arrangement of the visual feature coding vector of the image to be selected and the semantic feature coding vector of the text description to be selected based on the size of the eigenvalues ​​to obtain an ordered arrangement coding vector of the visual features of the image to be selected and an ordered arrangement coding vector of the semantic features of the text description to be selected, which can be expressed as follows:

[0050]

[0051]

[0052] in, Represents the visual feature encoding vector of the image to be selected, Represents the semantic feature encoding vector of the text to be selected, Indicates the sorting operation on vector elements. Represents the ordered arrangement coding vector of the visual features of the image to be selected, Represents the ordered arrangement encoding vector of the semantic features of the text to be selected.

[0053] That is, by reconstructing the sequence distribution of feature elements, the implicit interference introduced by the difference in arrangement order of the visual feature coding vector of the image to be selected and the semantic feature coding vector of the text description to be selected in the original space is eliminated, thereby providing a feature expression basis with consistent numerical distribution for cross-modal alignment. Specifically, through ordered arrangement, the visual feature coding vector of the image to be selected and the semantic feature coding vector of the text description to be selected are reordered according to the intensity of the element values, so that they form a comparable standardized structure in the numerical distribution dimension. The generated ordered arrangement coding vector of the visual features of the image to be selected and the ordered arrangement coding vector of the semantic features of the text description to be selected reduce the redundant noise of the feature sequence and enhance the robustness of cross-modal local feature matching.

[0054] Then, the ordered arrangement coding vectors of the visual features of the image to be selected and the ordered arrangement coding vectors of the semantic features of the text description to be selected are subjected to equal granularity feature segmentation to obtain a sequence of ordered coding vectors of the local visual features of the image to be selected and a sequence of ordered coding vectors of the local semantic features of the text description to be selected, which can be expressed as follows:

[0055]

[0056]

[0057] in, 、 、 and They represent the first, second, and third ordered encoding vectors of the local visual features of the image to be selected. and ordered encoding vectors of local visual features of the image to be selected, is the number of ordered coding vectors of the local visual features of the image to be selected, 、 、 and They represent the first, second, and third in the sequence of ordered encoding vectors of local semantic features of the selected text description. and The text to be selected describes the ordered encoding vector of local semantic features, represents the feature segmentation function.

[0058] That is, by aligning the feature segmentation granularity of the ordered arrangement coding vector of the visual features of the image to be selected and the ordered arrangement coding vector of the semantic features of the text description to be selected, the subsequent cross-modal interactive response reasoning can capture the potential correlation between the local visual area and the semantic fragment at the same structural scale. Specifically, cutting the globally ordered ordered arrangement coding vector of the visual features of the image to be selected and the ordered arrangement coding vector of the semantic features of the text description to be selected into a sequence of ordered coding vectors of the local visual features of the image to be selected and a sequence of ordered coding vectors of the local semantic features of the text description to be selected with consistent dimensions can eliminate the interference of the difference in feature dimensions of different modalities on fine-grained alignment, provide structurally matchable fragmented feature input for cross-modal local semantic interaction modeling, and enhance sensitivity to micro-semantic coupling relationships.

[0059] Specifically, the step S52 is expressed as follows:

[0060]

[0061] in, represents the linear transformation matrix, represents the ReLU activation function, represents matrix multiplication, express and The encoding matrix of the local semantic interaction response between the selected image and text description.

[0062] Specifically, by establishing a deep nonlinear interaction modeling mechanism for cross-modal local features, the potential coupling relationships and conflicts between visual and semantic segments are revealed. A neural network model is used to dynamically model the implicit semantic response pattern between the sequence of ordered encoding vectors of local visual features of the selected image and the ordered encoding vectors of local semantic features of the selected text description, thereby quantifying the inherent consistency of the image-text data pair at a fine-grained semantic level. The resulting encoding matrix of the local semantic interaction response between the selected image and text description can characterize the strength of explicit associations between cross-modal segments and identify subtle local semantic deviations, thereby improving the parsing depth and discriminant accuracy of subsequent cross-modal alignment evaluation.

[0063] In particular, considering that the adjustment of the segmentation granularity of the equal-granularity feature segmentation not only affects the regional division granularity dimension of the image-to-be-selected-text description local semantic interaction response coding matrix, it also directly affects the mutual construction expression framework between the sequence of the corresponding ordered coding vectors of the local visual features of the image to be selected and the ordered coding vectors of the local semantic features of the text description to be selected. Therefore, in a preferred example of the present application, the step S423 includes: first, based on the semantic-level interaction response connection density between each group of corresponding ordered coding vectors of the local visual features of the image to be selected and the ordered coding vectors of the local semantic features of the text description to be selected, each image-to-text description local semantic interaction response coding matrix to be selected in the sequence of the image-to-text description local semantic interaction response coding matrix to be selected is subjected to sparsity constraints to obtain a sequence of optimized image-to-text description local semantic interaction response coding matrices to be selected.

[0064] Specifically, since the image-to-be-selected-text description local semantic interaction response encoding matrix is ​​used as a carrier of spatial mutual construction quantitative representation, its row vector modulus represents the proximity constraint parameter. If the proximity constraint parameter, that is, the row vector modulus, is used as the intensity parameter of the low-rank spatial feature quantization, then the low-rank spatial feature quantization of the image-to-be-selected-text description local semantic interaction response encoding matrix, that is, the F norm, is It should follow the Poisson distribution relationship:

[0065]

[0066] in, represents factorial, represents the Frobenius norm, express The row vector norm of represents an exponential function with a natural constant as the base, represents the Poisson distribution parameter, which characterizes the expected frequency of semantic interaction between the image to be selected and the text description. From this, the Poisson distribution parameter can be solved .

[0067] In this way, cross-domain collaborative mutual feedback is introduced by introducing the local interaction constraint strength as and the mean expectation is In the case of a Poisson distribution dynamic process generated by and Describing the boundary-related feature expression of the interaction space metric descriptor, the semantic-level interaction response connection density between the two local regions is further determined as:

[0068]

[0069] in, express and The semantic level interaction response connection density between represents the absolute value operation, represents the two-norm.

[0070] Then, the proximity scale adjustment parameters are iteratively optimized and adjusted based on the semantic-level interaction response connection density:

[0071]

[0072] in, Indicates the optimized .

[0073] Finally, after optimizing the adjustment Reconstrain the image-text description local semantic interaction response encoding matrix to be selected Sparsity:

[0074]

[0075] in, express The corresponding optimized image-text description local semantic interaction response encoding matrix.

[0076] That is, while maintaining the desired connectivity In this case, the global mutual construction space quantization parameters are dynamically corrected through the regularization constraint driven by probability fluctuations, so that the mutual construction probability fluctuations of the structured information interaction in the cross-domain mutual construction system are restricted, thereby suppressing the risk of neighborhood overfitting and improving the overall feature expression performance.

[0077] Then, the sequence of the optimized image-to-be-selected-text description local semantic interaction response encoding matrix is ​​input into the LSTM transfer encoding module with an attention mechanism to obtain the semantic-level fine-grained interaction response encoding vector of the image-to-be-selected-text, which is expressed as follows:

[0078]

[0079]

[0080]

[0081] in, represents the matrix flattening operation, express The expanded image-text description local semantic interaction response encoding vector is obtained. Represents the exponential function operation with e as the base, is the feature modulation function based on the attention mechanism, express and The optimized image-text description local semantic interaction response encoding matrix between the selected images, express and The optimized image-text description local semantic interaction response encoding matrix between the selected images, represents the LSTM model, Represents the semantic-level fine-grained interactive response encoding vector of the image-text to be selected.

[0082] Specifically, the long-range dependency modeling capabilities of LSTM are leveraged to capture the evolutionary patterns of cross-regional semantic associations in the optimized encoding matrix of the local semantic interaction response between the image and text description to be selected. At the same time, the attention mechanism is leveraged to dynamically strengthen the weight distribution of key local interactions, thereby reconstructing a composite feature expression of multi-granular semantic alignment at the global level. The resulting semantic-level fine-grained interaction response encoding vector for the image and text to be selected not only incorporates the topological structure of semantic associations across local regions, but also highlights discriminative interaction features through attention-guided contextual encoding, providing a quantitative basis that combines both integrity and sensitivity for filtering decisions in data cleaning.

[0083] In the above multimodal large model data cleaning and management method, the step S6 determines whether to filter the multimodal data sample to be selected based on the semantic level fine-grained interactive response encoding vector of the image-text to be selected. Figure 6 Flowchart of sub-step S6 of the multimodal large model data cleaning and management method according to an embodiment of the present application. Figure 6 As shown, the step S6 includes the steps of: S61, feature decoding the semantic-level fine-grained interaction response encoding vector of the image-text to be selected to obtain a semantic-level alignment coefficient; S62, determining whether to filter the multimodal data sample to be selected based on a comparison between the semantic-level alignment coefficient and a preset semantic alignment threshold.

[0084] Specifically, step S61 performs feature decoding on the semantically fine-grained interaction response encoding vector of the image-to-be-selected text to obtain a semantic alignment coefficient. It should be understood that the semantically fine-grained interaction response encoding vector of the image-to-be-selected text serves as a deep modeling representation of the implicit semantic relationship between the image data and the text description, encompassing the degree of match and interaction information between the two at the fine-grained semantic level. To further transform high-dimensional semantic interaction features into interpretable alignment scores to support subsequent filtering decisions, this application utilizes a multi-layer perceptron (MLP) based on a regression task framework to achieve feature-to-scalar mapping. Specifically, during the training phase of the MLP model, a contrastive learning strategy is employed: positive samples (high-quality, manually annotated image-text pairs) and negative samples (noise pairs with randomly replaced text, in a ratio of 1:3) are constructed. The model is optimized using triplet loss (with an interval of α = 0.2) to ensure that the output for positive samples is close to 1 and for negative samples is close to 0. Label smoothing (ε = 0.1) is also introduced to mitigate overfitting. In practical applications, the semantic-level fine-grained interaction response encoding vector of the image to be selected and the text is input into an MLP model containing two hidden layers (using the GeLU activation function). Through the nonlinear transformation of the hidden layer, the high-dimensional semantic-level fine-grained interaction response encoding vector of the image to be selected and the text is gradually reduced in dimension, and the Sigmoid function is used to constrain the final output to a semantic alignment coefficient in the range of [0,1], so as to reflect the degree of semantic consistency between the image and the text.

[0085] Specifically, step S62 determines whether to filter the multimodal data samples to be selected based on the comparison between the semantic-level alignment coefficient and the preset semantic alignment threshold. That is, the semantic-level alignment coefficient is compared with the preset semantic alignment threshold (such as 0.7). If the semantic-level alignment coefficient is greater than or equal to the threshold, it is considered that the image samples and text description samples in the multimodal data samples to be selected have a high degree of semantic alignment and are retained; conversely, if the semantic-level alignment coefficient is less than the threshold, the group of data samples is filtered out. In this way, multimodal data samples with a high degree of semantic alignment can be effectively screened out, reducing the interference of low-quality, semantically mismatched data on model training, thereby improving the accuracy and stability of the model in the multimodal alignment task, enabling the model to better learn cross-modal semantic associations, and improving the model's semantic understanding ability.

[0086] In summary, the multimodal large model data cleaning and management method based on the embodiment of the present application is explained, which performs basic format filtering on the original multimodal dataset, and then quantitatively evaluates the image clarity and text fluency in the multimodal dataset through a unimodal quality assessment mechanism to screen out qualified image and text data samples. Furthermore, by introducing a semantic-level alignment assessment mechanism, a semantic-level interactive response analysis is performed on each group of corresponding image samples and image text descriptions in the dataset to quantitatively evaluate the degree of semantic alignment between the image samples and the text descriptions, thereby further screening out highly semantically aligned image-text data pairs. This method can ensure that the multimodal training samples not only meet the quality standards, but also achieve a high degree of alignment at the cross-modal semantic level by performing multi-level cleaning and management on the multimodal dataset, thereby effectively improving the accuracy and robustness of the multimodal large model in cross-modal understanding and generation tasks.

[0087] Furthermore, a multimodal large model data cleaning and management system is also provided.

[0088] Figure 7 FIG is a block diagram of a multimodal large model data cleaning and management system according to an embodiment of the present application. Figure 7 As shown, the multimodal large model data cleaning and management system 100 according to the embodiment of the present application includes: a multimodal data acquisition module 110, which is used to acquire an original multimodal data set; a multimodal data preprocessing module 120, which is used to perform initial data cleaning on the original multimodal data set and extract a multimodal data sample to be selected therefrom, wherein the multimodal data sample to be selected includes image data to be selected and a text description to be selected corresponding to the image data to be selected; a visual feature extraction module 130, which is used to extract visual features from the image data to be selected to obtain a visual feature code of the image to be selected. vector; a semantic feature extraction module 140, used to extract semantic features of the text description to be selected to obtain a semantic feature coding vector of the text description to be selected; a fine-grained alignment coding module 150, used to perform semantic-level fine-grained alignment coding on the visual feature coding vector of the image to be selected and the semantic feature coding vector of the text description to be selected to obtain a semantic-level fine-grained interactive response coding vector of the image to be selected and the text; a sample screening module 160, used to determine whether to filter the multimodal data sample to be selected based on the semantic-level fine-grained interactive response coding vector of the image to be selected and the text.

[0089] Here, those skilled in the art will understand that the specific operations of each module in the multimodal large model data cleaning and management system have been described in the above reference. Figures 1 to 6 The multimodal large model data cleaning and governance method has been introduced in detail, and therefore, its repeated description will be omitted.

[0090] The basic principles of the present invention have been described above in conjunction with specific embodiments. However, it should be noted that the advantages, strengths, and effects mentioned in the present invention are merely illustrative and non-limiting, and should not be construed as necessarily possessed by each embodiment of the present invention. Furthermore, the specific details of the above embodiments are provided for illustrative purposes and to facilitate understanding, and are not intended to be limiting. These details do not necessarily limit the present invention to being implemented using these specific details.

[0091] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, please refer to the relevant description of other embodiments. In the several embodiments provided by the present invention, it should be understood that the disclosed system and method can be implemented in other ways. For example, the system embodiment described above is only schematic. For example, the unit division is only a logical function division, and there may be other division methods in actual implementation. The units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the scheme of this embodiment.

[0092] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above and that the invention can be embodied in other specific forms without departing from the spirit or essential characteristics of the invention. Therefore, the embodiments should be considered in all respects as illustrative and non-restrictive, and the scope of the invention is defined by the appended claims, not the foregoing description, and all variations within the meaning and range of equivalents of the claims are intended to be encompassed therein. Any reference to a figure in a claim should not be construed as limiting the claim to which it relates.

[0093] In addition, it is obvious that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units stated in the system claims can also be implemented by one unit through software or hardware.

[0094] Finally, it should be noted that the above description has been provided for the purpose of illustration and description. In addition, the above embodiments are intended only to illustrate the technical solutions of the present invention and are not intended to be limiting. Although the technical solutions may be modified or replaced with equivalents with reference to the preferred embodiments, they do not depart from the spirit and scope of the technical solutions of the present invention.

Claims

1. A multimodal large model data cleaning and management method, characterized by: include: Obtain the original multimodal dataset; After performing initial data cleaning on the original multimodal data set, extracting a multimodal data sample to be selected therefrom, the multimodal data sample to be selected comprising image data to be selected and a text description to be selected corresponding to the image data to be selected; Performing visual feature extraction on the image data to be selected to obtain a visual feature coding vector of the image to be selected; Performing semantic feature extraction on the text description to be selected to obtain a semantic feature encoding vector of the text description to be selected; Performing semantic-level fine-grained alignment coding on the visual feature coding vector of the image to be selected and the semantic feature coding vector of the text description to be selected to obtain a semantic-level fine-grained interactive response coding vector of the image to be selected and the text description; Determining whether to filter the multimodal data sample to be selected based on the semantic-level fine-grained interactive response encoding vector of the image-text to be selected; Performing semantic-level fine-grained alignment encoding on the visual feature encoding vector of the image to be selected and the semantic feature encoding vector of the text description to be selected to obtain a semantic-level fine-grained interactive response encoding vector of the image to be selected and the text, including: Performing an ordered arrangement of the visual feature coding vector of the image to be selected and the semantic feature coding vector of the text description to be selected based on the size of the eigenvalues ​​to obtain an ordered arrangement coding vector of the visual features of the image to be selected and an ordered arrangement coding vector of the semantic features of the text description to be selected; Performing equal-granularity feature segmentation on the ordered arrangement coding vectors of the visual features of the image to be selected and the ordered arrangement coding vectors of the semantic features of the text description to be selected to obtain a sequence of ordered coding vectors of local visual features of the image to be selected and a sequence of ordered coding vectors of local semantic features of the text description to be selected; Inputting each corresponding set of the ordered coding vectors of the local visual features of the images to be selected and the ordered coding vectors of the local semantic features of the text descriptions to be selected into a semantic level transfer interactive response reasoning unit to obtain a sequence of the local semantic interactive response coding matrices of the images to be selected and the text descriptions to be selected; Semantic transfer encoding is performed on the sequence of the local semantic interaction response encoding matrices of the image to be selected and the text description to obtain the semantic-level fine-grained interaction response encoding vector of the image to be selected and the text description to be selected.

2. The multimodal large model data cleaning and management method according to claim 1 is characterized in that: After performing initial data cleaning on the original multimodal dataset, extracting multimodal data samples to be selected, including: Performing basic filtering on the original multimodal dataset to obtain a basic filtered original multimodal dataset; Performing a single-modality quality assessment on the original multimodal dataset after basic filtering to obtain an original multimodal dataset after secondary filtering; The multimodal data samples to be selected are extracted from the original multimodal data set after the secondary filtering.

3. The multimodal large model data cleaning and management method according to claim 2 is characterized in that: Performing basic filtering on the original multimodal dataset to obtain a basic filtered original multimodal dataset includes: Obvious invalid data samples in the original multimodal dataset are removed, where the obvious invalid data samples include images that cannot be loaded, text that is too short / too long, and specific format errors.

4. The multimodal large model data cleaning and management method according to claim 3 is characterized in that: Performing a single-modality quality assessment on the original multimodal dataset after basic filtering to obtain the original multimodal dataset after secondary filtering, including: Using a pre-trained model to evaluate the image clarity of each image data and the text fluency of each text description in the original multimodal dataset after the basic filtering; Based on a comparison between the image clarity and a preset clarity threshold, filtering image data in the original multimodal dataset that does not meet quality requirements; Based on the comparison between the text fluency and a preset fluency threshold, the text descriptions in the original multimodal dataset after basic filtering that do not meet the quality requirements are filtered out.

5. The multimodal large model data cleaning and management method according to claim 4 is characterized in that: Extracting visual features from the image data to be selected to obtain a visual feature encoding vector of the image to be selected includes: The image data to be selected is input into a visual encoder based on the ViT model to extract visual features to obtain a visual feature encoding vector of the image to be selected.

6. The multimodal large model data cleaning and management method according to claim 5 is characterized in that: Performing semantic transfer encoding on the sequence of the local semantic interaction response encoding matrices of the image to be selected and the text description to obtain the semantic-level fine-grained interaction response encoding vector of the image to be selected and the text, including: Based on the semantic-level interaction response connection density between each group of the corresponding ordered coding vectors of the local visual features of the images to be selected and the ordered coding vectors of the local semantic features of the text descriptions to be selected, sparsity constraints are applied to each of the local semantic interaction response coding matrices of the images to be selected and the text descriptions to be selected, so as to obtain a sequence of optimized local semantic interaction response coding matrices of the images to be selected and the text descriptions to be selected; The sequence of the optimized image-to-be-selected-text description local semantic interaction response encoding matrix is ​​input into the LSTM transfer encoding module with an attention mechanism to obtain the semantic-level fine-grained interaction response encoding vector of the image-to-be-selected-text.

7. The multimodal large model data cleaning and management method according to claim 6 is characterized in that: Determining whether to filter the multimodal data sample to be selected based on the semantic-level fine-grained interactive response encoding vector of the image-text to be selected includes: Performing feature decoding on the semantic-level fine-grained interactive response encoding vector of the image-text to be selected to obtain a semantic-level alignment coefficient; Based on the comparison between the semantic-level alignment coefficient and a preset semantic alignment threshold, it is determined whether to filter the multimodal data sample to be selected.

8. A multimodal large model data cleaning and management system for executing the method according to any one of claims 1 to 7, characterized in that: include: Multimodal data acquisition module, used to obtain original multimodal data sets; a multimodal data preprocessing module, configured to perform initial data cleaning on the original multimodal data set and extract multimodal data samples to be selected therefrom, wherein the multimodal data samples to be selected include image data to be selected and text descriptions to be selected corresponding to the image data to be selected; A visual feature extraction module, configured to extract visual features from the image data to be selected to obtain a visual feature encoding vector of the image to be selected; A semantic feature extraction module, configured to extract semantic features from the text description to be selected to obtain a semantic feature encoding vector of the text description to be selected; A fine-grained alignment coding module is used to perform semantic-level fine-grained alignment coding on the visual feature coding vector of the image to be selected and the semantic feature coding vector of the text description to be selected to obtain a semantic-level fine-grained interactive response coding vector of the image to be selected and the text description; The sample screening module is used to determine whether to filter the multimodal data sample to be selected based on the semantic-level fine-grained interactive response encoding vector of the image-text to be selected.

Images