Joint diversity data screening method and system for multimodal large model pre-training

Abstract

The present application relates to the technical field of multi-modal data processing, and particularly relates to a multi-modal large model pre-training joint diversity data screening method and system.The method comprises the following steps: encoding a text-image paired sample to obtain a text feature vector and an image feature vector; generating a cross-modal alignment feature vector through a gated cross-modal fusion layer; calculating a text diversity loss, an image diversity loss and an alignment diversity loss respectively, and constructing a joint diversity objective function; in each round of greedy selection iteration, dynamically adjusting a modal weight coefficient according to the current diversity loss of each modal, and traversing a candidate sample, calculating the loss increment of the joint diversity objective function after each candidate sample is added to the selected sample set, and finally obtaining a multi-modal pre-training target data set that satisfies a preset diversity constraint; and the adaptive balance of multi-modal diversity can be realized.

Description

Technical Field

[0001] This invention relates to the field of multimodal data processing technology, and in particular to a method and system for joint diversity data screening during multimodal large model pre-training. Background Technology

[0002] Multimodal Large Language Models (MLLMs) rely on massive amounts of text-image pairing data for contrastive learning pre-training, enabling deep semantic alignment between visual information and natural language. They serve as a core infrastructure for downstream tasks such as image description generation, visual question answering, cross-modal image-text retrieval, and medical image-assisted diagnosis. Visual language pre-training paradigms, exemplified by CLIP (Contrastive Language-Image Pre-training), simultaneously optimize text and visual encoders, mapping semantically related image-text pairs to a shared embedding space, exhibiting strong cross-modal transfer capabilities. As the number of model parameters continues to increase, the scale, quality, and diversity of the pre-training data directly determine the model's final generalization performance. Currently, mainstream industry training datasets (LAION, CC3M, etc.) are constructed from massive amounts of image and text data crawled from the internet, with sample sizes reaching hundreds of millions to billions, containing a large number of repetitive, low-quality, and semantically homogeneous samples. If the entire original data is used directly for pre-training, it will not only generate extremely high computing power and storage overhead, but also cause the model to overfit high-frequency repetitive semantic patterns, leading to feature space dimensionality collapse and significantly weakening the model's ability to understand niche, abstract, and professional scenarios.

[0003] Traditional pre-training data selection methods are mainly divided into three categories: domain similarity-based screening methods, which use pre-trained surrogate models to calculate the feature similarity between candidate samples and high-quality target domains, and prioritize the selection of samples that are semantically close to the target domain, which can slightly improve the accuracy of tasks within the domain; diversity screening methods for single-text modalities, with the DiSF algorithm as a typical example, design a greedy diversity selection strategy for pure text pre-training data, and make the distribution of text feature dimensions tend to be uniform by minimizing the Frobenius norm of the covariance matrix of selected text features, thereby achieving redundancy removal of text samples; and batch-level dynamic joint screening methods, represented by the JEST algorithm, dynamically complete the batch sample optimization during the training of multimodal models, rely on contrastive learning loss to measure the learnability of samples within a batch, and schedule training batches in real time to accelerate convergence.

[0004] Therefore, traditional pre-training data selection methods often suffer from modal diversity imbalance and insufficient coverage of cross-modal association patterns due to the lack of consideration for the joint distribution of multimodal feature spaces and the absence of adaptive modal balancing mechanisms and explicit constraints on cross-modal association patterns. Summary of the Invention

[0005] Based on this, in order to solve the above technical problems, a method and system for joint diversity data screening in multimodal large model pre-training is provided, which can achieve adaptive balance of multimodal diversity and reduce the computational resource consumption of multimodal pre-training.

[0006] A joint diversity data filtering method for pre-training a multimodal large model, the method comprising:

[0007] A candidate dataset including various text-image pairing samples is obtained. The text-image pairing samples are encoded using a text encoder and a visual encoder respectively to obtain text feature vectors and image feature vectors. The text feature vectors and image feature vectors are interactively fused through a gated cross-modal fusion layer to generate cross-modal alignment feature vectors. The text feature vectors, image feature vectors, and cross-modal alignment feature vectors are standardized to obtain standardized text feature vectors, standardized image feature vectors, and standardized alignment feature vectors.

[0008] Based on the standardized text feature vector, standardized image feature vector, and standardized alignment feature vector of the selected sample set, the text diversity loss, image diversity loss, and alignment diversity loss are calculated respectively. The learnable modality weight coefficients are used to weight and sum the various diversity losses to construct the joint diversity objective function.

[0009] Clustering is performed on the standardized aligned feature vectors of all samples to obtain each text-image semantic association pattern cluster, and the association pattern cluster number to which each sample belongs is recorded.

[0010] In each round of greedy selection iteration, the modality weight coefficients are dynamically adjusted according to the current diversity loss of each modality, and the candidate samples are traversed. The loss increment of the joint diversity objective function after each candidate sample is added to the selected sample set is calculated. The target candidate sample is added to the selected sample set by combining the distribution balance of the associated mode cluster to which the unselected candidate sample belongs in the selected sample set.

[0011] When the total number of samples in the selected sample set reaches the preset data budget, the iteration stops, and a multimodal pre-trained target dataset that meets the preset diversity constraints is obtained.

[0012] In one embodiment, a candidate dataset including various text-image pairing samples is obtained, and the text-image pairing samples are encoded using a text encoder and a visual encoder respectively to obtain text feature vectors and image feature vectors, including:

[0013] Obtain a candidate dataset including various text-image pairing samples, perform word segmentation on the text sequences in the candidate dataset to generate word sequences, and input the word sequences into a pre-trained text encoder;

[0014] The word embedding layer of the text encoder maps each word sequence into a continuous vector. After multi-layer self-attention transformation, each word representation is obtained. The word representations are then aggregated using an average pooling strategy to obtain a fixed-dimensional text feature vector.

[0015] Each image in the candidate dataset is divided into a sequence of non-overlapping image blocks. After linear projection, the image block embedding is obtained. Classification words are concatenated at the beginning of the sequence and then input into a pre-trained visual encoder.

[0016] The visual encoder performs multi-layer self-attention transformation, combines and aggregates the final output of the classification terms and the final output of all image blocks, and obtains a fixed-dimensional image feature vector after layer normalization.

[0017] In one embodiment, the text feature vector and image feature vector are interactively fused through a gated cross-modal fusion layer to generate a cross-modal aligned feature vector, including:

[0018] The text feature vector and the image feature vector are concatenated along the feature dimension to obtain the concatenated joint vector;

[0019] The joint vector is input into the gated network, and the gated vector is calculated through linear mapping and nonlinear transformation by the Sigmoid activation function.

[0020] The text feature vector and the image feature vector are mapped to the aligned feature space using the text projection matrix and the image projection matrix, respectively, to obtain the projected text feature vector and the projected image feature vector.

[0021] The gated vector is used to perform element-wise weighted fusion of the projected text feature vector and the projected image feature vector to generate a cross-modal aligned feature vector.

[0022] In one embodiment, based on the standardized text feature vector, standardized image feature vector, and standardized alignment feature vector of the selected sample set, text diversity loss, image diversity loss, and alignment diversity loss are calculated respectively. The various diversity losses are then weighted and summed using learnable modality weight coefficients to construct a joint diversity objective function, including:

[0023] Obtain the current selected sample set, extract the standardized text feature vector, standardized image feature vector and standardized alignment feature vector of each sample in the selected sample set, arrange each feature vector in rows, and construct the text feature matrix, image feature matrix and alignment feature matrix respectively;

[0024] Calculate the feature covariance matrices corresponding to the text feature matrix, image feature matrix, and alignment feature matrix respectively, and extract all off-diagonal elements in each feature covariance matrix. Square the off-diagonal elements and sum them. Use the summation result as the diversity loss value to obtain the text diversity loss, image diversity loss, and alignment diversity loss.

[0025] Modal weight coefficients are configured for the corresponding text, image, and alignment dimensions, respectively. The joint diversity objective function is obtained by weighting and summing each modal weight coefficient with the corresponding text diversity loss, image diversity loss, and alignment diversity loss.

[0026] In one embodiment, clustering is performed based on the standardized aligned feature vectors of all samples to obtain various text-image semantic association pattern clusters, and the cluster number to which each sample belongs is recorded, including:

[0027] Obtain the standardized alignment feature vectors of all candidate samples, arrange the standardized alignment feature vectors by row, and construct the alignment feature matrix;

[0028] The number of clusters to divide all candidate samples into clusters of associated patterns is set, and the number of sample features to be selected from the standardized aligned feature vector is selected as the initial cluster centers using a distributed initialization strategy.

[0029] Calculate the distance between each of the standardized alignment feature vectors and each of the initial cluster centers, assign each candidate sample to the cluster containing the nearest cluster center, and update the cluster center of the corresponding cluster based on the features of all samples within each cluster;

[0030] If the change between the new cluster center and the corresponding initial cluster center of each cluster is less than a preset threshold, or the number of times the cluster center is updated reaches the maximum number of iterations, the iteration terminates and the cluster number of the associated pattern to which each candidate sample belongs is obtained.

[0031] In one embodiment, in each round of greedy selection iteration, the modality weight coefficients are dynamically adjusted based on the current diversity loss of each modality, and the candidate samples are traversed to calculate the loss increment of the joint diversity objective function after each candidate sample is added to the selected sample set, including:

[0032] Read the text diversity loss, image diversity loss, and alignment diversity loss of the currently selected sample set, and dynamically adjust the text modality weight coefficient, image modality weight coefficient, and alignment modality weight coefficient based on the read modality diversity loss.

[0033] Iterate through all candidate samples that are not included in the selected sample set. For each candidate sample, simulate temporarily adding it to the current selected sample set to obtain a temporary selected sample set. Calculate the text diversity loss value, image diversity loss value, and alignment diversity loss value of the temporary selected sample set respectively.

[0034] By using the dynamically adjusted text modality weight coefficients, image modality weight coefficients, and alignment modality weight coefficients, the text diversity loss value, image diversity loss value, and alignment diversity loss value of the temporary selected sample set are weighted and summed to obtain the temporary joint diversity objective function;

[0035] Subtracting the joint diversity objective function from the temporary joint diversity objective function yields the loss increment corresponding to the candidate sample.

[0036] In one embodiment, determining whether to add a target candidate sample to the selected sample set, based on the distribution balance of the associated pattern clusters to which the unselected candidate samples belong in the selected sample set, includes:

[0037] For each candidate sample, query the corresponding association pattern cluster number and obtain the current number of selected samples in the selected sample set for the association pattern cluster;

[0038] Calculate the ratio of the total number of currently selected sample sets to the total number of associated pattern clusters to obtain the average number of samples that should be allocated to each cluster, and compare the current number of selected samples of the associated pattern cluster with the average number of samples.

[0039] If the number of currently selected samples exceeds the average number of samples, a preset coverage bias penalty is applied to the candidate samples belonging to the association pattern cluster, and the loss increment is combined with the coverage bias penalty to calculate the comprehensive score of each candidate sample.

[0040] The minimum guaranteed number of associated pattern clusters is calculated based on the total number of currently selected sample sets and the total number of associated pattern clusters. Each associated pattern cluster is checked to see if the current number of selected samples is lower than the minimum guaranteed number. If so, the candidate sample with the smallest loss increment is selected from the associated pattern clusters with the smaller number of samples lower than the minimum guaranteed number as the target sample for this round. If not, the candidate sample with the smallest comprehensive score is selected from all the candidate samples that have not yet been selected as the target sample for this round.

[0041] Add the selected target sample to the selected sample set, and update the modal feature matrix of each modality in the selected sample set and the selected sample count of each associated pattern cluster.

[0042] In one embodiment, when the total number of samples in the selected sample set reaches a preset data budget, the iteration stops, resulting in a multimodal pre-training target dataset that satisfies preset diversity constraints, including:

[0043] After adding the target sample to the selected sample set in each iteration, the total number of samples in the selected sample set is updated, and the greedy selection iteration stops when the total number of samples reaches a preset data budget threshold.

[0044] Extract all text-image paired samples from the selected sample set after the iteration stops to generate a multimodal pre-training target dataset.

[0045] A joint diversity data filtering system for pre-training multimodal large models, the system comprising:

[0046] A multimodal feature extraction module is used to acquire a candidate dataset including various text-image pairing samples. The text-image pairing samples are encoded using a text encoder and a visual encoder respectively to obtain text feature vectors and image feature vectors. The text feature vectors and image feature vectors are interactively fused through a gated cross-modal fusion layer to generate cross-modal alignment feature vectors. The text feature vectors, image feature vectors, and cross-modal alignment feature vectors are then standardized to obtain standardized text feature vectors, standardized image feature vectors, and standardized alignment feature vectors, respectively.

[0047] The joint diversity optimization module is used to calculate the text diversity loss, image diversity loss, and alignment diversity loss based on the standardized text feature vector, standardized image feature vector, and standardized alignment feature vector of the selected sample set, respectively. The module uses learnable modality weight coefficients to perform a weighted sum of the various diversity losses to construct the joint diversity objective function.

[0048] The association pattern clustering module is used to cluster and divide the samples according to the standardized aligned feature vectors of all samples, obtain each text-image semantic association pattern cluster, and record the association pattern cluster number to which each sample belongs.

[0049] The cross-modal balance constraint module is used to dynamically adjust the modal weight coefficients according to the current diversity loss of each modality in each round of greedy selection iteration, and to traverse the candidate samples, calculate the loss increment of the joint diversity objective function after each candidate sample is added to the selected sample set, and determine the target candidate sample to be added to the selected sample set by combining the distribution balance of the associated mode cluster to which the unselected candidate sample belongs in the selected sample set.

[0050] The filtering module is used to stop the iteration when the total number of samples in the selected sample set reaches the preset data budget, so as to obtain a multimodal pre-training target dataset that meets the preset diversity constraints.

[0051] The aforementioned method and system for joint diversity data filtering in multimodal large-scale model pre-training extracts single-modal features through dual text and visual encoders, and adaptively fuses them to generate alignment features using a gated cross-modal fusion layer. This enables the simultaneous acquisition of three scale-uniform feature representations: text semantics, visual content, and image-text matching relationships, addressing the problem that single features cannot measure cross-modal association differences. It calculates the diversity losses for text, image, and alignment separately and constructs a joint diversity objective function. Furthermore, in each iteration, it dynamically and adaptively adjusts the modal weights based on the diversity losses of each modality, automatically balancing the text and image modalities. The weighting of cross-modal alignment dimension in the sample selection process is determined; based on global standardized alignment features, image-text semantic association pattern clusters are pre-clustered, and candidate sample selection is constrained by the distribution balance of samples in each cluster during the greedy sample selection process, which can prioritize the selection of samples that can reduce the overall joint diversity loss; a greedy iterative strategy is adopted to output a target dataset that simultaneously satisfies the triple diversity constraints of text, image, and cross-modal alignment, reducing the computing power required for pre-training massive amounts of original image and text data, and exhibiting better generalization ability and recognition accuracy in downstream cross-modal tasks such as image description generation, visual question answering, and image and text retrieval. Attached Figure Description

[0052] Figure 1 This is a diagram illustrating the application environment of a joint diversity data filtering method for multimodal large model pre-training in one embodiment.

[0053] Figure 2 This is a schematic diagram of a joint diversity data screening method for multimodal large model pre-training in one embodiment;

[0054] Figure 3 This is a block diagram of a joint diversity data filtering system for multimodal large model pre-training in one embodiment;

[0055] Figure 4 This is a schematic diagram of the overall structure of a joint diversity data filtering system (MMD system) for multimodal large model pre-training in one embodiment;

[0056] Figure 5 This is a schematic diagram of the internal structure of the joint diversity optimization module in one embodiment;

[0057] Figure 6 This is a flowchart illustrating a multimodal pre-training data filtering scenario applied in a large-scale image and text retrieval platform, as shown in one embodiment.

[0058] Figure 7 This is a flowchart illustrating a small-sample multimodal data selection scenario applied in the field of medical imaging, as shown in one embodiment.

[0059] Figure 8 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0060] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0061] The joint diversity data filtering method for multimodal large model pre-training provided in this application can be applied to, for example... Figure 1 The application environment shown. For example... Figure 1 As shown, the application environment includes computer device 110. Computer device 110 can acquire a candidate dataset including various text-image pairing samples, and encode the text-image pairing samples using a text encoder and a visual encoder respectively to obtain text feature vectors and image feature vectors; interactively fuse the text feature vectors and image feature vectors through a gated cross-modal fusion layer to generate cross-modal alignment feature vectors; standardize the text feature vectors, image feature vectors, and cross-modal alignment feature vectors respectively to obtain standardized text feature vectors, standardized image feature vectors, and standardized alignment feature vectors; computer device 110 can calculate text diversity loss, image diversity loss, and alignment diversity loss respectively based on the standardized text feature vectors, standardized image feature vectors, and standardized alignment feature vectors of the selected sample set, and use learnable modal weight coefficients to adjust the loss of each diversity vector. The loss is weighted and summed to construct a joint diversity objective function. The computer device 110 can cluster the standardized aligned feature vectors of all samples to obtain various text-image semantic association pattern clusters, and record the cluster number to which each sample belongs. In each round of greedy selection iteration, the computer device 110 can dynamically adjust the modality weight coefficients based on the current diversity loss of each modality, traverse candidate samples, calculate the loss increment of the joint diversity objective function after each candidate sample is added to the selected sample set, and determine the target candidate sample to be added to the selected sample set by combining the distribution balance of the association pattern clusters to which the unselected candidate samples belong in the selected sample set. When the total number of samples in the selected sample set reaches the preset data budget, the computer device 110 can stop iterating, obtaining a multimodal pre-training target dataset that satisfies the preset diversity constraints. The computer device 110 can be, but is not limited to, various personal computers, laptops, smartphones, robots, unmanned aerial vehicles, tablets, etc.

[0062] In one embodiment, a joint diversity data filtering method (MMD) for multimodal large model pre-training is provided. MMD simultaneously optimizes text modal diversity, image modal diversity, and cross-modal alignment diversity within a unified filtering framework, thereby selecting a multimodal training subset with both high information content and high diversity under a limited data budget.

[0063] In one embodiment, such as Figure 2 As shown, a joint diversity data filtering method for pre-training a multimodal large model is provided, including the following steps:

[0064] Step 202: Obtain a candidate dataset including various text-image pairing samples, and encode the text-image pairing samples using a text encoder and a visual encoder respectively to obtain text feature vectors and image feature vectors; interactively fuse the text feature vectors and image feature vectors through a gated cross-modal fusion layer to generate cross-modal alignment feature vectors; standardize the text feature vectors, image feature vectors, and cross-modal alignment feature vectors respectively to obtain standardized text feature vectors, standardized image feature vectors, and standardized alignment feature vectors.

[0065] Computer devices can extract three complementary feature representations from raw text-image pairing data: text features, image features, and cross-modal alignment features. These three types of features reflect the distribution characteristics of the data in the text semantic space, visual semantic space, and cross-modal alignment space, respectively, providing a basis for subsequent diversity measurements.

[0066] In one embodiment, the joint diversity data screening method for multimodal large model pre-training may further include the process of obtaining text feature vectors and image feature vectors. Specifically, this process includes: acquiring a candidate dataset containing various text-image pairing samples; performing word segmentation on the text sequences in the candidate dataset to generate word sequences; inputting the word sequences into a pre-trained text encoder; mapping each word sequence to a continuous vector through the word embedding layer of the text encoder; obtaining each word representation after multi-layer self-attention transformation; aggregating the word representations using an average pooling strategy to obtain a fixed-dimensional text feature vector; dividing each image in the candidate dataset into non-overlapping image patch sequences; obtaining image patch embeddings after linear projection; concatenating classification words at the beginning of the sequences; and inputting them together into a pre-trained visual encoder; performing multi-layer self-attention transformation through the visual encoder; combining and aggregating the final output of the classification words and the final output of all image patches; and obtaining a fixed-dimensional image feature vector after layer normalization.

[0067] Computer devices can acquire text sequences from candidate datasets, perform word segmentation on the text sequences to generate corresponding word sequences, and input the generated word sequences into a pre-trained text encoder for processing. The text encoder first maps each word to a continuous low-dimensional vector through an internal word embedding layer, then feeds it into a multi-layer self-attention module to complete contextual semantic transformation, outputting the contextual representation corresponding to each word; finally, an average pooling strategy is used to aggregate the representation vectors of all valid words, filtering out numerical interference from padding words, and finally outputting a text feature vector with fixed dimensions.

[0068] The computer device can extract the original images paired with text from the candidate dataset, uniformly divide each complete image into several non-overlapping image patches, and form an image patch sequence. A linear projection operation is performed on the image patch sequence to convert the image pixel information into image patch embedding vectors. Independent classification word embeddings are concatenated at the beginning of the image patch embedding sequence, and the concatenated complete sequence is input into a pre-trained visual encoder. The visual encoder models the internal visual associations of the image through multi-layer self-attention transformation. After the transformation, the output representations of the classification words are extracted and aggregated with the output representations of all image patches. Then, layer normalization is applied to eliminate numerical offsets, and finally, a fixed-dimensional image feature vector is output.

[0069] Specifically, in this embodiment, the computer device can use a pre-trained text encoder to perform independent feature embedding on the text data. For the input text sequence... The text encoder first maps each word to a continuous vector through a word embedding layer, and then outputs a contextualized representation of each word after multiple layers of self-attention transformation. To obtain a fixed-dimensional sentence-level representation, the computer device can use an average pooling strategy to aggregate the word representations. (1); where, Represents the text feature vector. The length of the text sequence. For the first The output representation of each word after passing through the encoder For attention mask indicators, This is an indicator function that takes a value of 1 when the condition is true and 0 otherwise; this average pooling method eliminates the influence of filler words by averaging the effective words.

[0070] For image data, computer devices can use pre-trained visual encoders to perform independent feature embedding. Given an input image The visual encoder first divides the image into... A sequence of non-overlapping image patches is linearly projected and then fed into a visual transformer: (2); where, Represents the image feature vector. For the initial embedding of classification terms, For classifying lexical elements The final output features after the layer transformer Representation layer normalization operation; feature dimensions of text encoder and visual encoder and They can be different, each determined by its own model architecture.

[0071] To ensure the comparability and stability of features, computer equipment can standardize the extracted modal features: (3); among which, This represents the text or image feature vector before standardization. This represents the standardized corresponding feature vector. and These are the mean vector and variance vector of the features in the current batch, respectively. To prevent the numerical stability constant from being divided by zero; the standardized features have zero mean and unit variance, which facilitates subsequent diversity calculations.

[0072] In one embodiment, the joint diversity data filtering method for multimodal large model pre-training may further include a process of generating cross-modal aligned feature vectors. Specifically, this process includes: concatenating text feature vectors and image feature vectors along their feature dimensions to obtain a concatenated joint vector; inputting the joint vector into a gating network, performing linear mapping and a nonlinear transformation using a sigmoid activation function to calculate a gating vector; mapping the text feature vectors and image feature vectors to the aligned feature space using text projection matrices and image projection matrices respectively to obtain projected text feature vectors and projected image feature vectors; and using the gating vector to perform element-wise weighted fusion of the projected text feature vectors and projected image feature vectors to generate cross-modal aligned feature vectors.

[0073] The computer device can concatenate the encoded text feature vector and image feature vector along the feature dimension, resulting in a joint vector that fuses complete information from both unimodal features. This joint vector carries both textual semantic information and image visual information. Next, the computer device feeds the concatenated joint vector into a gated network. First, a linear mapping operation is used to transform the dimension of the joint vector, followed by a non-linear transformation using the Sigmoid activation function, ultimately outputting a gated vector. The value of each dimension in the gated vector is between 0 and 1, adaptively controlling the fusion ratio of text and image features in the corresponding dimension. The computer device can configure independent text projection matrices and image projection matrices. The text projection matrix is ​​used to perform a linear transformation on the original text feature vector, and the image projection matrix is ​​used to perform a linear transformation on the original image feature vector, mapping the two different unimodal features to the same aligned feature space, resulting in projected text feature vectors and projected image feature vectors, respectively.

[0074] Next, the computer device can perform element-wise weighted fusion operations on the projected text feature vector and the projected image feature vector based on the gating vector obtained above. By adaptively balancing the contribution of the two projected features by relying on the weight values ​​of each dimension of the gating vector, the final output is a cross-modal aligned feature vector that integrates bidirectional modal information and can characterize the deep semantic correspondence between the text and the image.

[0075] Specifically, after obtaining independent text and image features, computer devices can generate aligned feature representations through a cross-modal fusion layer to characterize the semantic alignment relationship between text-image pairs. Specifically, a gated fusion mechanism can be used to interactively fuse text and image features: (4); among which, For the gated vector, It is the Sigmoid activation function. For the gated weight matrix, For bias terms, Represents vector concatenation operation; gated vector Each element controls the fusion ratio of text features and image features in that dimension. The final representation of the aligned features is obtained through gated weighting: (5); among which, For cross-modal aligned feature vectors, This represents element-wise multiplication. and These are the projection matrices for text and image features, respectively; this gated fusion mechanism enables the alignment features to adaptively balance the contributions of text and image information, capturing the deep semantic correspondence between them. That is, each text-image sample pair... It can be mapped to three feature vectors , and This is used for diversity measurement and sample selection in subsequent modules.

[0076] Step 204: Based on the standardized text feature vector, standardized image feature vector, and standardized alignment feature vector of the selected sample set, calculate the text diversity loss, image diversity loss, and alignment diversity loss respectively. Use learnable modal weight coefficients to weight and sum the various diversity losses to construct the joint diversity objective function.

[0077] Computer devices can construct a unified objective function to simultaneously optimize data diversity across three dimensions: text modality, image modality, and cross-modal alignment. Specifically, the diversity measurement method based on the uniformity of covariance matrix eigenvalues ​​in single-modal scenarios can be extended to multimodal joint optimization scenarios, and learnable modality weight coefficients can be introduced to achieve adaptive balancing.

[0078] In one embodiment, the joint diversity data screening method for multimodal large model pre-training may further include a process of constructing an objective function. The specific process includes: obtaining the currently selected sample set; extracting the standardized text feature vector, standardized image feature vector, and standardized alignment feature vector for each sample in the selected sample set; arranging each feature vector by row to construct a text feature matrix, an image feature matrix, and an alignment feature matrix; calculating the feature covariance matrices corresponding to the text feature matrix, image feature matrix, and alignment feature matrix; extracting all off-diagonal elements from each feature covariance matrix; summing the squared off-diagonal elements; using the summation result as the diversity loss value to obtain the text diversity loss, image diversity loss, and alignment diversity loss; configuring modal weight coefficients for the corresponding text, image, and alignment dimensions; and performing a weighted summation of each modal weight coefficient with the corresponding text diversity loss, image diversity loss, and alignment diversity loss to obtain the joint diversity objective function.

[0079] The computer device can read the selected sample set in the current iteration round and sequentially extract the standardized text feature vector, standardized image feature vector, and standardized alignment feature vector corresponding to each sample in the set. It then arranges the feature vectors of the same type for all samples sequentially according to row dimensions and assembles them into text feature matrices, image feature matrices, and alignment feature matrices. These three feature matrices completely carry the standardized feature distribution information of the currently selected samples across the three modalities. Next, the computer device can first solve for the feature covariance matrix of the current modality based on the feature matrix corresponding to that modality. The covariance matrix can intuitively reflect the correlation between various feature dimensions under that modality. It extracts the values ​​at all off-diagonal positions within the covariance matrix, squares each off-diagonal value, and sums them up. The summed value is then used as the diversity loss value for that modality. The text diversity loss, image diversity loss, and alignment diversity loss are calculated in this way. Among them, the magnitude of the diversity loss can intuitively reflect the degree of dimensional collapse within the modality. The larger the loss value, the stronger the correlation between modal feature dimensions, the more concentrated the sample distribution, and the worse the diversity performance. The smaller the loss value, the more the feature dimensions are decorrelated with each other and the sample space is more uniformly distributed.

[0080] The computer device can pre-set three sets of modal weight coefficients corresponding to the text dimension, image dimension, and alignment dimension, respectively. Each weight coefficient has a value greater than zero, and the sum of the three sets of weight coefficients remains constant. The computer device can multiply the text diversity loss by the text modal weight coefficient, the image diversity loss by the image modal weight coefficient, and the alignment diversity loss by the alignment modal weight coefficient to obtain three weighted loss terms. These three weighted loss terms are then summed and fused to construct a joint diversity objective function that comprehensively measures the overall diversity level of the three modalities.

[0081] For each mode Computer devices can use the sum of squares of the off-diagonal elements of the covariance matrix of selected sample features as a dimensionality collapse metric for that mode. Given a set of selected samples... In modality The characteristic matrix on Its covariance matrix is ​​calculated as follows: (6); among which, For modality The feature covariance matrix, where the features have been pre-normalized to have a mean of zero. This represents the number of selected samples. When the covariance matrix is ​​an identity matrix, there is complete decorrelation between the feature dimensions, indicating that the data is uniformly distributed in all directions in the feature space, and there is no dimensionality collapse.

[0082] Modal The dimension collapse metric is defined as the sum of squares of the off-diagonal elements of the covariance matrix: (7); among which, For modality The smaller the value, the closer the feature dimensions within the modality are to decorrelation, and the higher the data diversity; this measure is equivalent to minimizing the Frobenius norm of the covariance matrix deviating from the identity matrix.

[0083] For cross-modal alignment dimensions, computer devices can also calculate alignment diversity measures based on the covariance matrix of alignment features: (8); among which, The covariance matrix of the alignment features. Measure the degree of feature decorrelation of selected samples in the cross-modal alignment space.

[0084] Next, the computer device can combine the diversity measures of the above three modalities into a joint optimization objective function: (9); among which, , , The weight coefficients for the text modality, image modality, and alignment dimension are respectively, satisfying... and Joint optimization objective function Minimizing this means that the selected subset simultaneously tends towards a state of high diversity in all three dimensions.

[0085] Step 206: Cluster the samples according to the standardized alignment feature vectors of all samples to obtain each text-image semantic association pattern cluster, and record the association pattern cluster number to which each sample belongs.

[0086] In one embodiment, the joint diversity data screening method for multimodal large model pre-training may further include a clustering process, specifically including: obtaining the standardized alignment feature vectors of all candidate samples; arranging the standardized alignment feature vectors by rows to construct an alignment feature matrix; setting the number of clusters to divide all candidate samples into associated pattern clusters; using a distributed initialization strategy to select the number of sample features from the standardized alignment feature vectors as initial cluster centers; calculating the distance between each standardized alignment feature vector and each initial cluster center; assigning each candidate sample to the cluster containing the nearest cluster center; and updating the cluster center of the corresponding cluster based on all sample features within each cluster; if the change between the new cluster center and the corresponding initial cluster center is less than a preset threshold, or the number of times the cluster center is updated reaches the maximum number of iterations, the iteration terminates, and the associated pattern cluster number to which each candidate sample belongs is obtained.

[0087] The computer device can traverse all candidate samples, obtain the standardized alignment feature vector corresponding to each candidate sample, and arrange all standardized alignment feature vectors sequentially according to row dimension to uniformly construct a global alignment feature matrix. The computer device can pre-set the number of clusters, which is the total number of text-image semantic association pattern clusters to be divided; using a distributed initialization strategy, sample features matching the number of clusters are selected from all global standardized alignment feature vectors as initial cluster centers. Next, the computer device can calculate the feature distance between the standardized alignment feature vector of each candidate sample and each initial cluster center, and assign each candidate sample to the association pattern cluster corresponding to the nearest cluster center according to the nearest distance principle; after completing the cluster assignment of all samples in this round, based on the standardized alignment feature vectors of all samples contained in each association pattern cluster, the cluster centers of the corresponding clusters are recalculated and updated to achieve iterative optimization of cluster centers.

[0088] After each update of the cluster centers, the computer calculates the numerical change between the new cluster centers and the corresponding initial cluster centers from the previous round, and compares this change with a preset threshold. Simultaneously, it counts the cumulative number of updates to the cluster centers. When the change in the cluster centers of each cluster is less than the preset threshold, or when the number of cluster iterations reaches the system's preset maximum number of iterations, the clustering operation is considered converged, and the clustering iteration process terminates. After clustering is complete, multiple independent text-image semantic association pattern clusters corresponding to different text-image matching relationships are obtained. Each association pattern cluster is assigned a unique number, which is the association pattern cluster number to which each candidate sample belongs.

[0089] Specifically, computer equipment can first use alignment features to cluster association patterns in candidate samples. Given a set of alignment features for all candidate samples... ,use -The mean clustering algorithm divides it into 1 Clusters of related patterns Each cluster represents a typical text-image semantic association pattern. The clustering objective is: (10); among which, For the first Cluster centers of a cluster of related patterns This is the preset number of association patterns. After clustering, each candidate sample is assigned to a cluster of association patterns.

[0090] To measure the coverage balance of the selected sample set across various association patterns, an association pattern coverage metric can be defined: (11); among which, For the selected set belonging to the first The number of samples in each associated pattern cluster. This represents the number of samples that should be allocated to each cluster under ideal equilibrium conditions. The range of values ​​is The closer the value is to 1, the more balanced the coverage of association patterns.

[0091] Step 208: In each round of greedy selection iteration, the modality weight coefficient is dynamically adjusted according to the current diversity loss of each modality, and the candidate samples are traversed. The loss increment of the joint diversity objective function after each candidate sample is added to the selected sample set is calculated. The distribution balance of the associated mode cluster to which the unselected candidate sample belongs in the selected sample set is combined to determine the target candidate sample to be added to the selected sample set.

[0092] To address the issue of inconsistent diversity scales across different modalities leading to a single modality dominating the selection process, the computer device employs an adaptive weight adjustment mechanism. During the greedy selection process of joint diversity optimization, explicit constraints are introduced to ensure that the selected sample set is diverse not only within each modality but also covers differentiated text-image semantic association patterns. Optimizing only the decorrelation degree within each modality cannot guarantee the diversity of cross-modal association types; additional constraints are needed to prevent excessive concentration of any particular association pattern.

[0093] In one embodiment, the provided method for joint diversity data screening in multimodal large model pre-training may further include a process of dynamically adjusting weights and calculating loss increments. Specifically, this process includes: reading the text diversity loss, image diversity loss, and alignment diversity loss of the currently selected sample set, and dynamically adjusting the text modality weight coefficients, image modality weight coefficients, and alignment modality weight coefficients based on the read modality diversity losses; traversing all candidate samples not included in the selected sample set, and for each candidate sample, simulating temporary addition to the currently selected sample set to obtain a temporary selected sample set, and calculating the text diversity loss value, image diversity loss value, and alignment diversity loss value of the temporary selected sample set respectively; using the dynamically adjusted text modality weight coefficients, image modality weight coefficients, and alignment modality weight coefficients, performing a weighted summation of the text diversity loss value, image diversity loss value, and alignment diversity loss value of the temporary selected sample set to obtain a temporary joint diversity objective function; and subtracting the joint diversity objective function from the temporary joint diversity objective function to obtain the loss increment corresponding to the candidate sample.

[0094] The computer device can read the text diversity loss, image diversity loss, and alignment diversity loss calculated in real time from the currently selected sample set, and dynamically update the corresponding modality weight coefficients based on the current degree of diversity deficiency in each modality. The computer device can traverse all candidate samples that are not currently included in the selected sample set. For each candidate sample, the candidate sample is temporarily incorporated into the current selected sample set to construct a temporary selected sample set. For the temporary selected sample set, the corresponding text feature matrix, image feature matrix, and alignment feature matrix are reconstructed, and the text diversity loss value, image diversity loss value, and alignment diversity loss value corresponding to the temporary selected sample set are calculated according to the preset diversity loss calculation rules, so as to truly restore the multimodal diversity changes of the overall sample set after the addition of the candidate sample.

[0095] Next, the computer device can use the text modality weight coefficients, image modality weight coefficients, and alignment modality weight coefficients, which have been dynamically adjusted in this iteration, to perform weighted calculations on the three types of diversity loss values ​​corresponding to the temporary selected sample set, and then sum them up to obtain the temporary joint diversity objective function corresponding to the temporary sample set. The computer device can then perform a difference calculation between the temporary joint diversity objective function and the joint diversity objective function of the original selected sample set before the iteration update to obtain the loss increment corresponding to the current candidate sample. Among them, the loss increment can intuitively represent the degree of contribution of a single candidate sample to the optimization of the overall multimodal joint diversity. The smaller the loss increment, the more significant the improvement effect of that sample on the overall dataset diversity.

[0096] In each iteration of the greedy selection, the computer device can dynamically adjust the weights based on the relative magnitude of the current diversity loss values ​​for each modality: (12); among which, To smooth the constant and prevent the denominator from being zero, this mechanism gives smaller weights to modes with larger current diversity loss values. Its core principle is to achieve "scale normalization" of each mode's loss term, that is, to multiply the diversity term of each mode by the reciprocal of its own magnitude to become a relative rate of change, thereby avoiding the excessive dominance of modes with larger absolute magnitudes in the selection process and achieving adaptive balance.

[0097] In the greedy selection process, a batch-level iterative strategy can be adopted. In each selection batch, starting from the batch containing... Selected sequentially from the superbatch of candidate samples The sample; in the first sample During round selection, iterate through all unselected candidate samples. Calculate the increment of the joint objective after adding it to the selected set: (13); Select the candidate sample that minimizes the increment and add it to the selected set. Through multiple rounds of greedy iteration, finally select a subset of samples that meet the budget constraint from the full data. The time complexity of the greedy strategy is O(n). ,in It represents the maximum value of the three modal feature dimensions.

[0098] In one embodiment, the joint diversity data screening method for multimodal large model pre-training may further include a process of updating the selected sample set. Specifically, this process includes: for each candidate sample, querying the corresponding associated pattern cluster number and obtaining the current number of selected samples for the associated pattern cluster in the selected sample set; calculating the ratio of the current total number of selected samples to the total number of associated pattern clusters to obtain the average number of samples that should be allocated to each cluster, and comparing the current number of selected samples for the associated pattern cluster with the average number of samples; if the current number of selected samples exceeds the average number of samples, applying a preset coverage bias penalty to the candidate samples belonging to the associated pattern cluster and incurring a loss... The incremental and coverage bias penalties are combined to calculate the comprehensive score of each candidate sample. The minimum guaranteed number of associated pattern clusters is calculated based on the total number of currently selected sample sets and the total number of associated pattern clusters. The current number of selected samples of each associated pattern cluster is checked one by one to see if it is lower than the minimum guaranteed number. If it is, the candidate sample with the smallest loss increment is selected from the associated pattern clusters with the lower guaranteed number as the target sample for this round. If not, the candidate sample with the smallest comprehensive score is selected from all the candidate samples that have not yet been selected as the target sample for this round. The selected target sample is added to the selected sample set, and the modal feature matrices of each modality in the selected sample set and the selected sample count of each associated pattern cluster are updated.

[0099] For each unselected candidate sample, the computer device can retrieve its pre-bound associated pattern cluster number. Based on this cluster number, it can query the number of samples selected for the corresponding semantic association pattern cluster in the currently selected sample set, obtaining the real-time sample distribution of each association pattern cluster, providing data for balance determination. The computer device can calculate the ratio between the total number of samples in the current selected sample set and the preset total number of association pattern clusters to obtain the theoretically average number of samples that a single association pattern cluster should be allocated. The current actual number of selected samples for each association pattern cluster is compared with this average number of samples to determine if there is an over-enrichment problem in the current cluster sample distribution. If the current number of selected samples for a certain association pattern cluster exceeds the average number of samples, it indicates that the semantic association pattern of that image and text has an excessively high proportion in the selected sample set, indicating a sample enrichment problem. A preset coverage bias penalty needs to be applied to all candidate samples belonging to that association pattern cluster. The computer device can fuse the loss increment corresponding to each candidate sample with the coverage bias penalty to obtain a comprehensive score for each candidate sample, using the penalty mechanism to suppress the repeated selection of high-frequency association pattern samples.

[0100] In this embodiment, the minimum guaranteed number of samples for each associated pattern cluster can be calculated in advance based on the total number of selected sample sets and the total number of associated pattern clusters to ensure sample coverage for niche associated patterns. If the current number of selected samples for an associated pattern cluster is lower than the minimum guaranteed number, the candidate sample with the smallest loss increment is selected first from that cluster as the target sample for this round to make up for the sample shortage of niche associated patterns. If the number of selected samples for all associated pattern clusters meets the minimum guaranteed requirement, the candidate sample with the smallest comprehensive score is directly selected from all unselected candidate samples as the target sample for this round. After determining the target candidate sample for this round, the target sample is formally included in the selected sample set, and the text feature matrix, image feature matrix, and alignment feature matrix corresponding to the selected sample set are updated simultaneously. At the same time, the selected sample count for each associated pattern cluster is updated, completing the state update of this round of greedy iteration.

[0101] Specifically, in each iteration of the greedy selection, the computer device can simultaneously consider the joint diversity increment and the associated pattern coverage constraint when evaluating candidate samples. Specifically, candidate samples... The overall score is: (14); among which, To add samples The subsequent increase in joint diversity loss, For the sample The cluster number of the associated pattern, This represents the number of samples in the selected set that belong to the same association pattern cluster. To constrain the penalty coefficient, This is a coverage deviation threshold. When the proportion of a certain association pattern cluster in the selected set exceeds the equilibrium proportion... When the probability of selection is multiplied by 1, a penalty is imposed on candidate samples belonging to that cluster, reducing their probability of being selected.

[0102] This embodiment also includes a cluster-level minimum guarantee mechanism to ensure that each associated pattern cluster has at least one... One sample was selected: (15); among which, To ensure the minimum required quantity, it is usually set to When the number of selected samples for a certain association pattern cluster is lower than the minimum guarantee, the constrained greedy algorithm prioritizes selecting the sample from the cluster that minimizes the increase in joint diversity loss until the minimum guarantee requirement is met.

[0103] In this embodiment, the overall process of the constrained greedy selection algorithm is as follows: First, the alignment features of candidate samples are clustered to obtain association pattern clusters; then, a greedy iteration is entered, and in each round, the sample with the best comprehensive score is selected from the candidate set and added to the selected set; during the selection process, the selected count of each cluster is dynamically updated, and the score is adjusted according to the constraints; the iteration is repeated until the selected set reaches the preset selection budget. The constrained greedy selection algorithm ensures balanced coverage of the selected samples across modal association patterns while guaranteeing joint diversity optimization.

[0104] Step 210: When the total number of samples in the selected sample set reaches the preset data budget, stop the iteration and obtain the multimodal pre-training target dataset that satisfies the preset diversity constraints.

[0105] In one embodiment, the provided method for joint diversity data screening in multimodal large model pre-training may further include a process for obtaining the final target dataset. The specific process includes: after adding the target sample to the selected sample set in each iteration, updating the total number of samples in the selected sample set, and stopping the greedy selection iteration when the total number of samples reaches a preset data budget threshold; extracting all text-image paired samples from the selected sample set after the iteration stops to generate the multimodal pre-training target dataset.

[0106] After each round of greedy iteration to select target samples and add them to the selected sample set, the computer device can update the current total number of samples in the selected sample set in real time and continuously compare the updated total number of samples with the preset data budget threshold to verify and monitor whether the iteration termination condition has been met. If the total number of samples in the current selected sample set has not reached the preset data budget threshold, the greedy iteration process of dynamic adjustment of modality weights, loss increment calculation, and cluster balance constraint selection is continued to optimize and expand the selected sample set. When it is detected that the total number of samples in the selected sample set has reached the preset data budget threshold, the computer device can immediately terminate all greedy selection iteration operations and no longer perform subsequent candidate sample selection and sample set update operations. After the iteration terminates, the computer device can extract all text-image pairing samples in the current selected sample set. These samples have undergone multiple rounds of multimodal diversity optimization and semantic association pattern balance constraint selection, and simultaneously meet multiple constraints such as text modality diversity, image modality diversity, cross-modal alignment diversity, and balanced distribution of image-text semantic association patterns. Finally, they are organized to generate a target dataset that can be directly used for pre-training of multimodal large models.

[0107] In one embodiment, a joint diversity data filtering method for multimodal large model pre-training, namely the MMD method, is provided. The complete interactive process unfolds in three stages: feature extraction, cluster preprocessing, and iterative selection. In the feature extraction stage, the original data source simultaneously sends batch data to both the text encoder and the visual encoder; the text encoder performs multi-layer self-attention encoding and average pooling on the text sequence, outputting text feature vectors. The visual encoder divides the image into image blocks and performs transformer encoding, outputting image feature vectors. Two types of features are simultaneously input into a gated fusion layer, and adaptively fused into an aligned feature vector using a Sigmoid gate vector. Subsequently, the three types of vectors—text features, image features, and alignment features—are uniformly fed into the feature standardization module for zero-mean unit variance normalization.

[0108] During the clustering preprocessing stage, standardized alignment features are fed into the association pattern clustering module, which performs... - The mean clustering algorithm divides the alignment features of all candidate samples into... Clusters of related patterns Each cluster represents a typical text-image semantic association pattern; after clustering, the cluster affiliation information and initial coverage value of each sample are passed to the constrained greedy selector.

[0109] During the iterative selection phase, the joint diversity optimization module receives three types of standardized feature matrices and calculates the sum of squares of the off-diagonal elements of the covariance matrix for each modality as the diversity loss; the current diversity loss value for each modality... , , It is sent to the adaptive weight adjustment module, which adjusts the weights according to a ratio inversely proportional to the loss value (i.e., ... The updated dynamic weight coefficients are calculated and returned; the joint diversity optimization module uses the weighted joint objective to calculate the loss increment for each candidate sample. This information is then passed to the constrained greedy selector; the selector simultaneously receives cluster affiliation information from the clustering module and uses a comprehensive scoring function. Each candidate sample is evaluated, and the sample with the best comprehensive score is selected. The selection result updates the feature matrix of the selected set and feeds it back to the joint diversity optimization module to recalculate the covariance matrix, while triggering the recalculation of weights. This iterative loop continues until the constrained greedy selector determines that the selected set has reached the preset data budget. Finally, the joint diversity optimization module outputs a joint diversity evaluation report based on the complete selected set, which includes the final diversity loss value of each modality and the correlation mode coverage balance index.

[0110] It should be understood that although the steps in the flowchart above are shown sequentially as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowchart above may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0111] In one embodiment, such as Figure 3 As shown, a joint diversity data filtering system for pre-trained multimodal large models is provided, including: a multimodal feature extraction module 310, a joint diversity optimization module 320, an association pattern clustering module 330, a cross-modal balance constraint module 340, and a filtering module 350, wherein:

[0112] The multimodal feature extraction module 310 is used to acquire a candidate dataset including various text-image pairing samples, and to encode the text-image pairing samples using a text encoder and a visual encoder respectively to obtain text feature vectors and image feature vectors; the text feature vectors and image feature vectors are interactively fused through a gated cross-modal fusion layer to generate cross-modal alignment feature vectors; the text feature vectors, image feature vectors, and cross-modal alignment feature vectors are standardized respectively to obtain standardized text feature vectors, standardized image feature vectors, and standardized alignment feature vectors.

[0113] The joint diversity optimization module 320 is used to calculate the text diversity loss, image diversity loss, and alignment diversity loss based on the standardized text feature vector, standardized image feature vector, and standardized alignment feature vector of the selected sample set, respectively. The learnable modality weight coefficients are used to perform a weighted summation of each diversity loss to construct the joint diversity objective function.

[0114] The association pattern clustering module 330 is used to perform clustering based on the standardized aligned feature vectors of all samples to obtain each text-image semantic association pattern cluster, and record the association pattern cluster number to which each sample belongs.

[0115] The cross-modal balance constraint module 340 is used to dynamically adjust the modal weight coefficients according to the current diversity loss of each modality in each round of greedy selection iteration, and to traverse the candidate samples, calculate the loss increment of the joint diversity objective function after each candidate sample is added to the selected sample set, and determine the target candidate sample to be added to the selected sample set by combining the distribution balance of the associated mode cluster to which the unselected candidate sample belongs in the selected sample set.

[0116] The filtering module 350 is used to stop the iteration when the total number of samples in the selected sample set reaches the preset data budget, so as to obtain a multimodal pre-trained target dataset that meets the preset diversity constraints.

[0117] In one embodiment, the multimodal feature extraction module 310 is further configured to acquire a candidate dataset including various text-image pairing samples, perform word segmentation on the text sequences in the candidate dataset to generate word sequences, and input the word sequences into a pre-trained text encoder; map each word sequence into a continuous vector through the word embedding layer of the text encoder, obtain each word representation after multi-layer self-attention transformation, and aggregate the word representations through an average pooling strategy to obtain a fixed-dimensional text feature vector; divide each image in the candidate dataset into a sequence of non-overlapping image patches, obtain image patch embeddings after linear projection, and concatenate classification words at the front end of the sequence before inputting them into a pre-trained visual encoder; perform multi-layer self-attention transformation through the visual encoder, combine and aggregate the final output of the classification words and the final output of all image patches, and obtain a fixed-dimensional image feature vector after layer normalization processing.

[0118] In one embodiment, the multimodal feature extraction module 310 is further configured to concatenate the text feature vector and the image feature vector along the feature dimension to obtain a concatenated joint vector; input the joint vector into a gating network, and calculate the gating vector through linear mapping and nonlinear transformation by the Sigmoid activation function; map the text feature vector and the image feature vector to the aligned feature space through the text projection matrix and the image projection matrix respectively to obtain the projected text feature vector and the projected image feature vector; and use the gating vector to perform element-wise weighted fusion of the projected text feature vector and the projected image feature vector to generate a cross-modal aligned feature vector.

[0119] In one embodiment, the joint diversity optimization module 320 is further configured to obtain the currently selected sample set, extract the standardized text feature vector, standardized image feature vector, and standardized alignment feature vector of each sample in the selected sample set, arrange each feature vector in rows to construct the text feature matrix, image feature matrix, and alignment feature matrix; calculate the feature covariance matrix corresponding to the text feature matrix, image feature matrix, and alignment feature matrix respectively, extract all off-diagonal elements in each feature covariance matrix, square the off-diagonal elements and sum them, and use the summation result as the diversity loss value to obtain the text diversity loss, image diversity loss, and alignment diversity loss; configure the modal weight coefficients corresponding to the text, image, and alignment dimensions respectively, and perform weighted summation of each modal weight coefficient with the corresponding text diversity loss, image diversity loss, and alignment diversity loss to obtain the joint diversity objective function.

[0120] In one embodiment, the association pattern clustering module 330 is further configured to obtain the standardized alignment feature vectors of all candidate samples, arrange the standardized alignment feature vectors in rows to construct an alignment feature matrix; set the number of clusters to divide all candidate samples into association pattern clusters, and select the number of sample features from the standardized alignment feature vectors as initial cluster centers using a distributed initialization strategy; calculate the distance between each standardized alignment feature vector and each initial cluster center, assign each candidate sample to the cluster containing the nearest cluster center, and update the cluster center of the corresponding cluster based on all sample features within each cluster; if the change between the new cluster center and the corresponding initial cluster center of each cluster is less than a preset threshold, or the number of times the cluster center is updated reaches the maximum number of iterations, the iteration terminates, and the association pattern cluster number to which each candidate sample belongs is obtained.

[0121] In one embodiment, the cross-modal balance constraint module 340 is further configured to read the text diversity loss, image diversity loss, and alignment diversity loss of the currently selected sample set, and dynamically adjust the text modality weight coefficient, image modality weight coefficient, and alignment modality weight coefficient based on the read modality diversity loss; traverse all candidate samples not included in the selected sample set, and for each candidate sample, simulate temporarily adding it to the current selected sample set to obtain a temporary selected sample set, and calculate the text diversity loss value, image diversity loss value, and alignment diversity loss value of the temporary selected sample set respectively; use the dynamically adjusted text modality weight coefficient, image modality weight coefficient, and alignment modality weight coefficient to perform a weighted summation of the text diversity loss value, image diversity loss value, and alignment diversity loss value of the temporary selected sample set to obtain a temporary joint diversity objective function; subtract the joint diversity objective function from the temporary joint diversity objective function to obtain the loss increment corresponding to the candidate sample.

[0122] In one embodiment, the cross-modal balance constraint module 340 is further configured to, for each candidate sample, query the corresponding associated mode cluster number and obtain the current number of selected samples of the associated mode cluster in the selected sample set; calculate the ratio of the total number of currently selected samples to the total number of associated mode clusters to obtain the average number of samples to be allocated to each cluster, and compare the current number of selected samples of the associated mode cluster with the average number of samples; if the current number of selected samples exceeds the average number of samples, apply a preset coverage deviation penalty to the candidate samples belonging to the associated mode cluster, and combine the loss increment with the coverage deviation penalty to calculate the loss increment for each candidate sample. The sample is comprehensively scored; the minimum guaranteed number of associated pattern clusters is calculated based on the total number of currently selected sample sets and the total number of associated pattern clusters. Each associated pattern cluster is checked to see if its current number of selected samples is lower than the minimum guaranteed number. If so, the candidate sample with the smallest loss increment is selected from the associated pattern clusters with the lower number of samples as the target sample for this round. If not, the candidate sample with the smallest comprehensive score is selected from all the candidate samples that have not yet been selected as the target sample for this round. The selected target sample is added to the selected sample set, and the modal feature matrices of each associated pattern cluster and the selected sample count of each associated pattern cluster are updated.

[0123] In one embodiment, the filtering module 350 is further configured to update the total number of samples in the selected sample set after adding the target sample to the selected sample set in each iteration, and stop the greedy selection iteration when the total number of samples reaches a preset data budget threshold; and extract all text-image paired samples in the selected sample set after the iteration stops to generate a multimodal pre-trained target dataset.

[0124] In another embodiment, a joint diversity data filtering system for multimodal large model pre-training, namely the MMD system, is provided, and its overall structure is as follows: Figure 4 As shown, the structure revolves around three main lines: multimodal feature extraction, joint diversity optimization, and constrained greedy selection. Figure 4 As shown, the original large-scale text-image pairing dataset (such as LAION or CC3M) is simultaneously input into the text encoder and visual encoder along two paths, respectively, and each independently completes the feature embedding of its respective modality. The text encoder performs multi-layer self-attention transformation on the input text sequence and outputs the text feature vector through average pooling. The visual encoder divides the input image into a sequence of image patches and outputs image feature vectors. Features from both the text encoder and the visual encoder are simultaneously fed into a cross-modal gating fusion layer. This layer adaptively controls the fusion ratio of text features and image features across various dimensions using a sigmoid gating vector, generating aligned feature vectors. The three types of features (text features, image features, and alignment features) are uniformly fed into the feature standardization module and subjected to zero-mean unit variance standardization to eliminate scale differences between modalities.

[0125] The standardized three types of features are input into the joint diversity optimization module, which calculates the sum of squares of the off-diagonal elements of the feature covariance matrix for each modality as the diversity loss value. , and and with modal weighting coefficients , , Weighted summation yields the joint diversity target The association pattern clusterer pre-processes all candidate samples using alignment features. -means clustering divides the samples into... Each sample is assigned a cluster of associated patterns and its cluster affiliation information is labeled. In each iteration, the constrained greedy selector receives the joint diversity loss increment from the joint diversity optimization module and cluster coverage information from the associated pattern clusterer. It then selects the optimal sample and adds it to the selected set using a comprehensive scoring function. Updates to the selected set are fed back to the joint diversity optimization module to recalculate the covariance matrix and diversity loss, and also back to the constrained greedy selector to update the selected count of each cluster. An adaptive weight adjuster dynamically adjusts the weight coefficients based on the reciprocal of the current diversity loss value for each modality. When the loss value of a particular modality is large, its weight is reduced, achieving an adaptive balance of multimodal diversity. The system iterates continuously through a closed-loop feedback loop: joint diversity optimization module → constrained greedy selector → selected set → joint diversity optimization module, until the selected set reaches the preset data budget.

[0126] In one embodiment, the internal structure of the joint diversity optimization module is as follows: Figure 5 As shown, the Joint Diversity Optimization module is the core computational engine of the MMD system, responsible for evaluating the contribution of candidate samples to the joint diversity objective in each greedy iteration. It receives three types of standardized feature inputs (text feature matrix, text feature matrix, and text feature matrix). Image feature matrix and alignment feature matrix The text features are fed into three independent covariance calculation units. The text covariance calculation unit calculates the covariance matrix from the text feature matrix. Then, the off-diagonal elements are extracted, and the sum of squares is calculated to obtain the text diversity loss. The image covariance calculation unit and the alignment covariance calculation unit each perform the same calculation process and output... and .

[0127] The diversity loss values ​​from the three modalities are fed into the modality loss aggregator, which also receives dynamic weight coefficients from the adaptive weight vector. , , ,according to Calculate the weighted joint loss. The candidate incremental evaluator evaluates each candidate sample that has not yet been selected. The simulation shows the new covariance matrix after adding it to the selected set, and the increment of the joint diversity loss is calculated. The minimum increment selector selects the sample from all candidate samples that makes the minimum increment selector the sample from all candidate samples. The smallest sample index is submitted to the selected set updater. After the updater officially adds the sample to the selected set, it feeds back the updated covariance matrix to the three covariance calculation units: the text covariance calculation unit, the image covariance calculation unit, and the alignment covariance calculation unit, for use in the next iteration. This feedback mechanism ensures that each round of selection decisions is based on the latest state of the selected set, avoiding suboptimal selections due to lag in covariance matrix information.

[0128] In one embodiment, subsets of CC3M and LAION-2B are used as candidate data sources, and a 1.5% subset is selected from the full dataset for pre-training of multimodal contrastive learning. To verify the technical advantages of MMD over existing data selection methods, comparative implementations are conducted on three downstream tasks: image caption generation (COCO Captions), visual question answering (VQAv2), and image-text retrieval (Flickr30k). Table 1 shows the performance differences between MMD and the comparative implementations; Table 2 shows the parameter configuration of the multimodal feature extraction module; Table 3 shows the parameter configuration of joint diversity optimization and balance constraints.

[0129] Table 1. Performance comparison of different data selection methods on downstream tasks (selection rate 1.5%)

[0130]

[0131] Table 2 Parameter Configuration of Multimodal Feature Extraction Module

[0132]

[0133] Table 3 Parameter Configuration for Joint Diversity Optimization and Balance Constraints

[0134]

[0135] As shown in Table 1, MMD outperforms all the comparative implementation methods in the main evaluation metrics. On the COCO Captions task, BLEU-4 achieves 30.6% accuracy, and CIDEr achieves 97.3%, representing improvements of 0.8% and 1.6% respectively compared to JEST. On the VQAv2 and Flickr30k tasks, MMD achieves 64.2% and 56.3% accuracy respectively, leading all the comparative methods.

[0136] This application provides a joint diversity data screening method and system for pre-training multimodal large models. By jointly optimizing diversity, it simultaneously minimizes the sum of squares of the off-diagonal elements of the covariance matrix across three dimensions: text modality, image modality, and cross-modal alignment, thus promoting a more uniform distribution of the selected subset across feature spaces. This mechanism directly addresses the root cause of dimensionality collapse, fundamentally improving the feature utilization of the selected data. This application designs an adaptive weight adjustment mechanism based on the reciprocal of the diversity loss value for each modality. When the diversity loss value of a certain modality is large, its weight is automatically reduced to prevent that modality from excessively influencing the overall selection decision. This mechanism solves the problem of inconsistent diversity scales between text and image modalities due to differences in feature dimensions and semantic granularity, ensuring that each modality is treated evenly in joint optimization. As shown in Table 3, this embodiment configures the initial weights as follows: , , The contributions of the three factors are dynamically balanced during the iteration process through an adaptive adjustment mechanism.

[0137] This application ensures that the selected sample set covers differentiated text-image semantic association types by aligning association pattern clustering and coverage constraints in the feature space. The introduction of a cluster-level minimum guarantee mechanism and constraint penalty coefficients effectively prevents a certain type of association pattern from excessively occupying selection slots. The joint optimization framework of this application can still effectively maintain selection quality even in scenarios with extremely limited data budgets. This effect stems from the fact that the joint diversity objective can maximize the feature coverage of each modality in a very small number of samples, making the limited training data more efficient. This application uses a gated fusion mechanism to generate aligned feature representations. The gate vector adaptively controls the fusion ratio of text and image features in each dimension through the Sigmoid activation function. Compared with simple feature concatenation or linear weighting, gated fusion can capture the deep semantic correspondence between text and images, generating more discriminative aligned representations and providing a more refined feature space for diversity measurement and association pattern clustering.

[0138] The one-time diversity screening scheme for static datasets provided in this application can efficiently screen subsets with both information content and diversity from massive multimodal data before pre-training begins. As shown in Table 1, using only 1.5% of the screened data, MMD achieves a CIDEr of 97.3% on COCO Captions, exceeding JEST's 95.7% using the same proportion of data. This demonstrates that the present invention can achieve better pre-training results under the same computational budget, effectively reducing the amount of data and training overhead required to achieve the target performance. The multimodal feature extraction module of this application has no hard dependency on encoder selection. The text encoder and visual encoder can be replaced with lightweight distillation models or pre-trained models with different architectures depending on the computing power. As shown in Table 2, this embodiment uses Contriever and ViT-B / 16 as encoders, both with a feature dimension of 768. However, this configuration can be flexibly replaced with lightweight models such as DistilBERT and MobileViT according to actual resource conditions without affecting the overall selection framework's operating logic. This application establishes for the first time a unified selection framework that simultaneously optimizes text modality diversity, image modality diversity, and cross-modal alignment diversity, bridging the technical gap between single-modal data selection methods and the requirements of multimodal pre-training. As shown in Table 1, MMD achieves superior performance compared to comparative implementations in three downstream tasks: image description generation, visual question answering, and image-text retrieval. Compared to Text-Only DiSF and Image-Only DiSF, which only target a single modality, MMD achieves improvements of 1.5% and 1.9% (VQAv2), respectively, validating the technical effectiveness of the unified multimodal diversity framework.

[0139] In one embodiment, the joint diversity data screening method and system for multimodal large-scale model pre-training provided in this application can be applied to the multimodal pre-training data screening scenario of a large-scale image and text retrieval platform. Specifically, the platform has over 2 billion text-image pairing data (such as the LAION-2B subset) crawled from the Internet, and needs to select 1.5% of high-quality subsets for multimodal contrastive learning pre-training to support image and text retrieval services for global users. The platform is deployed on a cluster server equipped with 8 GPUs. The challenge lies in ensuring that the selected subset possesses sufficient diversity in three dimensions—textual semantics, visual content, and cross-modal alignment—with an extremely low data budget, avoiding a decrease in retrieval recall due to dimensional collapse. The main process is as follows: Figure 6 As shown:

[0140] M1: In the data loading and preprocessing stage, 2 billion candidate image-text pairing data are loaded in batches of 512. The text is segmented and truncated into word sequences. The images are normalized in size and divided into image blocks. Standardized batch data is output for subsequent encoder consumption.

[0141] M2: Parallel feature extraction stage with dual encoders; the text stream outputs a dimension of [missing information]. Text feature vectors The image stream output dimension after passing through the ViT-B / 16 encoder is Image feature vectors Two types of features are simultaneously input into the cross-modal gating fusion layer.

[0142] M3: In the cross-modal aligned feature generation stage, the gated fusion layer calculates the gate vector using the Sigmoid activation function. Adaptively controls the fusion ratio of text features and image features in each dimension, with an output dimension of Alignment feature vectors All three features were standardized with zero mean and unit variance.

[0143] M4: In the association pattern clustering and coverage initialization stage, standardized alignment features are used to perform clustering on all candidate samples. -means clustering, divided into There are 10 clusters of association patterns, each representing a typical text-image semantic association pattern (such as "simple description - clear image", "abstract description - artistic image", etc.). An initial coverage index is calculated. .

[0144] M5: In the joint diversity loss calculation stage, the covariance matrix is ​​calculated for the three types of standardized features of the selected set. , , Extracting the sum of squares of off-diagonal elements yields the diversity loss for the three modes. , , Based on the initial weight configuration in Table 3 , , Weighted summation is the joint optimization objective function .

[0145] M6: Adaptive weight dynamic adjustment stage. The weight coefficients are recalculated based on the inverse proportion of the current modality diversity loss values. When the loss value of a certain modality is large, its weight is automatically reduced to avoid dominating the selection process, based on the smoothing constants in Table 3. To prevent numerical overflow, output the updated dynamic weight vector.

[0146] M7: Constrained greedy selection phase, calculate the joint diversity loss increment for each candidate sample in the superbatch. The comprehensive score is calculated by combining the coverage constraints of the associated pattern clusters. According to the penalty coefficients in Table 3 and coverage deviation threshold A penalty is imposed on candidates for over-concentrated clusters, and the sample with the best comprehensive score is selected in each round and added to the selected set.

[0147] M8: Iterative convergence and output stage. After the selected set is updated, it is fed back to M5 to recalculate the covariance matrix and diversity loss, and then fed back to M7 to update the selected count of each cluster. M5 to M8 are repeated until the number of selected samples reaches the selection budget (1.5% of the total data). Finally, the selected sample index list and joint diversity assessment report are output.

[0148] Based on the comparative data in Table 1, with a selection ratio of 1.5%, the MMD implementation in this example achieves an R@1 of 56.3% on the Flickr30k image-text retrieval task, a 3.9% improvement compared to Random Sampling's 52.4%, a 2.5% improvement compared to Text-Only DiSF's 53.8%, and a 1.2% improvement compared to JEST's 55.1%. This indicates that the selected subset can simultaneously maintain the discriminative power of both textual and visual semantics. Regarding computational efficiency, a batch-level greedy strategy is adopted to control the computational complexity of each selection round to within a certain range. The entire process of filtering 2 billion data points on an 8-GPU cluster can be completed within 48 hours.

[0149] In one embodiment, the joint diversity data screening method and system for multimodal large-scale model pre-training provided in this application can be applied to a small-sample multimodal data selection scenario in the field of medical imaging. Specifically, a medical institution has approximately 500,000 paired medical images and diagnostic reports, and needs to screen a high-quality subset from these for fine-tuning a general multimodal pre-trained model to adapt it to clinical tasks such as medical image description generation and pathological question answering. Due to limitations in medical data privacy compliance requirements and annotation costs, the data budget available for training is only 0.5% of the total. Furthermore, medical images have highly specialized visual features and strict terminology specifications, resulting in a significant bias in the feature representation capabilities of general text encoders and visual encoders in this domain. Adaptive weight adjustment and cross-modal balance constraints are needed to bridge the capability differences between modalities. The main process is as follows: Figure 7 As shown:

[0150] M1: The medical image and text data loading and domain adaptation preprocessing stage loads 500,000 medical image-diagnostic report paired data in batches of 512. The diagnostic reports are processed by medical entity word segmentation and terminology standardization. The medical images are processed by grayscale normalization and lesion area cropping and enhancement. The output is the domain-adapted batch data, i.e., the batch medical image and text data, which is the data after preprocessing image and text pairing.

[0151] M2: The domain feature extraction stage of the dual encoder, where the diagnostic report flows through the Contriever encoder to output text feature vectors. Medical images flow through the ViT-B / 16 encoder to output image feature vectors. Due to the difference in distribution between medical images and general pre-training data, the feature variance of the image modality is significantly greater than that of the text modality. Both types of features are simultaneously input into the cross-modal gating fusion layer.

[0152] M3: Cross-modal aligned feature fusion stage, the gated fusion layer output dimension is... Alignment feature vectors In the medical field, gating mechanisms tend to assign higher fusion weights to image features to compensate for the semantic sparsity of text terms, and the three features are standardized separately.

[0153] M4: Medical association pattern clustering stage, performing clustering on 500,000 candidate samples using standardized alignment features. -means clustering, divided into A cluster of medical association patterns, such as typical medical image-text association types like "normal image - negative report", "clear lesion - specific diagnosis", and "fuzzy image - vague description", are used to calculate the initial coverage.

[0154] M5: In the joint diversity loss and modal skew detection stage, the diversity loss of the three modalities is calculated separately. , , In the medical field, the diversity loss of image modalities is usually significantly lower than that of text modalities (because the visual features of medical images are more concentrated). The joint diversity framework can effectively detect and quantify the degree of skew between modalities.

[0155] M6: Adaptive weight skew compensation stage. Based on the adaptive adjustment mechanism in Table 3, when the text modality loss value... Much larger than the image modal loss value Automatically reduce text weight. And increase image weight To prevent text modality from dominating the selection process, ensure that the visual diversity requirements of medical images are not ignored, and output a compensated dynamic weight vector.

[0156] M7: Constrained Greedy Choice and Medical-Related Coverage Phase. A comprehensive score is calculated for candidate samples, based on the minimum coverage at the cluster level in Table 3. Ensure that each of the 20 medical association pattern clusters has at least a minimum number of samples selected, avoiding excessive occupation of selection slots by high-frequency patterns such as "normal imaging - negative report", and ensuring coverage of rare case association patterns.

[0157] M8: Iterative convergence and clinical validation output stage. After the selected set is updated, it is fed back to M5 and M7. M5 to M8 are repeated until the number of selected samples reaches 0.5% of the selection budget. Finally, the selected sample index list is output, and an evaluation report containing the coverage of each medical association pattern cluster is generated for clinical experts to review.

[0158] Based on the data in Table 1, with a selection ratio of 1.5%, MMD achieved a VQAv2 accuracy of 64.2%, a 2.5% improvement compared to RandomSampling's 61.7%, validating that this embodiment can effectively improve downstream multimodal understanding even with a limited data budget. In medical imaging scenarios, the adaptive weight skew compensation mechanism increased the contribution of image modality diversity by approximately 18%, effectively compensating for the intermodal capability imbalance caused by the sparsity of medical text terminology.

[0159] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 8 As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with external terminals via a network connection. When executed by the processor, the computer program implements a joint diversity data filtering method for multimodal large model pre-training. The display screen can be an LCD screen or an e-ink screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad mounted on the computer device casing, or an external keyboard, touchpad, or mouse.

[0160] Those skilled in the art will understand that Figure 8 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0161] In one embodiment, a computer device is provided, including a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement steps of a joint diversity data filtering method for pre-training a multimodal large model.

[0162] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program being executed by a processor to implement steps of a joint diversity data filtering method for pre-training a multimodal large model.

[0163] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0164] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0165] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A method for joint diversity data screening during pre-training of a multimodal large model, characterized in that, The method includes: A candidate dataset including various text-image pairing samples is obtained. The text-image pairing samples are encoded using a text encoder and a visual encoder respectively to obtain text feature vectors and image feature vectors. The text feature vectors and image feature vectors are interactively fused through a gated cross-modal fusion layer to generate cross-modal alignment feature vectors. The text feature vectors, image feature vectors, and cross-modal alignment feature vectors are standardized to obtain standardized text feature vectors, standardized image feature vectors, and standardized alignment feature vectors. Based on the standardized text feature vector, standardized image feature vector, and standardized alignment feature vector of the selected sample set, the text diversity loss, image diversity loss, and alignment diversity loss are calculated respectively. The learnable modality weight coefficients are used to weight and sum the various diversity losses to construct the joint diversity objective function. Clustering is performed on the standardized aligned feature vectors of all samples to obtain each text-image semantic association pattern cluster, and the cluster number of the association pattern to which each sample belongs is recorded. In each round of greedy selection iteration, the modality weight coefficients are dynamically adjusted according to the current diversity loss of each modality, and the candidate samples are traversed. The loss increment of the joint diversity objective function after each candidate sample is added to the selected sample set is calculated. The target candidate sample is added to the selected sample set by combining the distribution balance of the associated mode cluster to which the unselected candidate sample belongs in the selected sample set. When the total number of samples in the selected sample set reaches the preset data budget, the iteration stops, and a multimodal pre-trained target dataset that meets the preset diversity constraints is obtained.

2. The joint diversity data screening method for multimodal large model pre-training according to claim 1, characterized in that, A candidate dataset including various text-image pairing samples is obtained. The text-image pairing samples are then encoded using both a text encoder and a visual encoder to obtain text feature vectors and image feature vectors, including: Obtain a candidate dataset including various text-image pairing samples, perform word segmentation on the text sequences in the candidate dataset to generate word sequences, and input the word sequences into a pre-trained text encoder; The word embedding layer of the text encoder maps each word sequence into a continuous vector. After multi-layer self-attention transformation, each word representation is obtained. The word representations are then aggregated using an average pooling strategy to obtain a fixed-dimensional text feature vector. Each image in the candidate dataset is divided into a sequence of non-overlapping image blocks. After linear projection, the image block embedding is obtained. Classification words are concatenated at the beginning of the sequence and then input into a pre-trained visual encoder. The visual encoder performs multi-layer self-attention transformation, combines and aggregates the final output of the classification terms and the final output of all image blocks, and obtains a fixed-dimensional image feature vector after layer normalization.

3. The joint diversity data screening method for multimodal large model pre-training according to claim 2, characterized in that, The text feature vector and image feature vector are interactively fused through a gated cross-modal fusion layer to generate a cross-modal aligned feature vector, including: The text feature vector and the image feature vector are concatenated along the feature dimension to obtain the concatenated joint vector; The joint vector is input into the gated network, and the gated vector is calculated through linear mapping and nonlinear transformation by the Sigmoid activation function. The text feature vector and the image feature vector are mapped to the aligned feature space using the text projection matrix and the image projection matrix, respectively, to obtain the projected text feature vector and the projected image feature vector. The gated vector is used to perform element-wise weighted fusion of the projected text feature vector and the projected image feature vector to generate a cross-modal aligned feature vector.

4. The joint diversity data screening method for multimodal large model pre-training according to claim 1, characterized in that, Based on the standardized text feature vectors, standardized image feature vectors, and standardized alignment feature vectors of the selected sample set, text diversity loss, image diversity loss, and alignment diversity loss are calculated respectively. The joint diversity objective function is constructed by weighting and summing the individual diversity losses using learnable modality weight coefficients, including: Obtain the current selected sample set, extract the standardized text feature vector, standardized image feature vector and standardized alignment feature vector of each sample in the selected sample set, arrange each feature vector in rows, and construct the text feature matrix, image feature matrix and alignment feature matrix respectively; Calculate the feature covariance matrices corresponding to the text feature matrix, image feature matrix, and alignment feature matrix respectively, and extract all off-diagonal elements in each feature covariance matrix. Square the off-diagonal elements and sum them. Use the summation result as the diversity loss value to obtain the text diversity loss, image diversity loss, and alignment diversity loss. Modal weight coefficients are configured for the corresponding text, image, and alignment dimensions, respectively. The joint diversity objective function is obtained by weighting and summing each modal weight coefficient with the corresponding text diversity loss, image diversity loss, and alignment diversity loss.

5. The joint diversity data filtering method for multimodal large model pre-training according to claim 1, characterized in that, Clustering is performed on the standardized aligned feature vectors of all samples to obtain various text-image semantic association pattern clusters, and the cluster number to which each sample belongs is recorded, including: Obtain the standardized alignment feature vectors of all candidate samples, arrange the standardized alignment feature vectors by row, and construct the alignment feature matrix; The number of clusters to divide all candidate samples into clusters of associated patterns is set, and the number of sample features to be selected from the standardized aligned feature vector is selected as the initial cluster centers using a distributed initialization strategy. Calculate the distance between each of the standardized alignment feature vectors and each of the initial cluster centers, assign each candidate sample to the cluster containing the nearest cluster center, and update the cluster center of the corresponding cluster based on the features of all samples within each cluster; If the change between the new cluster center and the corresponding initial cluster center of each cluster is less than a preset threshold, or the number of times the cluster center is updated reaches the maximum number of iterations, the iteration terminates and the cluster number of the associated pattern to which each candidate sample belongs is obtained.

6. The joint diversity data screening method for multimodal large model pre-training according to claim 1, characterized in that, In each round of greedy selection iteration, the modality weight coefficients are dynamically adjusted based on the current diversity loss of each modality, and the candidate samples are traversed to calculate the loss increment of the joint diversity objective function after each candidate sample is added to the selected sample set, including: Read the text diversity loss, image diversity loss, and alignment diversity loss of the currently selected sample set, and dynamically adjust the text modality weight coefficient, image modality weight coefficient, and alignment modality weight coefficient based on the read modality diversity loss. Iterate through all candidate samples that are not included in the selected sample set. For each candidate sample, simulate temporarily adding it to the current selected sample set to obtain a temporary selected sample set. Calculate the text diversity loss value, image diversity loss value, and alignment diversity loss value of the temporary selected sample set respectively. By using the dynamically adjusted text modality weight coefficients, image modality weight coefficients, and alignment modality weight coefficients, the text diversity loss value, image diversity loss value, and alignment diversity loss value of the temporary selected sample set are weighted and summed to obtain the temporary joint diversity objective function; Subtracting the joint diversity objective function from the temporary joint diversity objective function yields the loss increment corresponding to the candidate sample.

7. The joint diversity data screening method for multimodal large model pre-training according to claim 6, characterized in that, Based on the distribution balance of the associated pattern clusters to which the unselected candidate samples belong in the selected sample set, the target candidate sample is determined to be added to the selected sample set, including: For each candidate sample, query the corresponding association pattern cluster number and obtain the current number of selected samples in the selected sample set for the association pattern cluster; Calculate the ratio of the total number of currently selected sample sets to the total number of associated pattern clusters to obtain the average number of samples that should be allocated to each cluster, and compare the current number of selected samples of the associated pattern cluster with the average number of samples. If the number of currently selected samples exceeds the average number of samples, a preset coverage bias penalty is applied to the candidate samples belonging to the association pattern cluster, and the loss increment is combined with the coverage bias penalty to calculate the comprehensive score of each candidate sample. The minimum guaranteed number of associated pattern clusters is calculated based on the total number of currently selected sample sets and the total number of associated pattern clusters. Each associated pattern cluster is checked to see if the current number of selected samples is lower than the minimum guaranteed number. If so, the candidate sample with the smallest loss increment is selected from the associated pattern clusters with the smaller number of samples lower than the minimum guaranteed number as the target sample for this round. If not, the candidate sample with the smallest comprehensive score is selected from all the candidate samples that have not yet been selected as the target sample for this round. Add the selected target sample to the selected sample set, and update the modal feature matrix of each modality in the selected sample set and the selected sample count of each associated pattern cluster.

8. The joint diversity data screening method for multimodal large model pre-training according to claim 7, characterized in that, When the total number of samples in the selected sample set reaches the preset data budget, the iteration stops, resulting in a multimodal pre-training target dataset that satisfies the preset diversity constraints, including: After adding the target sample to the selected sample set in each iteration, the total number of samples in the selected sample set is updated, and the greedy selection iteration stops when the total number of samples reaches a preset data budget threshold. Extract all text-image paired samples from the selected sample set after the iteration stops to generate a multimodal pre-training target dataset.

9. A joint diversity data filtering system for pre-training multimodal large models, characterized in that, The system includes: A multimodal feature extraction module is used to acquire a candidate dataset including various text-image pairing samples. The text-image pairing samples are encoded using a text encoder and a visual encoder respectively to obtain text feature vectors and image feature vectors. The text feature vectors and image feature vectors are interactively fused through a gated cross-modal fusion layer to generate cross-modal alignment feature vectors. The text feature vectors, image feature vectors, and cross-modal alignment feature vectors are then standardized to obtain standardized text feature vectors, standardized image feature vectors, and standardized alignment feature vectors, respectively. The joint diversity optimization module is used to calculate the text diversity loss, image diversity loss, and alignment diversity loss based on the standardized text feature vector, standardized image feature vector, and standardized alignment feature vector of the selected sample set, respectively. The module uses learnable modality weight coefficients to perform a weighted sum of the various diversity losses to construct the joint diversity objective function. The association pattern clustering module is used to cluster and divide the samples according to the standardized aligned feature vectors of all samples, obtain each text-image semantic association pattern cluster, and record the association pattern cluster number to which each sample belongs. The cross-modal balance constraint module is used to dynamically adjust the modal weight coefficients according to the current diversity loss of each modality in each round of greedy selection iteration, and to traverse the candidate samples, calculate the loss increment of the joint diversity objective function after each candidate sample is added to the selected sample set, and determine the target candidate sample to be added to the selected sample set by combining the distribution balance of the associated mode cluster to which the unselected candidate sample belongs in the selected sample set. The filtering module is used to stop the iteration when the total number of samples in the selected sample set reaches the preset data budget, so as to obtain a multimodal pre-trained target dataset that meets the preset diversity constraints.

Images