A large model pre-training data processing method for cross-modal semantic alignment

Abstract

The present application relates to the technical field of artificial intelligence, in particular to a large model pre-training data processing method for cross-modal semantic alignment, in the present application, the distribution of heterogeneous modalities in a unified space is mapped by quantization, and the semantic equivalent boundary is identified, effectively solving the alignment deviation caused by the representation deviation of modal attributes, using cross-attention projection path analysis to analyze the topological structure deformation, introducing local contrast loss to test the lower bound standard of mutual information, enhancing the structural tension and stability of the semantic joint representation architecture, dynamically updating the alignment loss scaling weight, optimizing the learning efficiency of difficult example samples in the pre-training process, strengthening the model's ability to distinguish hard positive and negative samples, reducing the information entropy loss in the feature compression process, injecting the constraint label tensor sequence into the data loader, strengthening the synchronization accuracy of the cross-modal control sequence, improving the quality of data throughput parallel processing, ensuring the global alignment consistency of the large model pre-training, and significantly enhancing the semantic consistency and task generalization performance of multi-modal representation.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a method for processing large model pre-training data with cross-modal semantic alignment. Background Technology

[0002] Traditional methods for pre-training data in large-scale cross-modal semantic alignment refer to the preprocessing of heterogeneous data sources to address their correlations before constructing a multimodal base model that integrates text, images, audio, or video. These methods employ hard association based on text keyword matching, or use pre-defined manual rules to select paired image-text data. Alternatively, they utilize existing object detectors to extract image region features and perform initial alignment with corresponding words in the text sequence using vector similarity calculations. Then, positional encoding and modal embedding vectors are used to concatenate discrete pixel blocks and character tokens in a unified coordinate space. Finally, static sampling strategies and fixed masking language modeling methods are used to clean and reassemble the training samples.

[0003] Traditional cross-modal pre-training data processing uses keyword matching or manually pre-defined rules for hard association, which leads to randomness in the identification of correlations between heterogeneous data sources and a lack of deep semantic connotation. It uses a pre-defined target detector to extract local features and simply concatenates them with the text sequence. However, due to the limitations of static sampling strategies and fixed masking patterns, the alignment accuracy between modalities depends entirely on the primary vector similarity. This makes it impossible to accurately capture the subtle and complex semantic shifts between different modalities in the feature space. As a result, key structural information is easily lost during the training sample cleaning process, leading to cross-modal association breaks or semantic distortion problems. Summary of the Invention

[0004] The purpose of this invention is to address the shortcomings of existing technologies by proposing a method for processing large model pre-training data with cross-modal semantic alignment.

[0005] To achieve the above objectives, the present invention adopts the following technical solution: a method for processing large model pre-training data with cross-modal semantic alignment, comprising the following steps: S1: Obtain the structured semantic features of cross-modal input pairing instances, quantify the mapping distribution coordinates of differentiated modalities in a unified semantic space, calculate the displacement difference of each modality relative to the absolute semantic origin, identify the core semantic equivalence boundary, and obtain the cross-modal semantic anchor benchmark dataset; S2: Based on the cross-modal semantic anchor benchmark dataset, identify the cross-attention projection path of heterogeneous modal data to semantic anchors, analyze the degree of topological deformation of adjacent projection levels, quantify the semantic distortion in the cross-modal bridging process, and obtain the cross-modal mapping fluctuation amplitude dataset. S3: Based on the cross-modal mapping fluctuation amplitude dataset, calculate the local contrast loss of cross-modal pairing instances and determine whether it is at the semantic equivalence tolerance boundary, check the lower bound of cross-modal joint representation mutual information, output hard sample mask instructions when the structural integrity condition is met, and generate a semantic alignment boundary constraint judgment matrix. S4: Based on the hard sample mask instruction state in the semantic alignment boundary constraint judgment matrix, extract the gradient contribution weight required for the corresponding cross-modal pairing, and dynamically update the scaling weight factor of the alignment loss by reconstructing the cross-entropy constraint conditions in the pre-training process to obtain the pre-training alignment loss weight modulation parameter set. S5: Based on the pre-trained alignment loss weight modulation parameter set, the updated weight factors are injected into the word segmentation sequence configuration of the current pre-training batch, and the tensor sequence including constraint tags is written into the data loader interface to obtain the large model pre-training cross-modal control sequence set.

[0006] As a further aspect of the present invention, the steps for obtaining the cross-modal semantic anchor benchmark dataset are as follows: S111: Based on the structured semantic features of cross-modal input pairing instances, extract the shared underlying semantic dimension in the multimodal alignment task, calculate the absolute spatial coordinates of each modal feature vector on the shared dimension, and obtain the basic semantic spatial coordinate parameters. S112: Based on the basic semantic space coordinate parameters, calculate the distribution distance of heterogeneous modalities with semantic correspondence when mapped to the same spatial coordinates, measure the semantic representation offset inherent in the differentiated modal attributes, and obtain the modality-specific semantic drift index. S113: Based on the modality-specific semantic drift index, the multimodal drift range is fused into the shared semantic coordinate system, defining the maximum tolerance boundary for differentiated modal data to be considered semantically equivalent in the feature space, and generating a cross-modal semantic anchor benchmark dataset.

[0007] As a further aspect of the present invention, the steps for obtaining the cross-modal mapping fluctuation amplitude dataset are as follows: S211: Based on the cross-modal semantic anchor benchmark dataset, identify the layer-by-layer cross-attention network that maps heterogeneous modal features to semantic anchor boundaries, record the distribution of attention weight matrix of each layer when the network performs projection approximation, and obtain the multi-layer attention projection weight sequence. S212: Based on the multi-layer attention projection weight sequence, extract the weight topology relationship between adjacent projection layers in the network, calculate the feature space distortion and topological distance difference that occur during the bridging transmission of cross-modal semantics, and form the cross-modal bridging trajectory distortion variable. S213: Based on the cross-modal bridging trajectory distortion variables, the distortion values ​​of the differentiated levels are mapped to the manifold space based on the tolerance boundaries of the semantic anchor points. The distortion severity level of the differentiated attention head is identified by comparing the upper and lower limits of the anchor points and tensor labels are attached to generate a cross-modal mapping fluctuation amplitude dataset.

[0008] As a further aspect of the present invention, the step of obtaining the semantic alignment boundary constraint determination matrix specifically includes: S311: Based on the cross-modal mapping fluctuation amplitude dataset, calculate the local contrastive learning loss for each cross-modal mapping instance, determine whether its local semantic representation has successfully fallen back to the tolerance boundary interval of semantic equivalence, record the boundary satisfaction state label, and generate a local contrastive edge satisfaction state set. S312: Based on the local comparison edge satisfying state set, calculate whether the cross-modal joint features satisfying the boundary reach the preset mutual information lower bound standard, evaluate the structural tension and stability of the semantic joint representation architecture, and generate a joint representation mutual information compliance record; S313: Based on the joint representation mutual information achievement record, extract the mutual information achievement status in sequence and cross-validate it with the global pre-training alignment rule of the large model. When the joint representation is determined to be completely closed, identify the hard positive and hard negative sample mask identifiers for the pairing, register the current tensor node in the mapping task stack, and generate the semantic alignment boundary constraint judgment matrix.

[0009] As a further aspect of the present invention, the step of obtaining the pre-trained alignment loss weight modulation parameter set specifically comprises: S411: Based on the hard positive and hard negative sample mask identifiers in the semantic alignment boundary constraint judgment matrix, extract the feature structures that need to be punished or rewarded in the alignment task, combine them with the corresponding semantic anchor point spatial location, and remove irrelevant semantics through tensor filtering to obtain the set of hard example comparison gradient contributions. S412: Based on the set of gradient contribution comparisons of difficult examples, combined with the alignment benchmark value and gradient scaling adjustment function corresponding to the current pre-training batch, calculate the alignment loss scaling factor required for the batch, reconstruct the preset threshold of cross-entropy constraints in the pre-training network, and obtain the reconstructed cross-entropy constraint scaling value. S413: Based on the reconstructed cross-entropy constraint ratio value, the ratio value is overlaid into the hyperparameter configuration item of the large model pre-training loss function, the version timestamp and architecture node number of the alignment pace are updated, and a pre-training alignment loss weight modulation parameter set is generated.

[0010] As a further aspect of the present invention, the steps for obtaining the large model pre-training cross-modal control sequence set are as follows: S511: Based on the pre-trained aligned loss weight modulation parameter set, the optimized loss weight and penalty term parameters are incorporated into the word segmentation array configuration structure of the current pre-training batch, and the multimodal attention mask matrix is ​​updated at the feature boundary to generate a multimodal mask aligned configuration structure. S512: Based on the multimodal mask alignment configuration structure, evaluate the semantic advancement pace of the current word segmentation array in the pre-trained model, bind the tensor-level pace rate with gradient constraint logic, identify the underlying sequence synchronization signal parameters according to the preset control beat, and obtain the semantic alignment pace sequence cache set; S513: Based on the semantic alignment pace sequence cache set, the synchronization signal and the multimodal word segmentation array content with attached constraint tags are sequentially written into the data loader interface of the large model in continuous tensor form to obtain the large model pre-training cross-modal control sequence set.

[0011] Compared with the prior art, the advantages and positive effects of the present invention are as follows: In this invention, by quantifying the mapping distribution of heterogeneous modalities in a unified space and identifying semantic equivalence boundaries, the alignment bias caused by the inherent representational shift of modal attributes is effectively solved. Cross-attention projection path analysis is used to analyze topological deformation, accurately quantify mapping fluctuations in the bridging process, and determine the degree of semantic distortion. Local contrastive loss is introduced to test the lower bound standard of mutual information, enhancing the structural tension and stability of the semantic joint representation architecture. Gradient contribution weights are extracted based on mask instructions, and cross-entropy constraints are reconstructed. Alignment loss scaling weights are dynamically updated to optimize the learning efficiency of difficult examples in the pre-training process, strengthen the model's ability to distinguish between hard positive and negative samples, reduce information entropy loss in the feature compression process, and inject constraint label tensor sequences into the data loader to enhance the synchronization accuracy of cross-modal control sequences, improve the quality of parallel data throughput processing, ensure global alignment consistency during large model pre-training, and significantly enhance the semantic consistency of multimodal representations and task generalization performance. Attached Figure Description

[0012] Figure 1 This is a schematic diagram of the main steps of the present invention. Detailed Implementation

[0013] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0014] Example 1 Please see Figure 1 This invention provides a technical solution: a method for processing large model pre-training data with cross-modal semantic alignment, comprising the following steps: S1: Obtain the structured semantic features of cross-modal input pairing instances, quantify the mapping distribution coordinates of differentiated modalities in a unified semantic space, calculate the displacement difference of each modality relative to the absolute semantic origin, identify the core semantic equivalence boundary, and obtain the cross-modal semantic anchor benchmark dataset; S2: Based on the cross-modal semantic anchor benchmark dataset, identify the cross-attention projection path of heterogeneous modal data to semantic anchors, analyze the degree of topological deformation of adjacent projection levels, quantify the semantic distortion in the cross-modal bridging process, and obtain the cross-modal mapping fluctuation amplitude dataset. S3: Based on the cross-modal mapping fluctuation amplitude dataset, calculate the local contrast loss of cross-modal pairing instances and verify whether they are within the tolerance boundary of semantic equivalence. Examine the lower bound of mutual information of cross-modal joint representation. When the structural integrity requirement is met, output the hard sample mask instruction and generate the semantic alignment boundary constraint decision matrix. S4: Based on the hard sample mask instruction state in the semantic alignment boundary constraint decision matrix, extract the gradient contribution weight required for the corresponding cross-modal pairing, and dynamically update the scaling weight factor of the alignment loss by reconstructing the cross-entropy constraint conditions in the pre-training process to obtain the pre-training alignment loss weight modulation parameter set. S5: Based on the pre-trained alignment loss weight modulation parameter set, the updated weight factors are injected into the word segmentation sequence configuration of the current pre-training batch, the attention mask matrix of the cross-modal boundary is updated, and the tensor sequence including constraint labels is written into the data loader interface to obtain the large model pre-training cross-modal control sequence set.

[0015] The specific steps for obtaining the cross-modal semantic anchor benchmark dataset are as follows: S111: Based on the structured semantic features of cross-modal input pairing instances, extract the shared underlying semantic dimension in the multimodal alignment task, calculate the absolute spatial coordinates of each modal feature vector on the shared dimension, and obtain the basic semantic spatial coordinate parameters. Structured semantic features of paired image and text modal input instances are obtained. A pre-trained multimodal feature extraction network retrieves both image visual feature vectors and text semantic feature vectors. The shared underlying semantic dimensions of the multimodal alignment task are analyzed to identify object categories, spatial relationships, and action attributes. A shared feature subspace, composed of 512 independent orthogonal feature axes, is extracted when the two modalities represent the same thing. Projection operations are performed on each pair of paired modal feature vectors, calculating their projection length on each shared dimension axis to determine their precise position in the multidimensional semantic coordinate system. For an instance containing an image of a runner and its corresponding text description, the extracted image feature vector has a coordinate value of 0.85 on the category dimension axis and 0.92 on the action activity dimension axis; the corresponding text feature vector has coordinate values ​​of 0.82 and 0.88 on the same dimension axis. The projection components in each dimension are integrated to determine the absolute spatial coordinates of the modal feature vectors in the shared dimension. The coordinate values ​​are normalized to be in the range of 0 to 1, yielding the basic semantic space coordinate parameters.

[0016] S112: Based on the basic semantic space coordinate parameters, calculate the distribution distance of heterogeneous modalities with semantic correspondence when mapped to the same spatial coordinates, measure the semantic representation offset inherent in the differentiated modal attributes, and obtain the modality-specific semantic drift index. Based on the fundamental semantic space coordinate parameters, for each image-text pair with semantic correspondence, the distribution distance in the 512-dimensional shared space is calculated. The distribution distance uses the Manhattan distance calculation logic, summing the absolute values ​​of the coordinate differences between the two modalities in each dimension. The image coordinate vector has a value of 0.85 in a specific dimension, and the text coordinate vector has a value of 0.82 in that dimension, with an absolute difference of 0.03. The absolute values ​​of the differences in all 512 dimensions are accumulated to obtain the overall distribution displacement value of the paired instances. This value measures the semantic representation shift caused by the acquisition device, encoding method, and modality-specific attributes. Quantization heterogeneity inherently introduces noise; therefore, the distribution distance is ratio-calculated to the internal feature variance of the modality. The internal feature variance of the image modality is 0.12, and the distribution displacement value is 0.36, corresponding to a ratio of 3. This ratio is then fed into a nonlinear mapping function trained on historical alignment data. The input is the displacement ratio, and the output is a continuous value between 0 and 1. Finally, a modality-specific semantic drift index is obtained.

[0017] S113: Based on the modality-specific semantic drift index, the multimodal drift range is fused into the shared semantic coordinate system, defining the maximum tolerance boundary for differentiated modal data to be considered semantically equivalent in the feature space, and generating a cross-modal semantic anchor benchmark dataset; Based on the modality-specific semantic drift index, spatial fusion is performed on the drift ranges of different modalities. The execution logic is to perform a weighted sum of the image modality drift (0.65) and the text modality drift (0.15), with the weights set according to the importance of each modality in the current training task, each accounting for 0.5, resulting in a comprehensive drift mean of 0.4. Using this mean as a benchmark, a hypersphere region centered on the basic semantic coordinates is constructed in a shared semantic coordinate system. The radius of the region is determined by multiplying the comprehensive drift mean by a tolerance coefficient. The tolerance coefficient is obtained through statistical analysis of the semantic consistency scores of the training set, and is set to the 95th percentile of the semantic consistency distribution, set to 1.25. The product of the comprehensive drift mean of 0.4 and the tolerance coefficient of 1.25 yields a value of 0.5, which defines the maximum tolerance boundary for semantic equivalence of differentiated modal data in the feature space. Heterogeneous feature vectors with a spatial distance less than or equal to 0.5 are considered semantically equivalent. By iterating through all pre-trained paired samples, the coordinate pairs that satisfy the boundary conditions, the drift index, and the boundary values ​​are stored in a structured manner to generate a cross-modal semantic anchor benchmark dataset.

[0018] The specific steps for obtaining the cross-modal mapping fluctuation amplitude dataset are as follows: S211: Based on a cross-modal semantic anchor benchmark dataset, identify a layer-by-layer cross-attention network that maps heterogeneous modal features to semantic anchor boundaries, record the distribution of attention weight matrices at each layer when the network performs projection approximation, and obtain the multi-layer attention projection weight sequence. Based on a cross-modal semantic anchor benchmark dataset, a layer-by-layer cross-attention network is used to identify the mapping of heterogeneous modal features to semantic anchor boundaries. The network adopts a standard 12-layer Transformer architecture, including a multi-head attention mechanism and a feedforward neural network layer. During the computation of each layer, the weight distribution during projection approximation is recorded. After extracting the query vector and key vector from each layer and performing a dot product operation, the attention score matrix is ​​obtained after scaling and activation by a normalization function. The matrix size is the number of attention heads multiplied by the sequence length, and the attention weight matrix is ​​extracted sequentially for each layer of the 12-layer network. In the first layer, the average attention weight of attention head 1 to the semantic anchor is 0.12; in the 12th layer, the average attention weight evolves to 0.85. The weight matrix set is arranged sequentially from the first layer to the 12th layer, and encapsulated according to temporal and logical hierarchy to obtain a multi-layer attention projection weight sequence.

[0019] S212: Based on the multi-layer attention projection weight sequence, extract the weight topology relationship between adjacent projection layers in the network, calculate the feature space distortion and topological distance difference that occur during the cross-modal semantics bridging transmission, and form the cross-modal bridging trajectory distortion variable; The weight topology relationship refers to calculating the Euclidean distance between two adjacent weight matrices in a multi-layer attention projection weight sequence, identifying the spatial evolution trajectory of the weight distribution during the projection process, and obtaining the weight topology relationship. Based on the multi-layer attention projection weight sequence, the topological structure relationship of weights in adjacent projection layers in the network is extracted, and attention weight matrices of two adjacent layers are selected, such as the matrices of layers 5 and 6. The Euclidean distance between the two layer weight matrices is calculated by subtracting the corresponding elements of the two matrices and squaring the result, summing the results for all positions, and then taking the square root. The weight matrix of layer 5 is 512 by 512, and the corresponding matrix elements of layer 6 are offset, resulting in an Euclidean distance of 2.45. This distance reflects the spatial evolution trajectory of the weight distribution during the projection process. The feature space distortion caused by cross-modal semantics during bridging transmission is calculated, and this distortion value is determined by the singular value decomposition result of the current layer weights. The top 10 largest singular values ​​of the weight matrix are extracted, and their energy proportions are calculated. The energy proportion of layer 5 is 0.88, which decreases to 0.75 in layer 6, resulting in a feature distortion of 0.13 units. The Euclidean distance of 2.45 is multiplied by the feature distortion value of 0.13, and the difference in the topological distance between the feature vectors of the two layers is subtracted. The topological distance difference is obtained by calculating the connectivity changes of the feature in the manifold space, and is set to 0.05. It is then used to form a cross-modal bridging trajectory distortion variable through logical operations.

[0020] S213: Based on cross-modal bridging trajectory distortion variables, the distortion values ​​of different levels are mapped to the manifold space based on the tolerance boundary of semantic anchor points. The distortion severity level of the differential attention head is identified by comparing the upper and lower limits of the anchor points and tensor labels are attached to generate a cross-modal mapping fluctuation amplitude dataset. Based on cross-modal bridging trajectory distortion, differentiated hierarchical distortion values ​​are mapped to the manifold space, and a tolerance boundary value of 0.5 is retrieved from the cross-modal semantic anchor benchmark dataset. The distortion value of 0.2685 generated at each layer is compared with the boundary value. A semantic shift is determined when the distortion value exceeds 50% of the boundary value (greater than 0.25). The distortion coefficient is obtained by calculating the ratio of the distortion value to the boundary value; the distortion value of layer 8 is 0.35, with a ratio of 0.7. This ratio is matched with a preset distortion level range: 0 to 0.3 indicates slight distortion, 0.3 to 0.6 indicates moderate distortion, and above 0.6 indicates severe distortion. The ratio of layer 8 is 0.7, and it is labeled as a severe distortion level. This determination process is repeated for each attention head in the network to identify the distortion severity level and attach tensor labels. Label 1 represents normal, label 2 represents moderate, and label 3 represents severe. All layers, all attention head distortion values, distortion level labels, and corresponding anchor boundary information are integrated to generate a cross-modal mapping fluctuation amplitude dataset.

[0021] The specific steps for obtaining the semantic alignment boundary constraint decision matrix are as follows: S311: Based on the cross-modal mapping fluctuation amplitude dataset, calculate the local contrastive learning loss for each cross-modal mapping instance, determine whether its local semantic representation has successfully fallen back to the tolerance boundary interval of semantic equivalence, record the boundary satisfaction state label, and generate a local contrastive edge satisfaction state set. Based on the cross-modal mapping fluctuation amplitude dataset, the local contrastive learning loss for cross-modal mapping instances is calculated. The coordinates of the current position of the mapping instance in the feature space are retrieved, and the corresponding semantic anchor center coordinates are matched from the anchor dataset. The cosine similarity between the two is calculated; when the similarity is 0.92, the negative log-likelihood loss is calculated. This loss value is fused with the corresponding bridging trajectory distortion variable; when the distortion variable is 0.2685, it is weighted and added to the loss function to determine whether the calculated local semantic representation has successfully fallen back to the semantic equivalence tolerance boundary interval. The judgment logic is as follows: calculate the Euclidean distance from the feature vector to the anchor center; a distance of 0.45 and a preset tolerance boundary of 0.50 indicate satisfaction, while a distance of 0.55 indicates non-satisfaction. A status label for each mapping instance boundary satisfaction is recorded: 1 for satisfaction and 0 for non-satisfaction. Processing is performed on the full batch data to generate a set of local contrastive edge satisfaction states.

[0022] S312: Based on the local comparison edge satisfying state set, calculate whether the cross-modal joint features satisfying the boundary reach the preset mutual information lower bound standard, evaluate the structural tension and stability of the semantic joint representation architecture, and generate joint representation mutual information compliance record; To evaluate the structural tension and stability of the semantic joint representation architecture, we calculate the eigenvalues ​​of the covariance matrix of the feature dimensions in the mutual information achievement records of the joint representation, and determine the information entropy retention rate of the semantic joint representation architecture in the feature compression process by analyzing the concentration of the eigenvalue distribution, thereby obtaining the structural tension and stability of the semantic joint representation architecture. Based on the local comparison edge satisfying state set, the cross-modal joint features satisfying boundary conditions are measured to see if they reach the preset lower bound standard of mutual information. The lower bound standard of mutual information is obtained by performing an integral estimation on the probability density distribution of positive sample pairs, set to 2.5 bits. The state set is extracted as a single sample pair, and the joint probability distribution and edge probability distribution are obtained. The mutual information is calculated by the expected value of the logarithm of their ratio. The calculated mutual information value is 2.8 bits, and the sample meets the standard. The structural tension and stability of the semantic joint representation architecture are evaluated by retrieving the feature dimension covariance matrix from the joint representation mutual information compliance record. The matrix size is 512 x 512. All eigenvalues ​​of this matrix are calculated, the degree of concentration of eigenvalue distribution is analyzed, and the ratio of the largest eigenvalue to the smallest eigenvalue is calculated. A ratio of 15 is within the preset range of 10 to 20, indicating that the structural tension is moderate. The eigenvalue distribution entropy is calculated, and the information entropy retention rate of the semantic joint representation architecture during the feature compression process is determined by analyzing the entropy value. An entropy value of 0.85 represents the retention of most of the effective information. The compliance status of each dimension and sample is recorded, generating a joint representation mutual information compliance record.

[0023] S313: Based on the joint representation mutual information achievement record, extract the mutual information achievement status in sequence and cross-validate it with the global pre-training alignment rule of the large model. When determining that the joint representation is completely closed, identify the hard positive and hard negative sample mask identifiers for the pairing, register the current tensor node in the mapping task stack, and generate the semantic alignment boundary constraint judgment matrix. Based on the mutual information compliance records of the joint representation, mutual information compliance status is extracted and cross-validated with the global pre-trained alignment rules. The alignment rules define a global statistical threshold for semantic consistency, requiring a global cosine similarity mean greater than 0.88. A logical AND operation is performed between the local compliance results and the global threshold. Sample identification is performed when the joint representation is completely closed-loop. Hard positive sample masking is performed for paired samples (i.e., semantically consistent samples but close to the boundary in feature space); hard negative sample masking is performed for samples (i.e., semantically inconsistent samples but close to interference samples in feature space). Positive samples with Euclidean distances between 0.48 and 0.50 are marked as hard positive samples and assigned a mask value of 101; negative samples with distances between 0.50 and 0.52 are marked as hard negative samples and assigned a mask value of 102. The current tensor node address and corresponding masking identifier are registered in the mapping task stack. Tensor data with boundary constraint information are integrated to generate a semantic alignment boundary constraint judgment matrix.

[0024] The specific steps for obtaining the pre-trained alignment loss weight modulation parameter set are as follows: S411: Based on the hard positive and hard negative sample mask identifiers in the semantic alignment boundary constraint judgment matrix, extract the feature structures that need to be punished or rewarded in the alignment task, combine them with the corresponding semantic anchor spatial location, and remove irrelevant semantics through tensor filtering to obtain the set of hard example comparison gradient contributions. In the alignment task, the feature structure that needs to be punished or rewarded refers to the hard positive and hard negative sample mask identifiers in the semantic alignment boundary constraint judgment matrix. By calculating the gradient magnitude of the feature vector in the semantic space, the feature dimension whose gradient magnitude exceeds the preset sensitivity threshold is identified, and the feature structure that needs to be punished or rewarded in the alignment task is obtained. Based on the hard positive and hard negative sample mask identifiers in the semantic alignment boundary constraint decision matrix, a retrieval mechanism is used to extract the feature structures that require key punishment or reward in the alignment task. The decision matrix is ​​traversed to locate tensor nodes with mask values ​​of 101 and 102. Corresponding feature vectors are extracted for each node, and sensitive dimensions are identified by calculating the gradient magnitude of the feature vectors in the semantic space. The partial derivative of the 128th dimension feature under the loss function is calculated; the gradient value is 0.045, exceeding the preset sensitivity threshold of 0.03, thus this dimension is determined to be a key feature structure requiring key punishment or reward. Combining the spatial location of the corresponding semantic anchor points, irrelevant semantic dimensions with gradient magnitudes less than the threshold are filtered out using tensor masks. The 5th dimension, with a gradient of 0.002, is also removed. The filtered high-sensitivity gradient data are then categorized to obtain a set of gradient contributions for hard examples.

[0025] S412: Based on the set of gradient contributions from difficult examples, combined with the alignment baseline value and gradient scaling adjustment function corresponding to the current pre-training batch, calculate the alignment loss scaling factor required for the batch, reconstruct the preset threshold of cross-entropy constraints in the pre-trained network, and obtain the reconstructed cross-entropy constraint scaling value. The alignment loss scaling factor refers to the ratio of the average gradient to the alignment baseline value corresponding to the current pre-training batch, obtained by nonlinearly mapping the gradient mean in the set of gradient contributions of hard examples using the exponential gradient scaling adjustment function, and calculating the ratio of the average gradient to the alignment baseline value. Based on the set of gradient contributions from difficult examples, the alignment benchmark is calculated using the corresponding alignment baseline value of the current pre-training batch. The alignment baseline value is determined by the average alignment loss statistics of the first 50 training batches, set to 0.42. The mean gradient from the set of gradient contributions from difficult examples is retrieved, specifically 0.85. An exponential gradient scaling function is used to perform a non-linear mapping on this mean value. This function operates on the natural logarithm as the base, using the quotient of the mean gradient divided by the temperature coefficient 0.07 as the exponent. The ratio of the mean gradient (0.85) to the alignment baseline value (0.42) is calculated to obtain the alignment loss scaling factor, which is 2.02. Subsequently, the pre-set threshold for cross-entropy constraints in the pre-trained network is reconstructed. The initial threshold of 0.5 is retrieved, and divided by the alignment loss scaling factor 2.02, yielding the reconstructed cross-entropy constraint scaling value.

[0026] S413: Based on the reconstructed cross-entropy constraint ratio, the ratio is overlaid into the hyperparameter configuration items of the pre-trained loss function of the large model, the version timestamp and architecture node number of the alignment pace are updated, and the pre-trained alignment loss weight modulation parameter set is generated. Based on the reconstructed cross-entropy constraint ratio, the numerical value is overlaid into the hyperparameter configuration of the pre-trained loss function. The coefficient of the penalty term in the loss function is modified, updated from the default configuration of 1.0 to 0.2475. To ensure the traceability of the training process, the alignment pace version timestamp and the current processing architecture node number are updated synchronously. The timestamp is recorded as 202604200830, and the node number is recorded as layer 12 head 8. The reconstructed ratio, penalty coefficient, and updated metadata are encapsulated to generate a pre-trained alignment loss weight modulation parameter set. This parameter set serves as a configuration instruction flow to the optimizer module, ensuring that parameter updates are performed with the new weight allocation logic during the next round of gradient backpropagation. After adopting this modulation parameter set, the alignment error rate is reduced by 22 percentage points when processing highly confusing semantic pairs, enhancing the aggregation of semantic space distribution.

[0027] The specific steps for obtaining the cross-modal control sequence set for large model pre-training are as follows: S511: Based on the pre-trained aligned loss weight modulation parameter set, the optimized loss weights and penalty term parameters are incorporated into the word segmentation array configuration structure of the current pre-training batch. The multimodal attention mask matrix is ​​updated at the feature boundary to generate the multimodal mask aligned configuration structure. Based on the pre-trained alignment loss weight modulation parameter set, the optimized loss weight of 0.2475 and the penalty term parameter are incorporated into the current pre-trained batch word segmentation array configuration structure. The word segmentation array is composed of concatenated text word segmentation sequences and image feature block sequences, with a total length set to 1024. At the feature boundaries of the word segmentation array, i.e., the intersection of the image sequence and the text sequence, a multimodal attention mask matrix update is performed. The connection weights of corresponding hard positive and hard negative samples in the mask matrix are reset according to the modulation parameters. The original weights of 1.0 are reduced to 0.25 after identifying boundary fluctuations. This adjustment strategy forces the model to avoid high-distortion paths in the attention allocation stage, ensuring that information flow conforms to the preset semantic alignment benchmark. After the weight reset is completed, the mask matrix and word segmentation sequences are concatenated using tensors to generate a multimodal mask alignment configuration structure. This configuration structure not only defines the flow intensity between different modal information at the current iteration step size, but also precisely limits the direction of cross-modal interaction.

[0028] S512: Based on the multimodal mask alignment configuration structure, evaluate the semantic advancement pace of the current word segmentation array in the pre-trained model, bind the tensor-level pace rate with gradient constraint logic, identify the underlying sequence synchronization signal parameters according to the preset control beat, and obtain the semantic alignment pace sequence cache set; The semantic advancement pace of the current word segmentation array in the pre-trained model refers to obtaining the attention score of each word segmentation tensor in the multimodal mask alignment configuration structure, calculating the attention center displacement vector between adjacent training iteration steps, determining the pace rate at the tensor level, and obtaining the semantic advancement pace of the current word segmentation array in the pre-trained model. Based on a multimodal mask alignment configuration structure, the semantic advancement pace of the current word segmentation array in the pre-trained model is evaluated. The attention score of each word segmentation tensor in the mask alignment configuration structure is obtained; for example, the attention score for the text segmentation "athlete" is 0.88. The displacement vector of the attention center between two adjacent training iteration steps is calculated. If the word segmentation attention center's coordinates in the feature space were 0.45 in the previous step and moved to 0.52 in the current step, the displacement vector magnitude is 0.07, which is defined as the tensor-level pace rate. The pace rate is bound to gradient constraint logic, and it is determined whether the rate exceeds a preset stability threshold of 0.1. When the rate is 0.07, the value is less than 0.1, indicating stable pace. Following a preset control rhythm (one synchronization pulse every 10 clock cycles), the underlying sequence synchronization signal parameters are identified. The parameter set includes the alignment stage identifier of each word segmentation array and the rate synchronization factor. Through real-time monitoring of the pace rate, a semantic alignment pace sequence cache set is generated. This cache set records the semantic drift trend of each word across different time dimensions, providing a crucial time benchmark for subsequent data loading and synchronization.

[0029] S513: Based on the semantic alignment pace sequence cache set, the synchronization signal and the multimodal word segmentation array content with attached constraint tags are sequentially written into the data loader interface of the underlying big model in the form of continuous tensors to obtain the big model pre-training cross-modal control sequence set. Based on the semantic alignment pace sequence cache set, the synchronization signal and the content of the multimodal word segmentation array with accompanying constraint tags are integrated. Continuous tensor data are streamed in a predetermined order, ensuring a strict correspondence between the data stream and the temporal logic within the model. Tensor streams containing word segmentation coordinates, attention weights, and mask identifiers are sequentially written to the underlying data loader interface. The data loader distributes the data to different computation nodes based on the synchronization signal parameters, ensuring that each multimodal data pair undergoes preprocessing according to the latest loss weights and masking strategy before entering the computation kernel. Through underlying sequence control, the pre-training process is aligned and adjusted at a microscale, resulting in a large-scale model pre-training cross-modal control sequence set. This sequence set serves as the core input to the training loop, directly driving weight convergence. During processing, integrity checks are performed on each set of tensor streams, comparing the data mapping relationship before and after writing. Through this precise streaming control, it can be ensured that the cross-modal pre-trained model is fully in the preset semantic alignment state when it receives input features. This not only solves the common modal collapse and semantic drift problems in the training of large models, but also significantly improves the model's semantic understanding accuracy and inference stability for long sequence multimodal inputs, laying the foundation for building high-performance cross-modal models.

[0030] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention in any other way. Any person skilled in the art may make changes or modifications to the above-disclosed technical content to create equivalent embodiments that can be applied to other fields. However, any simple modifications, equivalent changes, and modifications made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the protection scope of the present invention.

Claims

1. A method for processing large model pre-training data with cross-modal semantic alignment, characterized in that, Includes the following steps: S1: Obtain the structured semantic features of cross-modal input pairing instances, quantify the mapping distribution coordinates of differentiated modalities in a unified semantic space, calculate the displacement difference of each modality relative to the absolute semantic origin, identify the core semantic equivalence boundary, and obtain the cross-modal semantic anchor benchmark dataset; S2: Based on the cross-modal semantic anchor benchmark dataset, identify the cross-attention projection path of heterogeneous modal data to semantic anchors, analyze the degree of topological deformation of adjacent projection levels, quantify the semantic distortion in the cross-modal bridging process, and obtain the cross-modal mapping fluctuation amplitude dataset. S3: Based on the cross-modal mapping fluctuation amplitude dataset, calculate the local contrast loss of cross-modal pairing instances and determine whether it is at the semantic equivalence tolerance boundary, check the lower bound of cross-modal joint representation mutual information, output hard sample mask instructions when the structural integrity condition is met, and generate a semantic alignment boundary constraint judgment matrix. S4: Based on the hard sample mask instruction state in the semantic alignment boundary constraint judgment matrix, extract the gradient contribution weight required for the corresponding cross-modal pairing, and dynamically update the scaling weight factor of the alignment loss by reconstructing the cross-entropy constraint conditions in the pre-training process to obtain the pre-training alignment loss weight modulation parameter set. S5: Based on the pre-trained alignment loss weight modulation parameter set, the updated weight factors are injected into the word segmentation sequence configuration of the current pre-training batch, and the tensor sequence including constraint tags is written into the data loader interface to obtain the large model pre-training cross-modal control sequence set.

2. The method for processing large model pre-training data with cross-modal semantic alignment according to claim 1, characterized in that, The specific steps for obtaining the cross-modal semantic anchor benchmark dataset are as follows: S111: Based on the structured semantic features of cross-modal input pairing instances, extract the shared underlying semantic dimension in the multimodal alignment task, calculate the absolute spatial coordinates of each modal feature vector on the shared dimension, and obtain the basic semantic spatial coordinate parameters. S112: Based on the basic semantic space coordinate parameters, calculate the distribution distance of heterogeneous modalities with semantic correspondence when mapped to the same spatial coordinates, measure the semantic representation offset inherent in the differentiated modal attributes, and obtain the modality-specific semantic drift index. S113: Based on the modality-specific semantic drift index, the multimodal drift range is fused into the shared semantic coordinate system, defining the maximum tolerance boundary for differentiated modal data to be considered semantically equivalent in the feature space, and generating a cross-modal semantic anchor benchmark dataset.

3. The method for processing large model pre-training data with cross-modal semantic alignment according to claim 2, characterized in that, The specific steps for obtaining the cross-modal mapping fluctuation amplitude dataset are as follows: S211: Based on the cross-modal semantic anchor benchmark dataset, identify the layer-by-layer cross-attention network that maps heterogeneous modal features to semantic anchor boundaries, record the distribution of attention weight matrix of each layer when the network performs projection approximation, and obtain the multi-layer attention projection weight sequence. S212: Based on the multi-layer attention projection weight sequence, extract the weight topology relationship between adjacent projection layers in the network, calculate the feature space distortion and topological distance difference that occur during the bridging transmission of cross-modal semantics, and form the cross-modal bridging trajectory distortion variable. S213: Based on the cross-modal bridging trajectory distortion variables, the distortion values ​​of the differentiated levels are mapped to the manifold space based on the tolerance boundaries of the semantic anchor points. The distortion severity level of the differentiated attention head is identified by comparing the upper and lower limits of the anchor points and tensor labels are attached to generate a cross-modal mapping fluctuation amplitude dataset.

4. The method for processing large model pre-training data with cross-modal semantic alignment according to claim 3, characterized in that, The weight topology relationship refers to calculating the Euclidean distance between two adjacent weight matrices in a multi-layer attention projection weight sequence, identifying the spatial evolution trajectory of the weight distribution during the projection process, and obtaining the weight topology relationship.

5. The method for processing large model pre-training data with cross-modal semantic alignment according to claim 3, characterized in that, The steps for obtaining the semantic alignment boundary constraint determination matrix are as follows: S311: Based on the cross-modal mapping fluctuation amplitude dataset, calculate the local contrastive learning loss for each cross-modal mapping instance, determine whether its local semantic representation has successfully fallen back to the tolerance boundary interval of semantic equivalence, record the boundary satisfaction state label, and generate a local contrastive edge satisfaction state set. S312: Based on the local comparison edge satisfying state set, calculate whether the cross-modal joint features satisfying the boundary reach the preset mutual information lower bound standard, evaluate the structural tension and stability of the semantic joint representation architecture, and generate a joint representation mutual information compliance record; S313: Based on the joint representation mutual information achievement record, extract the mutual information achievement status in sequence and cross-validate it with the global pre-training alignment rule of the large model. When the joint representation is determined to be completely closed, identify the hard positive and hard negative sample mask identifiers for the pairing, register the current tensor node in the mapping task stack, and generate the semantic alignment boundary constraint judgment matrix.

6. The method for processing large model pre-training data with cross-modal semantic alignment according to claim 5, characterized in that, The evaluation of the structural tension and stability of the semantic joint representation architecture refers to calculating the eigenvalues ​​of the covariance matrix of the feature dimensions in the mutual information achievement records of the joint representation, determining the information entropy retention rate of the semantic joint representation architecture in the feature compression process by analyzing the concentration of the eigenvalue distribution, and thus obtaining the structural tension and stability of the semantic joint representation architecture.

7. The method for processing large model pre-training data with cross-modal semantic alignment according to claim 5, characterized in that, The specific steps for obtaining the pre-trained alignment loss weight modulation parameter set are as follows: S411: Based on the hard positive and hard negative sample mask identifiers in the semantic alignment boundary constraint judgment matrix, extract the feature structures that need to be punished or rewarded in the alignment task, combine them with the corresponding semantic anchor point spatial location, and remove irrelevant semantics through tensor filtering to obtain the set of hard example comparison gradient contributions. S412: Based on the set of gradient contribution comparisons of difficult examples, combined with the alignment benchmark value and gradient scaling adjustment function corresponding to the current pre-training batch, calculate the alignment loss scaling factor required for the batch, reconstruct the preset threshold of cross-entropy constraints in the pre-training network, and obtain the reconstructed cross-entropy constraint scaling value. S413: Based on the reconstructed cross-entropy constraint ratio value, the ratio value is overlaid into the hyperparameter configuration item of the large model pre-training loss function, the version timestamp and architecture node number of the alignment pace are updated, and a pre-training alignment loss weight modulation parameter set is generated.

8. The method for processing large model pre-training data with cross-modal semantic alignment according to claim 7, characterized in that, The feature structures that need to be penalized or rewarded in the alignment task refer to the hard positive and hard negative sample mask identifiers in the semantic alignment boundary constraint judgment matrix, the gradient magnitude of the feature vector in the semantic space is calculated, and the feature dimensions whose gradient magnitude exceeds the preset sensitivity threshold are identified to obtain the feature structures that need to be penalized or rewarded in the alignment task. The alignment loss scaling factor refers to obtaining the alignment baseline value corresponding to the current pre-training batch, using the exponential gradient scaling adjustment function to perform nonlinear mapping on the gradient mean in the set of gradient contributions for hard examples, and calculating the ratio of the gradient mean to the alignment baseline value to obtain the alignment loss scaling factor.

9. The method for processing large model pre-training data with cross-modal semantic alignment according to claim 7, characterized in that, The specific steps for obtaining the large model pre-training cross-modal control sequence set are as follows: S511: Based on the pre-trained aligned loss weight modulation parameter set, the optimized loss weight and penalty term parameters are incorporated into the word segmentation array configuration structure of the current pre-training batch, and the multimodal attention mask matrix is ​​updated at the feature boundary to generate a multimodal mask aligned configuration structure. S512: Based on the multimodal mask alignment configuration structure, evaluate the semantic advancement pace of the current word segmentation array in the pre-trained model, bind the tensor-level pace rate with gradient constraint logic, identify the underlying sequence synchronization signal parameters according to the preset control beat, and obtain the semantic alignment pace sequence cache set; S513: Based on the semantic alignment pace sequence cache set, the synchronization signal and the multimodal word segmentation array content with attached constraint tags are sequentially written into the data loader interface of the large model in continuous tensor form to obtain the large model pre-training cross-modal control sequence set.

10. The method for processing large model pre-training data with cross-modal semantic alignment according to claim 9, characterized in that, The semantic advancement pace of the current word segmentation array in the pre-trained model refers to obtaining the attention score of each word segmentation tensor in the multimodal mask alignment configuration structure, calculating the attention center displacement vector between adjacent training iteration steps, determining the pace rate at the tensor level, and obtaining the semantic advancement pace of the current word segmentation array in the pre-trained model.

Images