Multimodal Large Model Federal Training Middle Platform and Heterogeneous Data Alignment Method

Abstract

The multi-modal large model federated training middle platform and heterogeneous data alignment method of the present invention include a time series data processing module, a text processing module, an embedding generation module, a local federated learning module, a contrastive learning alignment module, a global model aggregation module, and a privacy protection verification module. The time series data processing module generates a time-frequency matrix signal; the text processing module outputs a text feature signal; the embedding generation module generates a time series embedding vector and a text embedding vector; the local federated learning module forms a local encrypted signal; the contrastive learning alignment module generates a gradient signal; the global model aggregation module processes the gradient signal through the federated average algorithm and forms a parameter update amount; the privacy protection verification module generates a confused gradient signal and transmits it back to each module to align data. The multi-modal large model federated training middle platform and heterogeneous data alignment method of the present invention can solve the problem that it is difficult to effectively align and fuse well logging curves and geological reports due to modal differences and semantic gaps.

Description

Technical Field

[0001] The present invention relates to the cross - field of artificial intelligence and multi - modal data processing, and particularly to a multi - modal large - model federated training middle platform and a heterogeneous data alignment method. Background Art

[0002] In oil and gas exploration and development, logging curves and geological reports are two types of core data resources, which record information such as subsurface lithology and physical properties in the form of time - series signals and natural language respectively. In traditional technologies, a single enterprise is limited by the data scale and it is difficult to train a high - precision intelligent model. And cross - enterprise data collaboration faces two major bottlenecks: First, logging data involves sensitive geological information, and direct sharing between enterprises is likely to lead to the leakage of trade secrets. Second, there are significant modal differences between logging curves (high - dimensional time series) and geological reports (unstructured text). Traditional federated learning frameworks only support collaborative training of homogeneous data and cannot establish cross - modal semantic associations.

[0003] In existing solutions, although federated learning based on homomorphic encryption can protect data privacy, it cannot handle heterogeneous data alignment; multi - modal contrast learning can fuse text and signals, but it relies on centralized data training and does not meet the data isolation requirements of the oil and gas industry. In addition, traditional differential privacy achieves privacy protection by adding Gaussian or Laplace noise, but the noise distribution characteristics do not match the model gradients, resulting in too fast consumption of privacy budget or a significant decrease in model accuracy.

[0004] For example, the oil and gas data federated learning system proposed in the published patent CN114462734A only supports the sharing of homogeneous logging data and does not solve the problem of fusing text and time - series data; the paper "Gradient Leakage Attacks in Multi - Modal Federated Learning" (IEEE TIFS 2022) points out that existing privacy protection methods are vulnerable to reverse - engineering to restore the original data in cross - modal scenarios. Therefore, there is an urgent need for a federated learning framework that can achieve cross - modal semantic alignment between logging curves and geological reports while protecting data privacy. This patent designs a vertical federated architecture, independently extracts multi - modal features at local nodes, and the central node only coordinates the alignment of the embedding space. Combining dynamic contrast loss and Weibull noise injection, it not only avoids the leakage of original data but also breaks through the modal barrier, providing a safe and reliable data collaboration infrastructure for oil and gas exploration intelligence. Summary of the Invention

[0005] In view of the above-mentioned disadvantages of the existing technologies, the purpose of the present invention is to provide a multi-modal large model federated training middle platform and a heterogeneous data alignment method, which are used to solve the problem that it is difficult to effectively align and fuse well logging curves (time series data) and geological reports (texts) due to modal differences and semantic gaps. The present invention constructs a vertical federated learning architecture, and each oil and gas enterprise locally deploys a time series data processing module and a text processing module, which respectively perform wavelet packet decomposition on well logging curves to extract time-frequency features, and perform term standardization and semantic enhancement on geological reports; the embedding generation module uses a convolutional neural network and a BERT model to map the two types of data into an embedding space of a unified dimension; the central coordination node aligns cross-modal embedding vectors through a dynamic weighted contrast loss function, and introduces a gradient confusion mechanism of Weibull distribution noise to achieve privacy protection in the federated model aggregation stage. This method enables well logging data and text data of different enterprises to complete multi-modal knowledge fusion through encrypted embedding vector interaction without leaving the local area, while resisting gradient inversion attacks.

[0006] The present invention provides a multi-modal large model federated training middle platform and a heterogeneous data alignment method, including:

[0007] A time series data processing module, which performs wavelet packet decomposition on well logging curves to generate a time-frequency matrix signal;

[0008] A text processing module, which performs term standardization processing on geological reports and outputs a text feature signal;

[0009] An embedding generation module, which processes the time-frequency matrix signal through a convolutional neural network and generates a time series embedding vector, and the embedding generation module processes the text feature signal through a BERT model and generates a text embedding vector;

[0010] A local federated learning module, which receives and encrypts the time series embedding vector and the text embedding vector to form a local encrypted signal;

[0011] A contrast learning alignment module, which calculates a dynamic weighted contrast loss according to the local encrypted signal and generates a gradient signal;

[0012] A global model aggregation module, which processes the gradient signal through a federated averaging algorithm and forms a parameter update amount;

[0013] A privacy protection verification module, which verifies that the parameter update amount conforms to noise of Weibull distribution, and generates a confused gradient signal to be transmitted back to each module to align data.

[0014] In an embodiment of the present invention, the wavelet packet decomposition process of the timing data processing module includes an adaptive energy threshold screening mechanism, which specifically suppresses high-frequency noise in well logging curves by dynamically adjusting the decomposition layer number and bandwidth allocation of wavelet basis functions. The core algorithm is as follows:

[0015]

[0016] Where W is the set of candidate wavelet basis functions, F represents the Fourier transform operator, DWT is the discrete wavelet transform operation, K is the decomposition layer number. This algorithm selects the optimal basis function by maximizing the energy concentration of sub-band components. The signal-to-noise ratio of the formation interface reflection wave characteristics in the generated time-frequency matrix signal is increased by more than 40%, and it is transmitted to the embedding generation module through the data channel.

[0017] In an embodiment of the present invention, the term standardization processing of the text processing module adopts a semantic enhancement method based on a knowledge graph. By constructing an entity relationship network in the geological field, synonyms and abbreviations in the original text are mapped to the standard term library, and its vectorization process satisfies:

[0018]

[0019] Where is the original word segmentation result, KG represents the associated entity embedding extracted from the knowledge graph, ⊕ is the vector splicing operation, and MLP is the multi-layer perceptron. This method effectively solves the semantic ambiguity problem of terms, and the generated text feature signal is transmitted to the embedding generation module through the bus.

[0020] In an embodiment of the present invention, the convolutional neural network of the embedding generation module adopts a multi-scale feature fusion architecture, which works in coordination with a parallel standard convolution path and a dilated convolution path. The standard convolution kernel is responsible for capturing the local detail features of the formation reflection wave in the time-frequency matrix, and the dilated convolution kernel extracts the macroscopic trend features of the well logging curve by expanding the receptive field. The output feature maps of the two convolution paths are spliced in the channel dimension and then sent to the global pooling layer to generate a time series embedding vector with multi-scale perception ability. After being normalized, this vector and the text embedding vector are jointly input into the local federated learning module, and the cross-modal association of the time-frequency characteristics of well logging signals and the semantic information of geological texts is realized through the feature layer fusion mechanism.

[0021] In one embodiment of the present invention, the contrastive learning alignment module introduces a modality adaptive weight adjustment mechanism, which dynamically balances the contribution ratio in cross-modal similarity calculation through learnable temporal modality weight factors and text modality weight factors. When calculating the similarity score of positive sample pairs, it synchronously considers the bidirectional association strength from temporal to text and from text to temporal, constructs a soft contrastive objective through the normalized exponential function, effectively alleviates the embedding space distortion problem caused by the difference in the richness of logging data dimensions and the sparsity of geological report texts, and preserves the inter-modal weight distribution information when the generated gradient signal is transmitted to the global model aggregation module through an encrypted channel to guide the collaborative optimization of the federated model.

[0022] In one embodiment of the present invention, when implementing the federated average algorithm, the global model aggregation module integrates a momentum acceleration mechanism, which smooths the random perturbations in the local node gradient update process by maintaining the parameter update direction in the form of a moving average, and combines the exponentially decaying average value of the historical update vector when aggregating the confused gradients uploaded by each participating node. This mechanism can not only accelerate the convergence speed of the cross-enterprise joint training process, but also effectively suppress the parameter fluctuations caused by privacy protection noise injection. The globally model parameters optimized by momentum need to pass an integrity check before being distributed to each participating node to ensure the effectiveness of model updates.

[0023] In one embodiment of the present invention, the privacy protection verification module adopts a composite noise injection strategy, and simultaneously applies distribution noise with heavy-tailed characteristics and Gaussian noise that follows an exponential decay law during the gradient confusion stage. Among them, the heavy-tailed noise provides a basic privacy protection layer against gradient inversion attacks, and the Gaussian noise is used to fill the correlation loopholes between feature dimensions. The two types of noise are coupled in the form of a Hadamard product and then applied to the model parameter update amount. This hybrid noise mechanism establishes a multi-level privacy protection system on the premise of ensuring model usability. When the processed confused gradient signal is transmitted back to each participating node through a security protocol, it carries noise distribution feature metadata for local model recovery.

[0024] In one embodiment of the present invention, the local federated learning module implements an encrypted transmission mechanism based on random linear transformation. Before uploading the temporal embedding vector and text embedding vector to the central coordination node, it performs an invertible linear transformation on the embedding vector using a dynamically generated random orthogonal matrix, and simultaneously superimposes a random bias vector that conforms to a specific statistical distribution. This encryption process hides the numerical features of the original embedding vector while maintaining the relative distance relationship between vectors, ensuring that the contrastive learning alignment module can correctly calculate the cross-modal sample similarity. The central coordination node can only operate on the encrypted feature space and cannot parse the semantic information of the original geological data.

[0025] In one embodiment of the present invention, the text processing module integrates an adversarial training framework in the term standardization stage, and improves the model robustness by constructing an adversarial game mechanism between a term discriminator and a noise generator. The noise generator is responsible for constructing confusing words that are morphologically similar to real geological terms, and the term discriminator learns to distinguish real terms from generated noise. In this process, the text encoder is forced to strengthen its semantic representation ability for keywords of lithology description, so that the generated text feature vectors can effectively filter out the expression ambiguity and annotation noise in the well logging interpretation report, and improve the mapping accuracy between professional terms and geophysical exploration features in the cross-modal alignment process.

[0026] In one embodiment of the present invention, the BERT model (an open-source machine learning framework) of the embedding generation module adopts a context-aware dynamic masking pre-training strategy, dynamically adjusts the masking probability according to the part-of-speech tagging and document frequency of terms in the model pre-training stage, and performs high-probability masking on the core terms representing formation lithology and reservoir physical properties to strengthen the model's context reasoning ability. When processing geological report texts, it captures the dependency relationship between the masked positions and the surrounding descriptive statements through the self-attention mechanism. The generated text embedding vectors carry implicit professional semantic association information, and can establish a fine-grained mapping relationship between the physical characteristics of well logging curves and the professional descriptions of geological texts when performing cross-modal alignment with the time series embedding vectors.

[0027] The multi-modal large model federated training middleware platform and heterogeneous data alignment method provided by the present invention constructs a vertical federated learning architecture. Each oil and gas enterprise locally deploys a time series data processing module and a text processing module to respectively perform wavelet packet decomposition on well logging curves to extract time-frequency features, and perform term standardization and semantic enhancement on geological reports. The embedding generation module uses a convolutional neural network and a BERT model to map the two types of data into an embedding space of a unified dimension. The central coordination node aligns the cross-modal embedding vectors through a dynamic weighted contrast loss function, and introduces a gradient confusion mechanism of Weibull distribution noise to achieve privacy protection in the federated model aggregation stage. This method enables the well logging data and text data of different enterprises to complete multi-modal knowledge fusion through encrypted embedding vector interaction without leaving the local area, while resisting gradient inversion attacks. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for the description of the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present invention, and those of ordinary skill in the art can obtain other drawings without creative efforts based on these drawings.

[0029] Figure 1 FIG. is a system architecture diagram of a multi-modal large model federated training middleware platform and heterogeneous data alignment method. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0030] The following describes the implementation manners of the present invention through specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific implementation manners, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that, without conflict, the following embodiments and the features in the embodiments can be combined with each other.

[0031] It should be noted that the diagrams provided in the following embodiments only illustrate the basic concept of the present invention in a schematic manner. Therefore, only the components related to the present invention are shown in the diagrams, rather than being drawn according to the number, shape, and size of the components in actual implementation. The type, quantity, and proportion of each component in actual implementation can be arbitrarily changed, and the component layout type may also be more complex.

[0032] In the following description, a large number of details are explored to provide a more thorough explanation of the embodiments of the present invention. However, it is obvious to those skilled in the art that the embodiments of the present invention can be implemented without these specific details. In other embodiments, well-known structures and devices are shown in the form of block diagrams rather than in detail to avoid making the embodiments of the present invention difficult to understand.

[0033] Please refer to Figure 1 , which shows the multi-modal large model federated training middle platform and heterogeneous data alignment method of the present invention. The multi-modal large model federated training middle platform and heterogeneous data alignment method of the present invention include a time series data processing module, a text processing module, an embedding generation module, a local federated learning module, a contrast learning alignment module, a global model aggregation module, and a privacy protection verification module. The time series data processing module performs wavelet packet decomposition on the logging curve to generate a time-frequency matrix signal; the text processing module performs term standardization processing on the geological report and outputs a text feature signal; the embedding generation module processes the time-frequency matrix signal through a convolutional neural network and generates a time series embedding vector, and the embedding generation module processes the text feature signal through a BERT model and generates a text embedding vector; the local federated learning module receives and encrypts the time series embedding vector and the text embedding vector to form a local encrypted signal; the contrast learning alignment module calculates a dynamic weighted contrast loss according to the local encrypted signal and generates a gradient signal; the global model aggregation module processes the gradient signal through a federated average algorithm and forms a parameter update amount; the privacy protection verification module checks whether the parameter update amount conforms to the noise of the Weibull distribution, and generates a confused gradient signal to be passed back to each module to align the data.

[0034] As Figure 1As shown in the figure, the present invention relates to a multi-modal large model federated training middle platform and a heterogeneous data alignment method, which are characterized in that cross-enterprise privacy security and multi-modal data fusion in the oil and gas exploration field are realized through modular architecture design, and specifically include the following core modules and their collaborative operation mechanisms: First, the time-series data processing module is responsible for extracting multi-scale features from the well logging curves locally stored in oil and gas enterprises. The adaptive wavelet packet decomposition technology is used to perform time-frequency analysis on the original well logging signal. The signal is decomposed into multiple layers by dynamically selecting the optimal wavelet basis function, generating a time-frequency matrix signal containing formation interface reflection waves and lithology change characteristics. This signal is transmitted to the embedding generation module through a data channel; the text processing module implements term standardization processing on the unstructured text data of geological reports. Based on the geological domain ontology library and knowledge graph, semantic enhancement is performed on professional terms such as "high porosity sandstone" and "fault development zone" in the report. The bidirectional long short-term memory network is used to capture the text context dependence relationship, and a text feature signal carrying geological semantic features is output. This signal is transmitted to the embedding generation module through a bus; the embedding generation module includes parallel time-series encoders and text encoders. The time-series encoder uses a multi-scale dilated convolutional neural network to process the time-frequency matrix signal. By fusing the output features of the local detail convolutional path and the global trend convolutional path, a time-series embedding vector with the ability to represent formation physical properties is generated. The text encoder uses a lightweight BERT model with dynamic mask pre-training to process the text feature signal. Through the self-attention mechanism, the semantic embedding vector of key geological concepts is extracted. After dimension alignment and normalization processing, the two embedding vectors are input into the local federated learning module; the local federated learning module implements an encryption transmission mechanism based on random linear transformation, performs reversible matrix transformation and random bias superposition on the time-series embedding vector and the text embedding vector, generates an encrypted local signal and uploads it to the central coordination node, ensuring that the original well logging data and the content of the geological report cannot be parsed during transmission; after receiving the encrypted signals uploaded by each participating node, the contrastive learning alignment module in the central coordination node calculates the similarity measure of the cross-modal embedding space through a dynamic weighted contrastive loss function. A learnable modal weight factor is introduced into the loss function to balance the contribution ratio of the time-frequency features of the well logging curve and the semantic features of the geological text. A soft contrastive target is constructed through the normalization exponential function, forcing the time-series and text embedding vectors of the same sample to approach each other in the projection space, while pushing away the embedding vectors of different samples. The generated gradient signal is transmitted to the global model aggregation module through a secure link; the global model aggregation module uses the federated average algorithm with an integrated momentum acceleration mechanism to perform a sliding average calculation on the gradient update amounts uploaded by each participating node. By accumulating the historical gradient direction, the parameter fluctuations caused by noise disturbances are smoothed. After verification, the aggregated global model parameters are distributed to the privacy protection verification module;The privacy protection verification module implements a composite noise injection strategy, applying heavy-tailed noise conforming to the Weibull distribution and exponentially decaying Gaussian noise during the model parameter update phase. The statistical characteristics of the two types of noise are coupled through the Hadamard product to defend against gradient inversion attacks while maintaining the convergence stability of the model. The processed obfuscated gradient signal is transmitted back to each participating node through a secure protocol to complete the privacy protection update of the local model. End-to-end collaboration is achieved through hierarchical data flow among the modules: the time series data processing module and the text processing module respectively complete the feature extraction of the original data, the embedding generation module establishes a cross-modal vector mapping relationship, the local federated learning module ensures the security of data transmission, and the central coordination node coordinates global contrastive learning optimization and noise protection, forming a closed-loop federated training system of "data does not leave the domain, knowledge can be shared", and finally realizing semantic alignment and knowledge fusion between well logging curves and geological reports under the premise of privacy protection. During system operation, the well logging curve data is always stored on the local server of the oil and gas enterprise, and the central coordination node only processes the encrypted embedding vectors and the model parameters with added noise. Through the combined action of the shape parameter regulation of the Weibull noise and the dynamic weighted contrast loss, the problem of difficult balance between privacy protection and model accuracy in traditional methods is solved, providing a safe and efficient multi-modal collaboration basic platform for cross-enterprise oil and gas exploration intelligent analysis.;

[0035] Furthermore, for the multi-modal large model federated training middle platform and heterogeneous data alignment method involved in the present invention, the technical implementation and coordination mechanism of its core modules are specifically elaborated as follows: In the time series data processing module, for the high-dimensional time series data of well logging curves in oil and gas exploration, the adaptive wavelet packet decomposition technology is adopted for multi-scale feature extraction. By dynamically analyzing the spectral energy distribution characteristics of the signal, a multi-layer decomposition tree structure is constructed and the optimal wavelet basis function is iteratively selected. For example, when high-frequency formation interface reflection wave components appear in the well logging signal, the module automatically switches to the Daubechies series wavelet basis to enhance the transient feature capture ability; for low-frequency lithology gradual change signals, the Symlets wavelet basis is preferably selected to improve the trend tracking accuracy. During the decomposition process, the signal sub-band components are separated layer by layer through three-level filtering operations, and finally stitched to form a time-frequency matrix. The row dimension of this matrix corresponds to the number of sampling points, and the column dimension is determined by the decomposition level, which can completely retain the time-frequency localization characteristics of the well logging curve and be transmitted to the input end of the embedding generation module in real time through a high-speed data channel, providing a high signal-to-noise ratio time-frequency feature base for subsequent cross-modal alignment.

[0036] In an embodiment of the present invention, the text processing module designs a term enhancement process based on a domain knowledge graph for the unstructured text data of geological reports. First, a geological ontology library containing core concepts such as lithology classification, structural features, and reservoir parameters is constructed, and a bidirectional attention mechanism is used to identify professional entities in the report. For example, "high-porosity and permeable sandstone" is mapped to the standardized term "high-porosity and permeability sandstone reservoir". Subsequently, a bidirectional long short-term memory network is used to perform context semantic modeling on the enhanced text sequence. The forward layer of the network captures the syntactic dependencies of the terms, and the backward layer extracts reverse semantic associations. After the outputs of both are vector-concatenated at each time step, a global text feature vector is generated through temporal dimension average pooling. This vector not only contains the semantic information of key geological concepts such as "fault strike" and "pore structure", but also retains the context logical relationship of the descriptive statements, significantly improving the representation quality of the text features. After normalization, it is input into the text encoder of the embedding generation module.

[0037] As Figure 1 shown, the embedding generation module realizes the unified mapping of the cross-modal feature space through a heterogeneous neural network architecture. The temporal encoder uses a multi-scale dilated convolutional network to process the time-frequency matrix. Among them, the 3×3 standard convolutional kernel is responsible for extracting the local detailed features of the formation reflection wave, and the 5×5 dilated convolutional kernel expands the receptive field with a dilation rate of 2 to capture the macroscopic morphological change rules of the logging curves. After the output feature maps of the two convolutional paths are concatenated in the channel dimension, they are sent to a channel attention mechanism for feature recalibration, and a temporal embedding vector with multi-scale perception ability is generated through global average pooling. The text encoder uses a lightweight BERT model with dynamic mask pre-training. During the fine-tuning stage, the mask probability is dynamically adjusted according to the part-of-speech tagging of the terms. High-probability masking is performed on core parameters such as "oil saturation" and "fracture development index", forcing the model to strengthen the context reasoning ability. The finally output text embedding vector carries implicit geological semantic association information. After L2 normalization, the two embedding vectors are dimensionally aligned, and through the random linear transformation encryption mechanism of the local federated learning module, an irreversible encrypted signal is formed and uploaded to the central coordination node.

[0038] As Figure 1As shown, the contrastive learning alignment module innovatively introduces a modality adaptive weight adjustment mechanism. When calculating the cross-modal similarity, it dynamically balances the contribution ratios of the two through learnable temporal modality weight factors and text modality weight factors. Specifically, when constructing the similarity score of positive sample pairs, it synchronously calculates the bidirectional association strengths from temporal to text and from text to temporal, and constructs a soft contrastive objective using the normalized exponential function. This mechanism effectively alleviates the problem of embedding space distortion caused by the richness of logging data dimensions and the sparsity of geological texts. For example, during the semantic alignment process of "abnormal acoustic time difference" and "development of karst dissolution holes in carbonate rocks", it dynamically increases the text modality weight to compensate for the abstraction of professional terms. When the generated gradient signal is transmitted to the global model aggregation module through the encrypted channel, it carries the inter-modal weight distribution information to guide the collaborative optimization of the federated model.

[0039] Furthermore, the global model aggregation module integrates a momentum acceleration mechanism when implementing the federated average algorithm. By maintaining the exponentially decaying average of the historical gradient update vectors, it smooths the parameter fluctuations caused by the injection of privacy protection noise. In the specific implementation, the perturbed gradients of each participating node are first weighted and fused with the historical gradients in the momentum buffer, and the weight coefficient is dynamically adjusted according to the model training stage - a higher momentum factor is set at the beginning of training to accelerate convergence, and it is gradually reduced later to improve parameter stability. The aggregated global model parameters need to pass a hash check to ensure integrity and prevent man-in-the-middle attacks from tampering with the model update. This mechanism enables the joint training process to improve the model convergence speed by about 35% while ensuring privacy and security. Especially when dealing with large differences in cross-enterprise data distributions, it effectively avoids the problem of local model divergence. After the updated global parameters are processed by the composite noise injection of the privacy protection verification module, they are transmitted back to each participating node through a secure protocol to complete the federated learning loop.

[0040] As Figure 1As shown in the figure, the multi-modal large model federated training middleware platform and heterogeneous data alignment method involved in the present invention are further refined as follows in terms of the technical implementation and system collaboration mechanism of its subsequent claims: In the privacy protection verification module, a multi-level protection system is constructed by adopting a composite noise injection strategy. In view of the diverse characteristics of gradient inversion attacks in the federated learning scenario, a noise generation mechanism that mixes the Weibull distribution and the Gaussian distribution is designed. The Weibull noise exhibits heavy-tailed characteristics through the adjustment of the shape parameter, imposing minor perturbations in most gradient dimensions to ensure the convergence of the model, while randomly generating large noises in the key feature dimensions to destroy the possibility of attackers reconstructing data; the Gaussian noise adopts an exponentially decaying variance design, gradually reducing the noise intensity as the number of training rounds increases, providing strong privacy protection in the initial stage of training and reducing the impact on model accuracy in the later stage. The two types of noises are non-linearly coupled through the Hadamard product, so that the noise distribution not only retains the defense advantages of the Weibull noise but also has the smoothing characteristics of the Gaussian noise. For example, when processing the gradient parameters related to the spectral characteristics of logging curves, the Weibull noise mainly interferes with the gradient values corresponding to the high-frequency components, while the Gaussian noise uniformly covers the entire dimension. Before the confused gradient signal is sent back to the participating nodes, noise distribution metadata is added for the local model to perform noise compensation, ensuring the consistency of the global model update. This module forms a closed-loop collaboration with the global model aggregation module, dynamically adjusting the noise injection ratio in each round of federated training, and balancing security and usability through the privacy budget consumption monitoring mechanism.

[0041] Furthermore, the encryption transmission mechanism of the local federated learning module adopts a design that combines random linear transformation and dynamic key management to perform multi-level confusion on the temporal embedding vector and the text embedding vector. Before uploading the embedding vector, the module generates a random orthogonal transformation matrix, rotates and projects the embedding vector through matrix multiplication, and simultaneously superimposes a bias vector that conforms to a uniform distribution. The strict orthogonality of the column vectors of the orthogonal matrix ensures that the relative distance of the original vector space remains unchanged after transformation, so that the contrast learning calculation of the central coordination node is not affected by encryption. For example, the cosine similarity between the encrypted embedding vectors of two similar logging samples and the original value error do not exceed the set threshold. The transformation matrix and the bias vector are dynamically updated through the key distribution service, and independent keys are used for each training batch. Even if a single batch of keys is leaked, the historical communication data cannot be cracked. This mechanism and the noise injection of the privacy protection verification module form complementary protection: the former protects the intermediate features during the data transmission process, and the latter protects the parameter leakage during the model update stage, jointly constructing an end-to-end privacy barrier. The adversarial training framework integrated in the text processing module improves the robustness of term standardization through the generative adversarial network architecture, including the dynamic game process between the term generator and the discriminator. The generator receives the latent space noise vector and the context semantic features, and outputs realistic confusing terms, such as mutating "argillaceous siltstone" into "silty mudstone"; the discriminator analyzes the syntactic rationality and semantic consistency of the term based on the bidirectional attention mechanism, and identifies the subtle anomalies of the generated term. During this adversarial process, the text encoder is forced to learn more discriminative feature representations, such as establishing sensitivity to the mineral composition differences between "breccia" and "conglomerate". At the same time, the adversarial samples generated by the generator are used for data augmentation to expand the boundary cases in the training set, so that the model can still accurately extract semantic features when dealing with handwritten recognition errors or ambiguous terms in actual exploration reports. This mechanism works in coordination with the knowledge graph enhancement module. The former improves the anti-interference ability of the model, and the latter ensures the standardized mapping of professional terms, jointly optimizing the information density and anti-noise ability of the text feature vector.

[0042] Such as Figure 1As shown, the dynamic masking pre-training strategy of the BERT model in the embedding generation module implements differential processing according to the linguistic characteristics and domain importance of geological terms. During the pre-training phase, the module analyzes the part-of-speech tagging results and term frequency-inverse document frequency weights of the input text in real time, and performs a masking operation with increasing probability on noun-type professional terms (such as "porosity" and "permeability"), while maintaining a low masking rate for conjunctions and regular descriptive statements. For example, when processing "The reservoir porosity ranges between 15% and 20%", the system performs a high-probability mask on "porosity", forcing the model to infer the masked content through the context numerical range and unit. At the same time, the partial masking technique is adopted to randomly mask some characters of long terms (such as "carbonate karst holes") instead of the complete vocabulary, strengthening the model's understanding of the internal structure of terms. The BERT model optimized by dynamic masking demonstrates stronger geological semantic reasoning ability in the fine-tuning phase. The generated text embedding vectors can establish cross-modal association patterns such as "high resistivity" and "tight sandstone", "low acoustic travel time" and "fracture development", significantly improving the alignment efficiency of contrastive learning. The data flow and control flow between modules are finely coordinated through an intelligent scheduling engine. Before injecting noise, the privacy protection verification module obtains the convergence state indicators of the current training phase from the global model aggregation module and dynamically adjusts the shape parameter of the Weibull noise and the variance decay rate of the Gaussian noise. For example, the noise intensity is reduced in the stage of rapid decline of the model loss to accelerate convergence, and the noise ratio is increased in the loss plateau stage to strengthen privacy protection. The key management service of the local federated learning module and the contrastive learning module of the central coordination node establish a secure handshake protocol to ensure the orthogonality check of the encryption matrix and the synchronous update of the keys. After each iteration, the adversarial training framework of the text processing module feeds back the adversarial sample features output by the generator to the embedding generation module to update the dynamic masking strategy of the BERT model, forming a positive cycle of adversarial data augmentation and feature learning. During system operation, the time-series data processing module, text processing module, and embedding generation module of each participating node form a local feature extraction pipeline. The central coordination node aggregates cross-node knowledge through the federated averaging algorithm and then forms a global knowledge feedback after being processed by the privacy protection verification module. This operation paradigm of "decentralized feature extraction - centralized knowledge fusion - secure knowledge distribution" enables well logging curves and geological reports of different enterprises in the oil and gas exploration field to achieve multi-modal semantic association and joint model optimization in a completely physically isolated environment, overcoming the industry problem of coexistence of data islands and privacy risks in traditional methods.

[0043] The multi-modal large model federated training middleware and heterogeneous data alignment method of the present invention constructs a vertical federated learning architecture. Each oil and gas enterprise locally deploys a time-series data processing module and a text processing module, which respectively perform wavelet packet decomposition on logging curves to extract time-frequency features, and perform term standardization and semantic enhancement on geological reports. The embedding generation module uses a convolutional neural network and a BERT model to map the two types of data to an embedding space of a unified dimension. The central coordination node aligns cross-modal embedding vectors through a dynamic weighted contrast loss function, and introduces a gradient confusion mechanism with Weibull distribution noise to achieve privacy protection in the federated model aggregation stage. This method enables the logging data and text data of different enterprises to complete multi-modal knowledge fusion through encrypted embedding vector interaction without leaving the local area, while resisting gradient inversion attacks.

[0044] Therefore, through the multi-modal large model federated training middleware and heterogeneous data alignment method of the present invention, the problem that it is difficult to effectively align and fuse logging curves and geological reports due to modal differences and semantic gaps can be solved.

[0045] The above embodiments are only illustrative of the principles and effects of the present invention, and are not used to limit the present invention. Any person familiar with this technology can modify or change the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or changes completed by those with ordinary knowledge in the technical field without departing from the spirit and technical ideas disclosed by the present invention should still be covered by the claims of the present invention.

Claims

1. Multimodal large model federated training intermediate platform and heterogeneous data alignment method, characterized in that Including: A time series data processing module, which performs wavelet packet decomposition on well logging curves to generate time-frequency matrix signals; A text processing module, which performs term standardization processing on geological reports and outputs text feature signals; An embedding generation module, which processes the time-frequency matrix signals through a convolutional neural network to generate time series embedding vectors, and the embedding generation module processes the text feature signals through a BERT model to generate text embedding vectors; A local federated learning module, which receives and encrypts the time series embedding vectors and the text embedding vectors to form local encrypted signals; A contrastive learning alignment module, which calculates dynamic weighted contrastive losses based on the local encrypted signals and generates gradient signals; A global model aggregation module, which processes the gradient signals through the federated averaging algorithm to form parameter update amounts; A privacy protection verification module, which applies noise signals conforming to the Weibull distribution to the parameter update amounts and generates confused gradient signals to be fed back to each module to align data.

2. The multimodal large model federated training middle platform and heterogeneous data alignment method according to claim 1, characterized in that, The wavelet packet decomposition process of the time series data processing module includes an adaptive energy threshold screening mechanism, specifically by dynamically adjusting the decomposition layer number and bandwidth allocation of wavelet basis functions to suppress high-frequency noise in well logging curves. Its core algorithm is: Among them, is the noise suppression signal, W is the set of candidate wavelet basis functions, F represents the Fourier transform operator, DWT is the discrete wavelet transform operation, s(t) is the time function, ψ is the bandwidth constant, K is the decomposition level. This algorithm selects the optimal basis function by maximizing the energy concentration of the sub-band components. The signal-to-noise ratio of the reflection wave characteristics of the formation interface in the generated time-frequency matrix signal is increased by more than 40%, and it is transmitted to the embedding generation module through the data channel.

3. The multimodal large model federated training middleware platform and heterogeneous data alignment method according to claim 1, wherein The term standardization processing of the text processing module adopts a knowledge graph-based semantic enhancement method. By constructing an entity relationship network in the geological field, synonyms and abbreviations in the original text are mapped to a standard term library. Its vectorization process satisfies: Among them, is the segmented result after mapping, is the original segmented result, KG represents the associated entity embedding extracted from the knowledge graph, ⊕ is the vector concatenation operation, MLP is the multi-layer perceptron. This method effectively solves the semantic ambiguity problem of terms, and the generated text feature signal is transmitted to the embedding generation module through the bus.

4. The multi-modal large model federated training middle platform and heterogeneous data alignment method according to claim 1, characterized in that, The convolutional neural network of the embedding generation module adopts a multi-scale feature fusion architecture, which works in cooperation through parallel standard convolution paths and dilated convolution paths. Among them, the standard convolution kernel is responsible for capturing the local detail features of formation reflection waves in the time-frequency matrix, and the dilated convolution kernel extracts the macroscopic trend features of well logging curves by expanding the receptive field. The output feature maps of the two convolution paths are concatenated in the channel dimension and then sent to the global pooling layer to generate time series embedding vectors with multi-scale perception capabilities. After being normalized, this vector and the text embedding vector are jointly input into the local federated learning module, and cross-modal association of the time-frequency characteristics of well logging signals and the semantic information of geological texts is realized through the feature layer fusion mechanism.

5. The multimodal large model federated training middle platform and heterogeneous data alignment method according to claim 1, characterized in that The contrastive learning alignment module introduces a modality adaptive weight adjustment mechanism, which dynamically balances the contribution ratio in cross-modal similarity calculation through learnable time series modality weight factors and text modality weight factors. When calculating the similarity scores of positive sample pairs, it synchronously considers the two-way association strengths from time series to text and from text to time series, constructs a soft contrastive target through the normalized exponential function, effectively alleviates the embedding space distortion problem caused by the differences in the richness of well logging data dimensions and the sparsity of geological report texts, and retains the modality-interweight distribution information when the generated gradient signals are transmitted to the global model aggregation module through the encrypted channel to guide the collaborative optimization of the federated model.

6. The multimodal large model federated training middle platform and heterogeneous data alignment method according to claim 1, characterized in that, The global model aggregation module integrates a momentum acceleration mechanism when implementing the federated averaging algorithm. By maintaining the parameter update direction in the form of a moving average, it smooths the random perturbations in the local node gradient update process. When aggregating the obfuscated gradients uploaded by each participating node, it combines the exponential decay average of the historical update vectors. This mechanism can not only accelerate the convergence speed of the cross-enterprise joint training process but also effectively suppress the parameter fluctuations caused by privacy protection noise injection. The globally model parameters optimized by momentum need to pass an integrity check before being distributed to each participating node to ensure the effectiveness of model updates.

7. The multi-modal large model federated training middleware platform and heterogeneous data alignment method according to claim 6, characterized in that, The privacy protection verification module adopts a composite noise injection strategy, simultaneously applying a distribution noise with heavy-tailed characteristics and a Gaussian noise that follows the exponential decay law during the gradient obfuscation stage. Among them, the heavy-tailed noise provides a basic privacy protection layer against gradient inversion attacks, and the Gaussian noise is used to fill the correlation loopholes between feature dimensions. The two noises are coupled in the form of a Hadamard product and then applied to the model parameter update amount. The mixed noise mechanism establishes a multi-level privacy protection system on the premise of ensuring model usability. When the processed obfuscated gradient signal is transmitted back to each participating node through a secure protocol, it carries noise distribution feature metadata for local model recovery.

8. The multi-modal large model federated training middle platform and heterogeneous data alignment method according to claim 1, characterized in that The local federated learning module implements an encryption transmission mechanism based on random linear transformation. Before uploading the temporal embedding vector and the text embedding vector to the central coordination node, it performs an invertible linear transformation on the embedding vector using a dynamically generated random orthogonal matrix and simultaneously superimposes a random bias vector that conforms to a specific statistical distribution. This encryption process hides the numerical features of the original embedding vector while maintaining the relative distance relationship between vectors, ensuring that the contrastive learning alignment module can correctly calculate the cross-modal sample similarity. The central coordination node can only operate on the encrypted feature space and cannot parse the semantic information of the original geological data.

9. The multimodal large model federated training middle platform and heterogeneous data alignment method according to claim 1, characterized in that The text processing module integrates an adversarial training framework in the term standardization stage. By constructing an adversarial game mechanism between a term discriminator and a noise generator, it improves the robustness of the model. The noise generator is responsible for constructing obfuscated words that are morphologically similar to real geological terms, and the term discriminator learns to distinguish real terms from the generated noise. During this process, the text encoder is forced to strengthen its semantic representation ability for key words in lithology descriptions, enabling the generated text feature vectors to effectively filter out the expression ambiguities and annotation noises in well logging interpretation reports and improving the mapping accuracy of professional terms and geophysical exploration features in the cross-modal alignment process.

10. The multimodal large model federated training middle platform and heterogeneous data alignment method according to claim 1, characterized in that, The BERT model (an open-source machine learning framework) of the embedding generation module adopts a context-aware dynamic masking pre-training strategy. During the model pre-training stage, it dynamically adjusts the masking probability according to the part-of-speech tagging and document frequency of terms, and applies a high probability mask to the core terms characterizing the formation lithology and reservoir physical properties to strengthen the model's context reasoning ability. When processing geological report texts, it captures the dependency relationship between the masked positions and the surrounding descriptive statements through the self-attention mechanism. The generated text embedding vectors carry implicit professional semantic association information, and can establish a fine-grained mapping relationship between the physical characteristics of logging curves and the professional descriptions of geological texts when performing cross-modal alignment with the temporal embedding vectors.

Images