Contract federated learning computing method and system based on multi-modal fusion

Abstract

The present application relates to the technical field of enterprise management system and artificial intelligence cross, and specifically provides a multi-modal fusion contract federated learning calculation method and system, comprising: each participant obtains original contract data locally, extracts table features and text features from structured table data and unstructured text data respectively, and dynamically fuses into fusion features of a unified feature space; the sensitivity score of the fusion features is calculated, the features are divided into core feature blocks, statistical feature blocks and metadata blocks according to the score, and homomorphic encryption, differential privacy noise and hash signature processing are used respectively to generate encrypted knowledge units; each participant trains a machine learning model based on the encrypted knowledge units locally, uploads the generated model parameter update value to the aggregation layer after privacy protection processing, and updates the global model by the aggregation layer. The present application realizes cross-domain collaborative analysis and privacy protection of heterogeneous ERP system contract data.

Description

Technical Field

[0001] This invention belongs to the field of interdisciplinary technology of enterprise management systems and artificial intelligence, specifically relating to a multimodal fusion contract federated learning computing method and system. Background Technology

[0002] With the development of enterprise groups, it has become common for multiple organizations to use heterogeneous ERP systems to manage contract data. Current mainstream solutions attempt to use federated learning technology to avoid the direct aggregation of raw data through model parameter interaction.

[0003] However, the field naming and structure vary greatly among different ERP systems, and federated learning technology directly aggregates model parameter updates provided by the parameter provider, leading to disordered feature dimensions; moreover, when participating parties aggregate the original contract data for local training, there is a risk of privacy leakage, which violates the principle of data minimization. Summary of the Invention

[0004] To address the aforementioned shortcomings of existing technologies, this invention provides a multimodal fusion-based contract federated learning computation method and system to solve the aforementioned technical problems.

[0005] In a first aspect, the present invention provides a multimodal fusion contract federated learning computation method, comprising: Each participating party obtains the original contract data locally, extracts corresponding features from the various formats of the original contract data, and merges all the extracted features into a unified feature space. The fused features are subjected to sensitivity scoring, and the fused features are divided into core feature blocks, statistical feature blocks, and metadata blocks based on the sensitivity scores; Homomorphic encryption is applied to the core feature block, differential privacy noise is added to the statistical feature block, and hash signature is performed on the metadata block to generate encrypted knowledge units; Each participant trains a machine learning model based on locally generated encrypted knowledge units, generates model parameter update values, performs privacy protection processing on the model parameter update values, and then uploads them to the aggregation layer so that the aggregation layer can update the global model.

[0006] In an optional implementation, the original contract data includes structured tabular data and unstructured text data; features are extracted from the various formats of the original contract data, and all extracted features are fused into a unified feature space, including: The structured tabular data is subjected to time-series feature capture, numerical augmentation, and tabular semantic modeling to extract tabular features; The unstructured text data is segmented, semantically encoded, and key clauses are located to extract text features; After mapping the table features and the text features to the same dimension through a fully connected layer, they are dynamically weighted and fused to generate the fused features.

[0007] In one optional implementation, the dynamic weighted fusion includes: The normalized information entropy is calculated for continuous fields in the structured tabular data, the category distribution entropy is calculated for discrete fields, and the entropy values ​​of all fields are weighted, averaged, and normalized to obtain the table quality score. The perplexity of the unstructured text data is calculated using a pre-trained language model, and a text quality score is calculated based on the perplexity. Based on the contract type and the table quality score, the table modal weights and text modal weights are calculated using the Logistic function; The table features and text features are weighted and fused according to the table modality weight and the text modality weight.

[0008] In an optional implementation, the contract type is mapped to a low-dimensional dense vector through a learnable embedding layer; the input to the Logistic function is a combination of the normalized value of the table quality score and the contract type embedding vector, and the parameters of the Logistic function are jointly trained and updated during the federated learning process as part of the local model.

[0009] In an optional implementation, the table semantic modeling includes: converting the structured table into a graph network, using contract entities as nodes and field relationships as edges, and extracting topological features through a graph convolutional network; the key clause localization is based on a multi-head attention mechanism to identify risk-sensitive clauses and output clause type labels.

[0010] In an optional implementation, sensitivity scoring of the fused features includes: For each feature field in the fusion features, first determine whether the field belongs to a predefined set of sensitive fields; if it belongs to the set of sensitive fields, then the business importance weight corresponding to the field is preset as the sensitivity score of the field; if it does not belong to the set of sensitive fields, then its sensitivity score is zero. The set of sensitive fields is predefined based on business rules and includes at least the amount field, the ID number field, and the bank account number field.

[0011] In an optional implementation, the fused features are divided into core feature blocks, statistical feature blocks, and metadata blocks based on sensitivity scores, including: The features are sorted in descending order of sensitivity score, and the top k features whose cumulative sensitivity percentage reaches a preset threshold are included in the core feature block. The remaining features are assigned to the statistical feature block, and differential privacy noise is added after calculating their mean and variance. The sample number, feature dimensions, and version number are included in the metadata block.

[0012] In one optional implementation, when each participant trains a machine learning model based on locally generated encrypted knowledge units, each feature block in the encrypted knowledge unit is decrypted locally and merged into a complete fused feature tensor, which serves as the input to the machine learning model; the machine learning model is a neural network model.

[0013] In an optional implementation, after receiving the privacy-preserving updated model parameters, the aggregation layer performs malicious update filtering on the received updates. The malicious update filtering method includes: Select a representative subspace vector from the model parameter update values ​​of each participant; Calculate the cosine similarity between the representative subspace vectors of each participant; For each participant, calculate its average similarity to all other participants; Calculate the median m of the average similarity among all participants and the median absolute deviation MAD; If the average similarity of a participant is less than m-β×MAD, then the participant is determined to be a malicious node and its updated value is removed, where β is a preset coefficient.

[0014] Secondly, the present invention provides a multimodal fusion contract federated learning computing system, comprising: The feature extraction module is used by each participating party to obtain the original contract data, extract corresponding features from the various formats of the original contract data, and merge all the extracted features into a fusion feature in a unified feature space. The feature segmentation module is used to perform sensitivity scoring on the fused features and divide the fused features into core feature blocks, statistical feature blocks, and metadata blocks based on the sensitivity scores. The feature encryption module is used to apply homomorphic encryption to the core feature block, add differential privacy noise to the statistical feature block, perform hash signature on the metadata block, and generate encrypted knowledge units. The model training module is used by each participant to train a machine learning model based on locally generated encrypted knowledge units, generate model parameter update values, and upload the model parameter update values ​​to the aggregation layer after privacy protection processing so that the aggregation layer can update the global model.

[0015] The beneficial effects of this invention are as follows: The multimodal fusion contract federated learning computation method and system provided by this invention, through local knowledge extraction and differentiated encryption strategies, ensures that the original contract data never leaves the local machine, effectively avoiding GDPR compliance risks and memory attack leakage hazards; dynamically allocates multimodal feature weights based on contract type and data quality, solving the problem of joint analysis failure caused by the disorder of field dimensions in heterogeneous ERP systems; and ensures the convergence stability of the global model by timely removing malicious node updates. This invention synergistically optimizes privacy protection, feature depth, and system security, providing reliable technical support for contract risk analysis and compliance review for multi-organizational enterprise groups under the premise that data does not leave the domain. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is a schematic flowchart of a method according to an embodiment of the present invention.

[0018] Figure 2 This is another illustrative flowchart of a method according to an embodiment of the present invention.

[0019] Figure 3 This is a schematic block diagram of a system according to an embodiment of the present invention. Detailed Implementation

[0020] To enable those skilled in the art to better understand the technical solutions of this invention, the technical solutions of the embodiments of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this invention, and not all embodiments. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this invention.

[0021] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein in the description of the invention is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention.

[0022] The multimodal fusion contract federated learning computation method provided in this embodiment of the invention is executed by a computer device, and correspondingly, the multimodal fusion contract federated learning computation system runs on the computer device.

[0023] Figure 1 This is a schematic flowchart illustrating a method according to an embodiment of the present invention. Wherein, Figure 1 The executing entity can be a multimodal fusion contract-based federated learning computing system. Depending on different requirements, the order of steps in this flowchart can be changed, and some can be omitted.

[0024] like Figure 1 As shown, the method includes: S1. Each participating party obtains the original contract data locally, extracts corresponding features from the various formats of the original contract data, and merges all the extracted features into a fused feature in a unified feature space; S2. Perform sensitivity scoring on the fused features, and divide the fused features into core feature blocks, statistical feature blocks, and metadata blocks based on the sensitivity scores; S3. Homomorphic encryption is applied to the core feature block, differential privacy noise is added to the statistical feature block, and hash signature is performed on the metadata block to generate encrypted knowledge units; S4. Each participant trains a machine learning model based on locally generated encrypted knowledge units, generates model parameter update values, performs privacy protection processing on the model parameter update values, and uploads them to the aggregation layer so that the aggregation layer updates the global model.

[0025] Please refer to Figure 2 The steps of the method provided by the present invention will be described in detail below.

[0026] In one embodiment of the present invention, based on step S1, the following will provide a possible embodiment and describe its specific implementation in a non-limiting manner.

[0027] The original contract data includes two modalities: structured tabular data and unstructured text data. Features are extracted from each modality through different processing channels, and finally fused into a feature vector of a unified dimension. The following detailed explanation of each sub-step is provided with reference to a specific implementation example.

[0028] S101. Processing of Structured Tabular Data For time-series fields (such as installment payment dates, delivery milestone dates, etc.) in structured tabular data, this embodiment uses a Bidirectional Long Short-Term Memory (BiLSTM) network to capture temporal features. Specifically, the time-series fields in the contract table are arranged in chronological order to construct a time-series vector. Taking the installment payment plan of a procurement contract as an example, the table contains fields such as payment date, payment amount, and payment percentage, arranged in chronological order as a sequence of length T. This sequence is input into the BiLSTM network. The BiLSTM captures past and future information of the sequence through two LSTM layers (forward and backward) to fully understand the time dependencies of the payment plan. The specific parameters of the BiLSTM network are set as follows: hidden layer dimension 64, number of layers 2, and dropout rate 0.2. After processing by BiLSTM, a fixed-length temporal feature vector is output, which encodes payment pattern information throughout the payment cycle, such as payment frequency, amount fluctuation range, and final payment concentration.

[0029] For core numerical fields such as amount and quantity, this embodiment performs standardization. For categorical fields (such as payment method, currency type, contract status, etc.), one-hot encoding is used for discretization.

[0030] This embodiment transforms structured tables into graph networks for semantic modeling. Specifically, contract entities in the table (such as buyers, sellers, guarantors, etc.) are used as nodes in the graph network, and the relationships between fields are used as edges. After constructing the graph network, a Graph Convolutional Network (GCN) is used to encode the graph structure. The GCN aggregates the feature information of each node and its neighboring nodes through multiple layers of graph convolution operations, outputting the embedded representation of each node. The embedded representations of each node are concatenated and mapped to a fixed-dimensional table semantic feature vector through a fully connected layer.

[0031] S102. Processing of Unstructured Text Data For unstructured text content such as contract terms and technical attachments, this embodiment first performs context-aware word segmentation. Then, a domain-specific dictionary, containing a legal terminology database, is loaded.

[0032] During word segmentation, a dictionary-based maximum matching algorithm is used to prioritize the identification and preservation of the integrity of specialized terms. For rare words not included in the dictionary, a statistical-based segmentation method is used for supplementary segmentation.

[0033] After word segmentation, the text undergoes hierarchical semantic encoding. First, character-level encoding is performed, mapping each character to a fixed-dimensional embedding vector to address the recognition of uncommon words. Character-level encoding uses a pre-trained character vector table with a dimension of 128.

[0034] The character-level encoded sequence is then fed into a BiLSTM encoding layer to obtain a contextual semantic representation. The output of BiLSTM is a context-aware vector for each position, which integrates the semantic information before and after the character.

[0035] After obtaining the contextual semantic representation of the text, risk-sensitive clauses are identified based on a multi-head attention mechanism. This embodiment adopts the multi-head self-attention mechanism in the Transformer architecture, setting the number of attention heads to 8, with each head having a dimension of 64.

[0036] Multi-head attention mechanisms employ multiple parallel attention heads to focus on different semantic subspaces within the text. For example, one head might focus on statements related to monetary amounts, another on statements related to time, and a third on statements related to the responsible party. The outputs of each head are concatenated and mapped through a fully connected layer to output the probability distribution of each text fragment belonging to different clause types.

[0037] The types of clauses include, but are not limited to: disclaimers, limitations of liability, confidentiality clauses, breach of contract clauses, jurisdiction clauses, payment clauses, and termination clauses. The clause type with the highest probability is output as the tag for that segment, along with the start and end positions of that clause in the original text, achieving precise clause location.

[0038] S103. Feature Dimension Alignment and Dynamic Weighted Fusion Since the original dimensions of the table feature vector and the text feature vector are different (the table feature dimension is, for example, 256 dimensions, and the text feature dimension is, for example, 384 dimensions), this embodiment maps them to the same dimension D through two independent fully connected layers.

[0039] Specifically, the target dimension D is set to 128. Table feature vectors. via fully connected layer Mapped to ,in The weight matrix is ​​128×256. This is a 128-dimensional bias vector. (Text feature vector) via fully connected layer Mapped to ,in The weight matrix is ​​128×384. It is a 128-dimensional bias vector. The ReLU activation function is max(0,x), used to introduce nonlinearity.

[0040] After dimensional alignment, the table features and text features are dynamically weighted and fused. The fusion weights are dynamically calculated based on the contract type and feature quality.

[0041] Specifically, for each contract, a table quality score and a text quality score are first calculated. The table quality score is calculated based on the information entropy of the table fields: for continuous fields, the values ​​are divided into k intervals (k is determined by the Sturges formula), and the probability of each interval is calculated. Entropy The entropy values ​​of each field are weighted, averaged, and normalized to a score of 0-100. The text quality score is determined based on the perplexity (PPL) calculated by the pre-trained language model: TextQuality=100 / (1+log(PPL)). The lower the PPL, the higher the coherence and the higher the score.

[0042] Contract types are mapped to low-dimensional dense vectors through a learnable embedding layer. Suppose there are K types of contract types (such as purchase contracts, sales contracts, service agreements, lease contracts, etc.), and each type is mapped to a d-dimensional vector (d=16). This embedding layer constitutes a K×d parameter matrix E.

[0043] Normalized values ​​of the table quality scores Embedded vectors of contract types Concatenate into input vector After linear transformation Then input the Logistic function (S-shaped function): , Where w is the learnable weight vector and b is the bias.

[0044] The final fusion features are: This fusion feature simultaneously incorporates the structured information of the table and the semantic information of the text, and its weights are dynamically adjusted based on contract type and data quality.

[0045] All parameters of the above fusion layer (W) tab W txt (w, b, E) are all part of the local model and are trained and updated together with the parties during the federated learning process.

[0046] In one embodiment of the present invention, based on step S2, the following will provide a possible embodiment and describe its specific implementation in a non-limiting manner.

[0047] S201. Perform sensitivity scoring on the fused features, including: Let the fused feature vector be ,in For the dimensions of the fusion features (in this embodiment) For the first... Feature fields ( ), its sensitivity score Determined according to the following rules:

[0048] in, For a predefined set of sensitive fields, For fields The importance of the business is preset with weights. .

[0049] The above formula means that for each feature field in the fused features, it is first determined whether the field belongs to the sensitive field set. If it falls under this category, then the business importance corresponding to that field will be assigned a pre-defined weight. This serves as its sensitivity score; if it does not belong to the category, the sensitivity score is zero.

[0050] Sensitive Field Set The sensitive field set is predefined during system deployment based on business rules. In this embodiment, the sensitive field set includes at least the following three types of fields: Amount fields: Total contract amount, installment payment amount, penalty amount, security deposit amount, etc. Identification fields include: supplier's unified social credit code, legal representative's ID number, bank account number, etc. Fields classified as trade secrets include: discount rate, internal cost price, and profit margin.

[0051] In actual deployment, the sensitive field set can be configured and expanded by each participant according to its own business needs and compliance requirements. The sensitive field set of all participants is configured before federated learning starts, and the configuration information is only stored locally.

[0052] For each field in the sensitive field set, its business importance weight is pre-set. The weight settings meet the requirements. A higher value indicates a more sensitive field. This example provides an exemplary weight configuration: For monetary fields, the total contract amount, as core business data, has the most serious consequences if leaked, so its weight is set to 1.0; installment amounts, although also monetary information, are slightly less sensitive than the total contract amount, so its weight is set to 0.9. This is because installment amounts, combined with payment cycles, can infer business volume and still have high privacy protection requirements. For identity fields, ID card numbers are sensitive personal information, involving high compliance requirements of regulations such as GDPR, so their weight is set to 1.0; bank account numbers are financially sensitive information, so their weight is set to 0.9. For trade secret fields, internal discount rates are core business strategies, and their leakage will seriously affect the company's bargaining power, so their weight is set to 1.0; profit margins reflect the company's operating conditions and, although also trade secrets, are slightly less sensitive than discount rates, so their weight is set to 0.8.

[0053] For all After performing the above judgment and assignment operations on each feature field one by one, the complete sensitivity score vector is obtained:

[0054] This rating vector With fused feature vectors Equal length, with each component taking values ​​within the following ranges. The score vector is calculated and stored locally and does not participate in federated transmission.

[0055] S202. Based on the sensitivity score, the fused features are divided into core feature blocks, statistical feature blocks, and metadata blocks, including: Obtain the sensitivity score vector After that, all The features are sorted in descending order of sensitivity score. Let the sorted feature index sequence be denoted as . ,satisfy:

[0056] For features with the same sensitivity score, they are arranged in the order of feature index.

[0057] For the sorted... Features (corresponding to the original index) ), its sensitivity percentage Defined as the proportion of the sensitivity score of this feature in the sum of all feature sensitivity scores:

[0058] The denominator is the sum of the sensitivity scores of all features. When none of the features belong to the sensitive field set, the denominator is zero, indicating that there are no sensitive features and all features are directly included in the statistical feature block.

[0059] After arrangement Cumulative sensitivity percentage of each location for:

[0060] Cumulative percentage Starting with the first feature, increments are made one by one, with values ​​ranging from... Monotonically increasing to .

[0061] Set preset threshold (In this embodiment) The core feature block consists of the preceding features that satisfy the following conditions. Composed of several features:

[0062] That is, starting from the first feature, the sensitivity percentage is accumulated, and when the accumulated percentage first reaches or exceeds the threshold... Stop at time, and move forward Each feature is assigned to the core feature block.

[0063] Core feature blocks Represented as:

[0064] in Represents the fused feature vector The Each component.

[0065] The core feature block contains the most sensitive subset of features in the entire fused feature set. If these features are leaked, they will cause the most serious privacy consequences. Therefore, they require the highest level of security protection (homomorphic encryption).

[0066] All remaining features (i.e., indexed as) The characteristics) are included in the statistical feature block. :

[0067] The number of features in a statistical feature block is .

[0068] For each statistical feature block, calculate its mean and variance. Let the value of each feature in the statistical feature block be denoted as _____. ,but: Mean:

[0069] variance:

[0070] Subsequently, Laplace noise was added to both the mean and variance to achieve differential privacy protection. The statistic after adding noise is:

[0071] in This indicates a mean of 0 and a scale parameter of . The Laplace distribution, For global sensitivity, For privacy budget. Because this system standardizes features before encryption (mean 0, variance 1), global sensitivity is low. Set to a fixed value of 2, privacy budget .

[0072] Metadata blocks Includes non-sensitive information that does not involve privacy:

[0073] in: The total number of contract samples used by this participant for training; : Total number of dimensions of fused features (in this embodiment) ); Version identifier of the local knowledge extraction module; : The generation timestamp of the encrypted knowledge unit.

[0074] The aforementioned metadata does not contain any contractual business data and will not cause privacy leaks.

[0075] Based on the above division, a sample contract... The dimensional fusion feature is split into three data blocks:

[0076] in For the front A highly sensitive feature, For the rest Statistics of each feature (mean and variance after adding noise). This is non-sensitive metadata.

[0077] In one embodiment of the present invention, based on step S3, the following will provide a possible embodiment and describe its specific implementation in a non-limiting manner.

[0078] In traditional federated learning paradigms, although the raw data does not leave the domain, it resides in plaintext on computing resources such as GPU memory and CPU memory during local model training. This stage presents a serious security blind spot: if a participant's local server is infected with an Advanced Persistent Threat (APT) Trojan, suffers a memory snapshot attack, or internal operations personnel maliciously export process memory, the original contract data (including sensitive information such as contract amount, ID number, and bank account number) will be directly leaked.

[0079] The encrypted knowledge unit generated in this step is essentially a secure data form of the original contract data after feature-level anonymization and encryption. This encrypted knowledge unit replaces the original contract data as the training input for the local machine learning model, ensuring that: during forward and backward propagation, all computational operations are performed on the encrypted / anonymized features; even if an attacker exports local memory or intercepts process data during training, they will only obtain ciphertext features or noisy statistics, and will not be able to reverse-engineer the original contract information; the original contract data only exists in plaintext form briefly in memory during the knowledge extraction phase, and is immediately released once extraction is complete, significantly reducing the time window for data exposure.

[0080] S301. Core Feature Block Includes the most sensitive front One feature field (in this embodiment) The characteristics are determined by a cumulative sensitivity ratio of 70%, including contract amount, ID number, and discount rate. This embodiment uses the Paillier homomorphic encryption algorithm for encryption.

[0081] Paillier encryption is an additive homomorphic public-key encryption scheme based on the difficult problem of integer factorization. Its core properties are:

[0082] That is, the product of two ciphertexts equals the sum of their corresponding plaintexts. When the forward propagation of the local model involves weighted summation of multiple features, these operations can be performed directly on the Paillier ciphertext, ensuring that the features remain in ciphertext form throughout the entire computation process, and are only decrypted before the final activation function is calculated, thereby minimizing the exposure of the plaintext.

[0083] Paillier encryption uses an asymmetric key system. The key generation process is as follows: Randomly select two independent large prime numbers and ,satisfy .calculate The public key is The private key is The key length is set to 2048 bits. The public key is used for encryption, and the private key is used for decryption. Since the cryptographic knowledge unit is only used locally, the key pairs are generated independently by each participant. The private key does not leave the local machine, and the public key is only used for local encryption operations.

[0084] For each feature value in the core feature block Using public key Perform encryption:

[0085] in For randomly selected integers ( This is used to ensure that the same plaintext produces different ciphertexts each time it is encrypted, satisfying semantic security. Random numbers are introduced. After that, even if two identical feature values ​​are encrypted repeatedly, their ciphertext results will be different, and attackers cannot infer the feature value by comparing the ciphertext.

[0086] After encryption, the core feature block Convert to ciphertext core feature block :

[0087] During the forward propagation of local model training, when features in core feature blocks are involved, the model directly performs linear operations (such as weighted summation) on the ciphertext. Due to Paillier's additive homomorphism, multiplication and addition operations of model weights are compatible with encryption operations. Nonlinear operations during training (such as the ReLU activation function) require local decryption of the ciphertext before performing the nonlinear transformation. The decryption key (private key) is strictly stored in a local secure area, is only accessed during inference, and is immediately erased from memory after execution.

[0088] S302. Statistical Feature Block Features with moderate sensitivity scores (the remaining features that do not reach 70% cumulative sensitivity). This embodiment uses a Laplace mechanism to add differential privacy noise.

[0089] The addition of noise to statistical feature blocks is to defend against parameter reverse engineering attacks after local model training. That is, attackers can reverse engineer the training data features by analyzing the updated model parameters generated during local training.

[0090] In this step, for each feature value in the statistical feature block... Add Laplace noise directly:

[0091] in This indicates a mean of 0 and a scale parameter of . The Laplace distribution, For global sensitivity, Budget for privacy.

[0092] Because this system standardized the features before encryption (mean = 0, variance = 1), the value range of each feature has been unified to a similar scale, thus improving global sensitivity. Set to a fixed value of 2. Privacy Budget This ensures that individual information based on statistical characteristics is effectively protected.

[0093] After adding noise, statistical feature blocks Convert to noisy statistical feature blocks :

[0094] This noisy feature block is directly used as one of the input data for training the local model. During training, even if an attacker reverse-engineers the input features by analyzing the model gradient, they can only obtain an approximation with added Laplacian noise, and cannot reconstruct the true features.

[0095] S303. Metadata Block Includes non-sensitive information that does not involve privacy (sample size) Feature Dimension (Version number, timestamp, etc.). This embodiment uses the national cryptographic SM3 algorithm to generate hash signatures.

[0096] For metadata blocks After serializing its content into a byte string, perform an SM3 hash operation:

[0097] in It is a 256-bit hash value. This hash value is then packaged and stored together with the metadata block in a cryptographic knowledge unit.

[0098] Before each local model training session, the SM3 hash value of the metadata block is recalculated. and with stored Perform a comparison. If... If the result is positive, it proves that the metadata block has not been tampered with since its generation, and training can proceed normally; otherwise, a security alarm will be issued and the training process will be terminated.

[0099] After the above three steps, the three encrypted data blocks are assembled into a complete encrypted knowledge unit according to the predetermined data format. :

[0100] The encrypted knowledge unit is stored in the local secure storage area of ​​the participating party. In one embodiment of the present invention, based on step S4, the following will provide a possible embodiment and describe its specific implementation in a non-limiting manner.

[0101] S401. Each participant trains a machine learning model based on locally generated encrypted knowledge units and generates updated values ​​for model parameters.

[0102] Each participant generated encrypted knowledge units locally. Before starting local model training, the encrypted knowledge units must first be restored to a complete fused feature tensor that can be directly processed by the neural network model.

[0103] At the start of training, the integrity of the metadata block is first verified. The version number in the metadata is also used to verify the version consistency of the local knowledge extraction module. If the version number does not match the current system version, it means that the encrypted knowledge unit was generated by an older version module, and its feature space definition may have changed. In this case, the re-extraction process needs to be triggered to prevent feature misalignment caused by version inconsistency.

[0104] Core feature blocks The stored data is homomorphically encrypted by Paillier. Highly sensitive feature ciphertext Because the encryption process uses locally generated key pairs by each participant, the decryption operation is performed entirely locally, and the private key does not need to leave the domain.

[0105] For each ciphertext Using locally stored Paillier private keys Decryption execution:

[0106] in After decryption, the plaintext feature value is obtained. Restored to core feature blocks :

[0107] The decryption operation is performed in memory, and the plaintext feature values ​​only exist briefly during the forward propagation computation of the current training batch. After each training batch is completed, the plaintext feature values ​​are immediately erased from memory, and the ciphertext data of the corresponding batch is re-decrypted during the next training batch, ensuring that the residence time of the plaintext in memory is minimized.

[0108] Decrypted core feature blocks Noisy statistical feature blocks (Include The individual feature values ​​and information such as the number of samples and feature dimensions in the metadata block are rearranged according to the original feature index order and combined into a complete fused feature tensor. :

[0109] This is the input feature vector used in the local machine learning model. This feature vector has undergone a complete protection cycle of de-identification encryption-decryption: the original data is encrypted after knowledge extraction and decrypted upon training startup, but the restored features are already de-identified fused features, containing no directly identifiable original contract information (absolute amounts have been standardized, text has been vectorized, and sensitive statistics have been superimposed with noise). Even if an attacker obtains this information at this point... It is also impossible to reverse engineer and recover the original sensitive data such as the total contract amount and ID number.

[0110] In actual training, participants typically possess multiple contract samples. Assume there are multiple local contracts. Each of the contract samples is converted into one through the above process. If there are 1 / 2 eigenvectors, then the local training set can be represented as a feature matrix. The corresponding tag set is (such as risk level labels, default probability labels, etc.).

[0111] The structure and training process of a machine learning model include: In this embodiment, the local machine learning model uses a multi-layer neural network, and its structure is as follows: Given a feature vector of an input sample Each layer is calculated sequentially, and the model outputs the predicted risk probability.

[0112] This embodiment uses the cross-entropy loss function to measure the model's prediction. With real labels The difference between them:

[0113] Training employs the mini-batch stochastic gradient descent (SGD) optimization algorithm, with a batch size of 64 and a learning rate of [missing value]. Each round of local training iterates through all local samples. One cycle (epoch).

[0114] In each training batch, the model calculates the gradient of the loss function with respect to each parameter. And update the parameters using the backpropagation algorithm:

[0115] in This is the learning rate.

[0116] After one round of local training, the updated model parameter values ​​are obtained. :

[0117] in These are the initial parameters for the local model (i.e., the global model parameters issued after the previous round of federated aggregation). The updated parameters are obtained after one epoch of local training. This refers to the information that each participating party needs to upload to the aggregation layer.

[0118] S402. After performing privacy protection processing on the updated model parameter values, the data is uploaded to the aggregation layer so that the aggregation layer can update the global model.

[0119] After each participant completes model training locally, they generate updated model parameter values. Before uploading it to the aggregation layer, it needs to undergo privacy protection processing to prevent the aggregation layer or third parties from inferring the participants' local data information by analyzing parameter update values.

[0120] To prevent the gradient of a single sample from dominating the overall update direction (which could lead to privacy leaks or training instability), gradient clipping is first performed on the parameter update values:

[0121] in The cropping threshold (in this embodiment) ), Update the L2 norm of the vector for the parameters. Pruning restricts the norm of the updated vector to a certain value. Within this limit, the maximum impact of a single sample on the overall update is restricted, which is a prerequisite for the subsequent addition of differential privacy noise.

[0122] Based on gradient clipping, Gaussian noise is added to the clipped parameter update values ​​to achieve differential privacy protection.

[0123] in The noise is Gaussian noise that is independent in each dimension. Noise scale parameter (in this embodiment) ), The cropping threshold, It is an identity matrix.

[0124] For scenarios with higher security requirements, public key pairs from the aggregation layer can be further used on top of differential privacy protection. The data is uploaded after homomorphic encryption. The aggregation layer performs the aggregation operation in ciphertext before decryption, ensuring that the aggregation layer cannot access the plaintext parameter update values ​​of any participant throughout the process. This embodiment employs a lightweight additive homomorphic encryption scheme, encrypting only the gradient vector as a whole, resulting in controllable computational overhead.

[0125] Based on the above embodiments, in order to further improve the reliability of federated learning provided by the above embodiments, in one embodiment, a data processing method for the aggregation layer is provided, including: Each participating party will update the model parameters after privacy protection processing. Upload the following metadata to the aggregation layer: unique identifier of the participant (after anonymization); number of local samples. (For weight calculation during aggregation); local model training loss value (for evaluating training quality); timestamp and epoch number. Uploads use secure communication protocols (such as TLS 1.3 encrypted channels) to prevent parameters from being intercepted during transmission.

[0126] After receiving the uploaded data from each participant, the aggregation layer performs federated average aggregation:

[0127] in The total number of participants. As a participant The number of local samples, This represents the total number of global samples. This weighted averaging strategy ensures that participants with larger datasets contribute more significantly to the global model update.

[0128] For parameter updates that have undergone homomorphic encryption, the aggregation layer calculates a weighted sum in the ciphertext state, obtains the ciphertext aggregation result, and then decrypts it using its own private key. .

[0129] The aggregation layer updates the global value obtained from aggregation. Update the global model:

[0130] in The global learning rate (in this embodiment) ), These are the global model parameters for the current round. To prevent model divergence, the global learning rate is usually set to a positive number less than 1, which helps to smooth the updates.

[0131] Updated global model The parameters are distributed to all participants by the aggregation layer, and each participant uses them as the initial model parameters for the next round of local training. This process is repeated until the global model converges.

[0132] During the aggregation process, the aggregation layer simultaneously performs malicious update filtering. Specifically, the aggregation layer collects data from all participating parties. Next, the directional similarity matrix (based on cosine similarity) among the participants is calculated, and abnormal update values ​​that deviate significantly from the mainstream update direction are identified and removed using the MAD dynamic thresholding method. This mechanism effectively defends against malicious participants' attempts to disrupt the global model through poisoning attacks. For details on the implementation, please refer to the description of malicious update filtering in step S50.

[0133] In some embodiments, the multimodal fusion contract federated learning computing system may include multiple functional modules composed of computer program segments. The computer programs of each program segment in the multimodal fusion contract federated learning computing system may be stored in the memory of a computer device and executed by at least one processor to perform (see details). Figure 1 (Description) Functionality of contract-based federated learning computation with multimodal fusion.

[0134] In this embodiment, the multimodal fusion contract federated learning computing system can be divided into multiple functional modules based on its functions, such as... Figure 3 As shown. The module referred to in this invention is a series of computer program segments that can be executed by at least one processor and perform a fixed function, and is stored in memory. In this embodiment, the functions of each module will be described in detail in subsequent embodiments.

[0135] The feature extraction module is used by each participating party to obtain the original contract data, extract corresponding features from the various formats of the original contract data, and merge all the extracted features into a fusion feature in a unified feature space. The feature segmentation module is used to perform sensitivity scoring on the fused features and divide the fused features into core feature blocks, statistical feature blocks, and metadata blocks based on the sensitivity scores. The feature encryption module is used to apply homomorphic encryption to the core feature block, add differential privacy noise to the statistical feature block, perform hash signature on the metadata block, and generate encrypted knowledge units. The model training module is used by each participant to train a machine learning model based on locally generated encrypted knowledge units, generate model parameter update values, and upload the model parameter update values ​​to the aggregation layer after privacy protection processing so that the aggregation layer can update the global model.

[0136] Those skilled in the art will clearly understand that the techniques in the embodiments of the present invention can be implemented using software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solutions in the embodiments of the present invention, or the parts that contribute to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium such as a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, or any other medium capable of storing program code. It includes several instructions to cause a computer device (which may be a personal computer, a server, or a second device, network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention.

[0137] The same or similar parts between the various embodiments in this specification can be referred to mutually. In particular, the device embodiments are basically similar to the method embodiments, so the description is relatively simple, and the relevant parts can be referred to the description in the method embodiments.

[0138] In the embodiments provided by this invention, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the system embodiments described above are merely illustrative. For instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between systems or modules may be electrical, mechanical, or other forms.

[0139] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules; that is, they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0140] In addition, the functional modules in the various embodiments of the present invention can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module.

[0141] Although the present invention has been described in detail with reference to the accompanying drawings and preferred embodiments, the present invention is not limited thereto. Various equivalent modifications or substitutions can be made to the embodiments of the present invention by those skilled in the art without departing from the spirit and essence of the invention, and such modifications or substitutions should all be within the scope of the present invention. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should also be covered within the protection scope of the present invention.

Claims

1. A multimodal fusion contract federated learning computation method, characterized in that, include: Each participating party obtains the original contract data locally, extracts corresponding features from the various formats of the original contract data, and merges all the extracted features into a unified feature space. The fused features are subjected to sensitivity scoring, and the fused features are divided into core feature blocks, statistical feature blocks, and metadata blocks based on the sensitivity scores; Homomorphic encryption is applied to the core feature block, differential privacy noise is added to the statistical feature block, and hash signature is performed on the metadata block to generate encrypted knowledge units; Each participant trains a machine learning model based on locally generated encrypted knowledge units, generates model parameter update values, performs privacy protection processing on the model parameter update values, and then uploads them to the aggregation layer so that the aggregation layer can update the global model.

2. The method according to claim 1, characterized in that, The original contract data includes structured tabular data and unstructured text data; features are extracted from the various formats of the original contract data, and all extracted features are merged into a unified feature space, including: The structured tabular data is subjected to time-series feature capture, numerical augmentation, and tabular semantic modeling to extract tabular features; The unstructured text data is segmented, semantically encoded, and key clauses are located to extract text features; After mapping the table features and the text features to the same dimension through a fully connected layer, they are dynamically weighted and fused to generate the fused features.

3. The method according to claim 2, characterized in that, The dynamic weighted fusion includes: The normalized information entropy is calculated for continuous fields in the structured tabular data, the category distribution entropy is calculated for discrete fields, and the entropy values ​​of all fields are weighted, averaged, and normalized to obtain the table quality score. The perplexity of the unstructured text data is calculated using a pre-trained language model, and a text quality score is calculated based on the perplexity. Based on the contract type and the table quality score, the table modal weights and text modal weights are calculated using the Logistic function; The table features and text features are weighted and fused according to the table modality weight and the text modality weight.

4. The method according to claim 3, characterized in that, The contract type is mapped to a low-dimensional dense vector through a learnable embedding layer; the input to the Logistic function is a combination of the normalized value of the table quality score and the contract type embedding vector, and the parameters of the Logistic function are jointly trained and updated as part of the local model during the federated learning process.

5. The method according to claim 2, characterized in that, The semantic modeling of the table includes: transforming the structured table into a graph network, using contract entities as nodes and field relationships as edges, and extracting topological features through a graph convolutional network; the key clause localization is based on a multi-head attention mechanism to identify risk-sensitive clauses and output clause type labels.

6. The method according to claim 1, characterized in that, Sensitivity scoring is performed on the fused features, including: For each feature field in the fusion features, first determine whether the field belongs to a predefined set of sensitive fields; if it belongs to the set of sensitive fields, then the business importance weight corresponding to the field is preset as the sensitivity score of the field; if it does not belong to the set of sensitive fields, then its sensitivity score is zero. The set of sensitive fields is predefined based on business rules and includes at least an amount field, an ID number field, and a bank account number field.

7. The method according to claim 1, characterized in that, Based on the sensitivity score, the fused features are divided into core feature blocks, statistical feature blocks, and metadata blocks, including: The features are sorted in descending order of sensitivity score, and the top k features whose cumulative sensitivity percentage reaches a preset threshold are included in the core feature block. The remaining features are assigned to the statistical feature block, and differential privacy noise is added after calculating their mean and variance. The sample number, feature dimensions, and version number are included in the metadata block.

8. The method according to claim 1, characterized in that, When each participant trains a machine learning model based on locally generated encrypted knowledge units, each feature block in the encrypted knowledge unit is decrypted locally and then merged into a complete fused feature tensor, which serves as the input to the machine learning model; the machine learning model is a neural network model.

9. The method according to claim 1, characterized in that, After receiving the privacy-preserving updated model parameters, the aggregation layer performs malicious update filtering on the received updates. The methods for malicious update filtering include: Select a representative subspace vector from the model parameter update values ​​of each participant; Calculate the cosine similarity between the representative subspace vectors of each participant; For each participant, calculate its average similarity to all other participants; Calculate the median m of the average similarity among all participants and the median absolute deviation MAD; If the average similarity of a participant is less than m-β×MAD, then the participant is determined to be a malicious node and its updated value is removed, where β is a preset coefficient.

10. A multimodal fusion contract-based federated learning computing system, characterized in that, include: The feature extraction module is used by each participating party to obtain the original contract data, extract corresponding features from the various formats of the original contract data, and merge all the extracted features into a fusion feature in a unified feature space. The feature segmentation module is used to perform sensitivity scoring on the fused features and divide the fused features into core feature blocks, statistical feature blocks, and metadata blocks based on the sensitivity scores. The feature encryption module is used to apply homomorphic encryption to the core feature block, add differential privacy noise to the statistical feature block, perform hash signature on the metadata block, and generate encrypted knowledge units. The model training module is used by each participant to train a machine learning model based on locally generated encrypted knowledge units, generate model parameter update values, and upload the model parameter update values ​​to the aggregation layer after privacy protection processing so that the aggregation layer can update the global model.

Images