Differential privacy protection method and system based on multi-modal correlation, computer device and storage medium

Abstract

The application provides a multi-modal correlation-based differential privacy protection method and system, a computer device and a storage medium. The multi-modal correlation-based differential privacy protection method comprises the following steps: acquiring multi-source heterogeneous data, and configuring data labels for all data in the multi-source heterogeneous data, wherein the data labels at least comprise device identifiers and timestamps; filtering a plurality of candidate feature vectors from feature vectors of all data according to the data labels; calculating cross-modal correlation scores of the plurality of candidate feature vectors; for any candidate feature vector, calculating a final privacy budget of the candidate feature vector according to the cross-modal correlation score and a basic privacy budget of the candidate feature vector; and adding noise corresponding to the final privacy budget to the candidate feature vector to obtain a noise vector. The embodiment of the application can guarantee data privacy while preserving the correlation between data, which is helpful to improve the fault diagnosis accuracy of industrial equipment.

Description

Technical Field

[0001] This application relates to the field of privacy protection technology, specifically to a differential privacy protection method, system, computer device, and storage medium based on multimodal correlation. Background Technology

[0002] In industrial equipment fault diagnosis scenarios, differential privacy is often used to protect multimodal operational data and prevent the risk of leakage during data flow. Existing differential privacy methods uniformly set a privacy budget based on global statistical characteristics such as total data volume, distribution, and number of nodes, applying the same level of protection to all data without distinguishing the business value of the data. Fault-related anomaly signals are inherently weak, and uniform noise addition makes them easily masked, resulting in the loss of critical diagnostic information and failing to support accurate fault prediction and root cause analysis.

[0003] Furthermore, existing technologies employ a model of adding noise locally first and then performing correlation analysis, with each entity independently completing privacy processing. This fails to identify the spatiotemporal and semantic relationships between data from different modalities and sources. Industrial fault diagnosis relies on a fault evidence chain formed by multimodal data. Adding noise separately in advance would sever the relationships between data, resulting in an incomplete evidence chain and a loss of the ability to perform multi-data fusion diagnosis. Summary of the Invention

[0004] In view of this, this application proposes a differential privacy protection method, system, computer device and storage medium based on multimodal association, to solve the problem that the prior art cannot preserve the association between data while ensuring privacy.

[0005] The first aspect of this application proposes a differential privacy protection method based on multimodal correlation, the method comprising: Acquire multi-source heterogeneous data and configure data tags for all data in the multi-source heterogeneous data, wherein the data tags include at least a device identifier and a timestamp; Based on the data tag, several candidate feature vectors are selected from the feature vectors of all data. The several candidate feature vectors correspond to the same device identifier, and the difference between the timestamps of any two candidate feature vectors is less than a preset time threshold. Calculate the cross-modal association score of the candidate feature vectors, whereby the cross-modal association score is used to characterize the confidence that the candidate feature vectors jointly point to the same industrial event; For any candidate feature vector, the final privacy budget of the candidate feature vector is calculated based on the cross-modal association score and the basic privacy budget of the candidate feature vector. Add noise corresponding to the final privacy budget to the candidate feature vector to obtain a noise vector.

[0006] This application's embodiments achieve spatiotemporal window filtering of multimodal data through device identifiers and timestamps, accurately aggregating multi-source heterogeneous feature vectors corresponding to the same industrial event. It innovatively introduces cross-modal correlation scores to quantify the event correlation confidence of multimodal data, dynamically and adaptively adjusting the privacy budget of each feature vector based on the correlation scores and completing differential privacy noise addition processing. Compared to existing technologies that use a globally unified privacy budget and equalized noise addition, this solution can differentiate the privacy protection strength according to the event contribution value of the data. It retains higher data fidelity for feature vectors with high correlation and carrying key evidence of faults, while strengthening privacy protection for low-correlation redundant data. This effectively avoids the problem of weak fault signals being masked by uniform noise, maximizing the preservation of key feature information required for industrial fault diagnosis while ensuring data privacy and security.

[0007] In this embodiment of the application, after obtaining the noise vector, the method further includes: For each of the noise vectors that correspond one-to-one with the candidate feature vectors, the entity information corresponding to each noise vector is identified. Candidate triples are obtained by predicting the relationship between the entity information of any two noise vectors; the triples include the entity information of the two noise vectors and the relationship between the entity information. When the cross-modal association score of the aforementioned candidate feature vectors is greater than a preset association score threshold, a corresponding data aggregation event entity is created in the knowledge graph; All candidate triples are obtained through the entity information of the aforementioned noise vectors. The data aggregation event entities are then used as the core for reconstruction and integration to form structured knowledge graph data.

[0008] This application, after differential privacy noise addition, further performs entity recognition, relation extraction, event entity aggregation, and knowledge graph reconstruction on the noise vector, overturning the technical drawbacks of the traditional "noise first, then association" approach that leads to broken associations. By structuring and fusing scattered entity triples around event entities in a secondary manner, the multimodal data associations that were originally fragmented by noise disturbances are re-collected, repaired, and solidified, constructing a complete chain of evidence for fault events. This ensures both the privacy and security of data flow to external parties and fully preserves the spatiotemporal and semantic association characteristics of multimodal data. It solves the problem of incomplete fault evidence chains and inability to support multimodal fusion diagnosis and root cause analysis caused by traditional differential privacy protection, significantly improving the business usability of industrial data after privacy protection.

[0009] In this embodiment of the application, calculating the cross-modal association score of the plurality of candidate feature vectors includes: Determine the modality type corresponding to the plurality of candidate feature vectors, and obtain the preset correlation weights between different modality types; For the combinations of candidate feature vectors belonging to different modal types among the several candidate feature vectors, calculate the feature similarity between each group of candidate feature vectors; Based on the feature similarity and association weights of each group of candidate feature vectors, the association score of each group of candidate feature vectors is calculated. The cross-modal association scores corresponding to all groups are summed to obtain the cross-modal association scores corresponding to the candidate feature vectors.

[0010] This application's embodiments achieve a quantitative assessment of the correlation strength of multi-source heterogeneous data through modal weight configuration, cross-modal feature similarity matching, and hierarchical weighted summation. Compared to traditional methods that rely solely on single-dimensional statistical features to determine data correlation, this solution fully adapts to the multimodal heterogeneous characteristics of industrial data. It can accurately characterize the degree of coupling and correlation between different sensor data, log data, and image data, and quantitatively represent the confidence that multiple data points jointly point to the same fault event. This provides a precise and quantifiable basis for the subsequent dynamic and differentiated allocation of privacy budgets, ensuring the rationality and accuracy of adaptive privacy protection strategies.

[0011] In this embodiment of the application, the final privacy budget of the candidate feature vector is calculated based on the cross-modal association score and the basic privacy budget of the candidate feature vector, including: A pre-defined, monotonically increasing privacy budget mapping function is used, with cross-modal association scores as the sole input variable. The cross-modal association scores corresponding to the candidate feature vectors are input into the privacy budget mapping function to obtain the budget adjustment coefficient; The basic privacy budget of the candidate feature vector is corrected and calculated based on the budget adjustment coefficient to obtain the final privacy budget corresponding to the candidate feature vector.

[0012] This application's embodiments achieve dynamic and adaptive adjustment of the privacy budget by constructing a monotonically increasing mapping function with cross-modal association scores as the sole input. This overcomes the shortcomings of existing differential privacy technologies, which suffer from fixed and rigid privacy budgets and fail to differentiate between business values. As a result, feature vectors with higher association scores and higher fault evidence value ultimately have larger privacy budgets, lower noise levels, and more complete information retention; while irrelevant data with low association levels maintain strong privacy protection. This achieves a dynamic match between privacy protection strength and data business value, accurately balancing the dual needs of privacy security and fault feature retention in industrial scenarios.

[0013] In this embodiment of the application, noise corresponding to the final privacy budget is added to the candidate feature vector to obtain a noise vector, including: Based on the preset data sensitivity and the final privacy budget, the corresponding noise scale parameters are calculated. Random noise is generated based on the noise scale parameters; The random noise is added to the values ​​of each dimension of the candidate feature vector to obtain the perturbed noise vector.

[0014] In this embodiment of the application, the entity information corresponding to each noise vector is identified, including: An industrial domain ontology library containing multiple entity information is preloaded, and a classification model is pre-trained. The entity information includes at least one of the following: equipment, components, fault modes, and operation and maintenance types. The classification model is obtained by training a mixture of standard feature vectors and multiple preset noise vectors. The multiple preset noise vectors are generated by adding random noise of different intensities to the standard feature vectors. The standard feature vectors are extracted based on sample data corresponding to each entity information in the industrial domain ontology library. For any noise vector, the noise vector is input into the classification model, and the probability distribution result of the noise vector in multiple entity information in the industrial domain ontology is output. Based on the probability distribution results, the entity information corresponding to the highest probability is selected as the target entity information for noise vector matching.

[0015] This application employs a robust classification model trained with a mixture of standard feature vectors and noise vectors of varying intensities to achieve accurate entity recognition of vectors perturbed by noise. Unlike traditional models that can only recognize clean features, this model is adaptable to differential privacy perturbation scenarios and possesses extremely strong noise robustness. It can accurately match and infer entities such as equipment, components, fault modes, and maintenance operations even when features are perturbed by noise and some details are distorted.

[0016] In this embodiment of the application, candidate triples are obtained by predicting the relationship between the entity information of any two noise vectors, including: Semantic relationship prediction is performed on the entity information of any two noise vectors to obtain the semantic association between the entity information; Based on the semantic association and the entity information of any two noise vectors, corresponding candidate triples are generated.

[0017] An embodiment of the second aspect of this application provides a differential privacy protection system based on multimodal correlation, comprising: The data acquisition module is used to acquire multi-source heterogeneous data and configure data tags for all data in the multi-source heterogeneous data. The data tags include at least a device identifier and a timestamp. The candidate feature vector filtering module is used to filter out several candidate feature vectors from the feature vectors of all data according to the data label. The several candidate feature vectors correspond to the same device identifier, and the difference between the timestamps of any two candidate feature vectors is less than a preset time threshold. The cross-modal association score calculation module is used to calculate the cross-modal association score of the plurality of candidate feature vectors, wherein the cross-modal association score is used to characterize the confidence that the plurality of candidate feature vectors jointly point to the same industrial event; The final privacy budget calculation module is used to calculate the final privacy budget of any candidate feature vector based on the cross-modal association score and the basic privacy budget of the candidate feature vector. The noise vector generation module is used to add noise corresponding to the final privacy budget to the candidate feature vector to obtain a noise vector.

[0018] An embodiment of the third aspect of this application provides a computer device including a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the computer instructions to perform the differential privacy protection method based on multimodal association described in the first aspect.

[0019] An embodiment of the fourth aspect of this application provides a computer-readable storage medium storing computer instructions for causing a computer to execute the differential privacy protection method based on multimodal association described in the first aspect.

[0020] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0021] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the scope of this application. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings: Figure 1 The illustration shows a flowchart of a differential privacy protection method based on multimodal correlation provided in an embodiment of this application; Figure 2 This illustration shows a schematic diagram of a differential privacy protection system based on multimodal correlation provided in an embodiment of this application; Figure 3 This illustration shows a schematic diagram of the structure of a computer device according to an embodiment of this application; Figure 4 A schematic diagram of a storage medium provided in one embodiment of this application is shown. Detailed Implementation

[0022] Exemplary embodiments of this application will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of this application are shown in the drawings, it should be understood that this application may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided to enable a more thorough understanding of this application and to fully convey the scope of this application to those skilled in the art.

[0023] It should be noted that, unless otherwise stated, the technical or scientific terms used in this application shall have the ordinary meaning as understood by one of ordinary skill in the art to which this application pertains.

[0024] The technical scenarios involved in the embodiments of this application are described below.

[0025] Existing technologies, when performing differential privacy processing, base their privacy budget ε decisions on the macroscopic, global statistical characteristics of the data, specifically including the size of the local data volume, the diversity of data distribution, and the number of nodes participating in the computation. This results in a privacy protection mechanism that is value-indiscriminate and context-independent. The mechanism's "perspective" remains at the surface statistical level of the dataset, treating it as a homogeneous whole, unable to delve into the data's internal structure to perceive and distinguish the specific semantic content and analytical value of individual data points. For this system, a routine data point recording normal device operation with extremely low information content and a weak anomaly signal capturing the eve of a major malfunction are indistinguishable at the privacy protection decision level; they will both be subjected to similarly strong privacy noise determined by the same set of macroscopic indicators.

[0026] Therefore, the existing technology cannot solve the technical problem that this proposal aims to address: "how to accurately retain core diagnostic value information while protecting privacy." Under this "value-neutral" protection model, in order to meet overall privacy and security requirements, the applied noise level will inevitably and easily drown out those crucial pre-fault warning signals with already low signal-to-noise ratios. As a result, while the privacy of the data source is protected, the most valuable analytical information in the data is lost, and the ultimately constructed knowledge graph will lose its ability to perform accurate fault prediction and deep root cause analysis.

[0027] The architecture of existing technologies dictates that privacy processing is performed independently and locally by each participating entity, prior to any cross-entity knowledge discovery or correlation computation. When a participating entity (e.g., factory A) decides how to add noise to its time-series data, it has no way of knowing whether another entity (e.g., factory B) has simultaneously captured a related, corroborating fault image. This results in a correlation-blind privacy processing flow. At the most critical privacy decision-making stage, the system completely severs the inherent semantic and spatiotemporal connections between data from different sources and modalities. It protects isolated data points or model parameters, rather than the "cross-modal event evidence chain" composed of these data points, which possesses a higher information dimension. It abandons the identification and utilization of the correlation value between data at the source.

[0028] Therefore, the existing technology cannot solve the core technical problem of "protecting and utilizing cross-modal correlated knowledge" that this proposal aims to address. Complex fault diagnosis in the industrial field is essentially a reasoning process based on cross-validation of multimodal information. Existing technology indiscriminately weakens each link in the evidence chain at the very first step of data fusion (privacy processing), causing this most valuable correlated knowledge chain to be incomplete or even completely broken before entering the knowledge graph. It protects individual data points but destroys the correlated knowledge between data points, which contradicts the goal of this application to preserve the overall correlated knowledge between data points.

[0029] According to an embodiment of this application, a differential privacy protection method based on multimodal correlation is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0030] This embodiment provides a differential privacy protection method based on multimodal correlation. Figure 1 This is a flowchart of a differential privacy protection method based on multimodal correlation according to an embodiment of this application, such as... Figure 1 As shown, the process includes the following steps: Step S101: Obtain multi-source heterogeneous data and configure data tags for all data in the multi-source heterogeneous data.

[0031] Specifically, multi-source heterogeneous industrial data can be obtained from data sources such as the Industrial Internet of Things, sensor networks, Manufacturing Execution Systems (MES), operation and maintenance log systems, and fault image acquisition devices. This multi-source heterogeneous data includes, but is not limited to, sensor time-series data, equipment structured fault data, operation and maintenance text log data, and fault image data.

[0032] More specifically, each data entry is assigned a unique data tag, which includes at least: a device identifier and a timestamp; the device identifier is used to uniquely identify the industrial device that generated the data entry, and the timestamp is used to mark the time when the data entry was generated.

[0033] In some specific embodiments, after acquiring multi-source heterogeneous data, the method further includes: for data of different modalities in the multi-source heterogeneous data, calling the corresponding neural network model to convert the data into feature vectors of fixed length (e.g., 512 dimensions), for example: using a Temporal Convolutional Network (TCN) to process sensor time-series data segments; using an industrial BERT model to process text descriptions in operation and maintenance logs; and using a ResNet to process images of equipment fault locations.

[0034] In some specific embodiments, after converting the data into feature vectors, the method further includes: A globally preset, low base privacy budget is uniformly assigned to all feature vectors. This value ensures that all data, even isolated data, receives basic privacy protection.

[0035] Step S102: Select several candidate feature vectors from the feature vectors of all data according to the data labels.

[0036] Specifically, the plurality of candidate feature vectors correspond to the same device identifier, and the difference between the timestamps of any two candidate feature vectors is less than a preset time threshold (e.g., 5 minutes).

[0037] More specifically, in the process of screening candidate feature vectors: first, all feature vectors of the data are grouped according to the device identifier, and the device identifiers of all feature vectors in each group correspond to the same industrial equipment; then, within the same device group, the difference between the timestamps of any two feature vectors in the group is judged, and feature vectors with a difference less than a preset time threshold are selected to obtain a set of feature vectors in the same spatiotemporal range; all feature vectors in this set of feature vectors are determined as several candidate feature vectors for this cross-modal correlation analysis.

[0038] Step S103: Calculate the cross-modal correlation score of the candidate feature vectors.

[0039] Specifically, the cross-modal association score is used to characterize the confidence that the several candidate feature vectors jointly point to the same industrial event.

[0040] In some specific embodiments, step S103 above includes steps S1031-S1034: Step S1031: Determine the modality type corresponding to the plurality of candidate feature vectors, and obtain the preset correlation weights between different modality types.

[0041] Specifically, the corresponding modality type can be determined based on the original data source corresponding to each candidate feature vector. The modality type includes at least one of time-series data modality, text data modality, image data modality, and structured fault code modality. More specifically, association weights between different modalities can be pre-configured, such as the association weight between image data modality and text data modality, and the association weight between time-series data modality and text data modality. These association weights characterize the contribution of different modalities of data in jointly forming the chain of evidence for an industrial event.

[0042] Step S1032: For the candidate feature vector combinations belonging to different modal types among the several candidate feature vectors, calculate the feature similarity between each group of candidate feature vectors.

[0043] Specifically, several candidate feature vectors are traversed, and two candidate feature vectors of different modal types are selected and combined. A similarity calculation function is used to calculate the feature similarity of each group of candidate feature vectors in the vector space. The feature similarity is used to reflect the degree of semantic association between different modal data.

[0044] Step S1033: Based on the feature similarity and association weights corresponding to each group of candidate feature vectors, calculate the association score corresponding to each group of candidate feature vectors.

[0045] Specifically, the feature similarity corresponding to each candidate feature vector combination is weighted and calculated with the preset association weight between the modality type corresponding to the combination, and the weighted calculation result is used as the association score corresponding to the candidate feature vector combination.

[0046] Step S1034: Sum the association scores corresponding to all groups to obtain the cross-modal association scores corresponding to the candidate feature vectors.

[0047] Specifically, the association scores corresponding to all candidate feature vector combinations of different modal types are traversed, and all association scores are summed. The sum is used as the cross-modal association score corresponding to several candidate feature vectors. The higher the cross-modal association score, the closer the association between the candidate feature vectors and the higher the confidence that they all point to the same industrial fault or maintenance event.

[0048] In steps S1031-S1034, the cross-modal correlation scores corresponding to several candidate feature vectors can be calculated using the following formula:

[0049] in, This represents the cross-modal association score corresponding to several candidate feature vectors. The higher the value, the greater the value of the aggregate analysis of this set of data; This represents the association weights between different modal types. and These are candidate feature vectors from different modalities among a number of candidate feature vectors; This represents a similarity calculation function (e.g., cosine similarity).

[0050] Step S104: For any candidate feature vector, calculate the final privacy budget of the candidate feature vector based on the cross-modal association score and the basic privacy budget of the candidate feature vector.

[0051] In some specific embodiments, step S104 above includes steps S1041-S1043: Step S1041: Preset a monotonically increasing privacy budget mapping function.

[0052] Step S1042: Input the cross-modal association scores corresponding to the plurality of candidate feature vectors into the privacy budget mapping function to obtain the budget adjustment coefficient.

[0053] In steps S1041-S1042, a monotonically increasing privacy budget mapping function is pre-constructed. The privacy budget mapping function takes as input the cross-modal association scores corresponding to the candidate feature vectors. The output is a budget adjustment factor used to adjust the size of the privacy budget. .

[0054] Step S1043: Based on the budget adjustment coefficient, the basic privacy budget of the candidate feature vector is corrected and calculated to obtain the final privacy budget corresponding to the candidate feature vector.

[0055] Specifically, the final privacy budget corresponding to the candidate feature vector can be calculated using the following formula:

[0056] in, Indicates the final privacy budget, Indicates the basic privacy budget, This represents the budget adjustment coefficient, where the final privacy budget is positively correlated with the cross-modal association score. A higher cross-modal association score results in a larger budget adjustment coefficient, a larger final privacy budget, and a lower corresponding data perturbation intensity. This means that data clusters deemed to collectively constitute a high-value event evidence chain will have each data point within them allocated a higher privacy budget; while isolated data points that do not form effective connections will have a lower privacy budget. level.

[0057] Step S105: Add noise corresponding to the final privacy budget to the candidate feature vector to obtain a noise vector.

[0058] Specifically, a differential privacy perturbation mechanism can be selected based on the numerical characteristics of the candidate feature vectors. For continuous value feature vectors in this embodiment, a Laplace mechanism can be used.

[0059] In some specific embodiments, step S105 above includes steps S1051-S1053: Step S1051: Based on the preset data sensitivity and the final privacy budget, the corresponding noise scale parameter is calculated.

[0060] Specifically, the amount of noise added by the Laplace mechanism depends on the scale parameter. ,and With the final privacy budget Inversely proportional, the specific relationship is as follows: ( (for preset data sensitivity), where the final privacy budget The larger the value, the higher the corresponding noise scale parameter. The smaller the value, the lower the intensity of the superimposed random noise, and the higher the fidelity of the noise vector information after perturbation.

[0061] Step S1052: Generate random noise based on the noise scale parameters.

[0062] Step S1053: Add the random noise to the values ​​of each dimension of the candidate feature vector to obtain the perturbed noise vector.

[0063] Specifically, for each dimension of each candidate feature vector, a value from... Random noise sampled from the distribution. Due to It is dynamic and therefore high-value data ( big, The less noise is added to low-value data, the higher its information fidelity; conversely, the more noise is added to low-value data.

[0064] In some specific embodiments, after step S105 above, the method further includes: Step S106: For each of the noise vectors that correspond one-to-one with the candidate feature vectors, identify the entity information corresponding to each noise vector.

[0065] In some specific embodiments, step S106 above includes steps S1061-S1063: Step S1061: Preload an industrial domain ontology library containing information on multiple entities and pre-train a classification model.

[0066] Specifically, the entity information includes at least one of the following: equipment, component, fault mode, and operation and maintenance type.

[0067] Specifically, the classification model is obtained by training a mixture of standard feature vectors and multiple preset noise vectors. The multiple preset noise vectors are generated by adding random noise of different intensities to the standard feature vectors. The standard feature vectors are extracted from sample data corresponding to the information of each entity in the industrial domain ontology library.

[0068] In some specific embodiments, the specific training process of the classification model is as follows: First, based on the various entity sample data already labeled in the industrial ontology library, including the original multimodal data corresponding to equipment entities, component entities, fault mode entities, and operation and maintenance entities, unified feature encoding is performed through a preset feature extraction network (TCN temporal convolutional network, image feature extraction network, and text feature extraction network) to extract standard feature vectors with consistent dimensions and semantic alignment. These feature vectors are clean features without added noise and are used as positive sample training data for the model.

[0069] Secondly, in order to simulate differential privacy perturbation scenarios and improve the model's noise robustness, multiple privacy budgets of different sizes are configured for each standard feature vector, and Laplace random noise of different intensities is generated according to different privacy budgets. Random noise of each intensity is superimposed on the corresponding standard feature vector to generate multiple preset noise vectors, forming a diverse noise feature sample set covering weak perturbation, medium perturbation and strong perturbation.

[0070] Finally, all standard feature vectors are mixed with all preset noise vectors to construct a hybrid training dataset, and the entity labels corresponding to the ontology are used as supervision labels. A classification loss function is used to iteratively train the model, continuously optimizing the model parameters to ensure that the model maintains stable and accurate entity classification ability for noise vectors after differential privacy perturbation. The trained classification model exhibits strong noise robustness and can directly perform entity inference on noise vectors after privacy perturbation.

[0071] Step S1062: For any noise vector, input the noise vector into the classification model and output the probability distribution result of the noise vector in the ontology library of the industrial domain for multiple entities.

[0072] Step S1063: Based on the probability distribution results, select the entity information corresponding to the highest probability as the target entity information for noise vector matching.

[0073] Specifically, when a noise vector is input into a classification model, the model outputs a probability distribution that indicates which entity in the ontology is most likely to correspond to the noise vector. For example, a noise vector from an operations log might be linked with a high probability to the entity "Operations Operation: Replace Bearing"; a noise vector from a sensor might be linked to "Sensor: Spindle Vibration Sensor".

[0074] Step S107: Perform relation prediction on the entity information of any two noise vectors to obtain candidate triples.

[0075] Specifically, the triplet includes entity information of any two noise vectors and the association between the entity information.

[0076] In some specific embodiments, step S107 above includes steps S1071-S1072: Step S1071: Perform semantic relationship prediction on the entity information of any two noise vectors to obtain the semantic association relationship between the entity information.

[0077] Step S1072: Based on the semantic association and the entity information of any two noise vectors, generate corresponding candidate triplet groups.

[0078] In steps S1071-S1072, semantic relationship prediction can be performed on the entity information of any two noise vectors using a pre-trained relation classification model or preset classification rules. For example: When predicting the semantic relationship between "entity: high frequency vibration" and "entity: bearing wear", the relationship classification model will extract the isSymptomOf (is a symptom of...) relationship between them with a high probability, thereby generating a candidate triple: (bearing wear, isSymptomOf, high frequency vibration).

[0079] Step S108: When the cross-modal association score of the plurality of candidate feature vectors is greater than the preset association score threshold, a corresponding data aggregation event entity is created in the knowledge graph.

[0080] Specifically, when the cross-modal association score exceeds a preset association score threshold, several candidate feature vectors are determined to be closely correlated spatiotemporally and semantically, thus belonging to valid industrial fault event data. Based on this, a new data aggregation event entity is instantiated in the knowledge graph, assigning a unique event entity identifier to each of the aforementioned candidate feature vectors, generating a unique event entity instance (e.g., FaultEvent_#XYZ123). This data aggregation event entity serves as a unified hub for multimodal entity information, relationships, and fault evidence chains, providing a core anchor point for the subsequent unified reconstruction and integration of scattered candidate triples, achieving event-level aggregation of fragmented fault data.

[0081] Step S109 involves obtaining all candidate triples through the entity information of the several noise vectors, and then reconstructing and integrating them with the data aggregation event entity as the core to form structured knowledge graph data.

[0082] Specifically, all candidate triples are obtained from the entity information of several noise vectors, and reconstructed into structured knowledge centered on the event entities aggregated in this data. For example, scattered entities (bearing wear, isSymptomOf, high-frequency vibration) and other related entities will be transformed into a series of logically clear triples: (CNC machine tool A, hasEvent, FaultEvent_#XYZ123) (FaultEvent_#XYZ123, hasSymptom, high-frequency vibration) (FaultEvent_#XYZ123, hasRootCause, Bearing Wear) (FaultEvent_#XYZ123, associatedLog, Operations Log_SN456) These structured triples built around event entities are imported in batches and persistently stored in graph databases such as Neo4j.

[0083] This proposal, through the aforementioned steps, departs from the existing technology's "one-size-fits-all" privacy processing approach that blindly identifies connections between all data. Instead, it establishes a novel mechanism capable of perceiving the inherent relationships between multimodal data and intelligently assessing the value of data analysis accordingly, ultimately achieving a dynamic match between the strength of privacy protection and data value. By prioritizing the protection of the integrity of high-value "cross-modal event evidence chains," it provides provable privacy security while ensuring that the final industrial knowledge graph retains its core, high-fidelity knowledge content when performing advanced analytical tasks such as complex fault diagnosis and predictive maintenance.

[0084] The embodiments of this application have the following technical effects: Innovation Point 1: Relationship-Driven Adaptive Privacy Budget Allocation Mechanism The core technical feature of this invention is a dynamic differential privacy budget allocation mechanism. This mechanism no longer uses a fixed privacy budget or one based solely on global statistical features, but instead uses a real-time calculated cross-modal association score. —That is, the strength of cross-modal correlation between multimodal data—is its most direct and decisive input variable. The system establishes a functional relationship. This allows each data point within a cross-modal data cluster with higher correlation strength to be dynamically and automatically allocated a higher privacy budget. The technical effects and advantages are as follows: By allocating high privacy budgets (meaning imposing extremely low noise) to strongly correlated data clusters that collectively point to critical events (such as precursors to failures), this method can pass through these "golden signals" crucial for diagnosis and prediction with extremely high fidelity. It fundamentally resolves the fatal contradiction in existing technologies where critical signals must be buried in noise to meet privacy requirements.

[0085] This approach concentrates valuable budgetary resources on protecting the most valuable information, while applying standard-strength protection to less valuable, isolated data points. Compared to existing techniques that waste budgets evenly across massive amounts of routine data, this approach achieves a qualitative leap in data usability while maintaining the same level of overall privacy protection.

[0086] Innovation Point Two: A Quantitative Evaluation Model for Cross-Modal Association Value To achieve the first innovation, this invention designs a model that can transform the abstract relationships between multimodal data into calculable quantitative indicators. This model achieves real-time, automated evaluation of the value of cross-modal data aggregation analysis in industrial scenarios through three consecutive steps: "configuring data labels for the data," "converting data from different modalities into fixed-length feature vectors," and "calculating the cross-modal association scores of several candidate feature vectors."

[0087] Table 1: Comparison of Relevant Value Assessment Results

[0088] Table 1 illustrates that even if the EGT anomaly signals are exactly the same, the quantitative evaluation model of this invention can accurately distinguish their true aggregate analysis value by analyzing the accompanying modal data.

[0089] Innovation Point 3: A Noise-Robust Knowledge Extraction and Event Fusion Pipeline This invention designs a knowledge extraction pipeline specifically for processing feature vectors after differential privacy perturbation. This pipeline uses a "noise-enhanced trained classifier" to link entities, predicts the relationship between entity information of any two noise vectors to obtain candidate triples, and finally, through "creating event entities," it centralizes and structures the scattered multimodal information.

[0090] Corresponding to the above implementation methods of differential privacy protection based on multimodal association, this application also provides a differential privacy protection system based on multimodal association, used to execute the differential privacy protection method based on multimodal association described in any of the above embodiments. Figure 2 As shown, this differential privacy protection system based on multimodal correlation includes: The data acquisition module is used to acquire multi-source heterogeneous data and configure data tags for all data in the multi-source heterogeneous data. The data tags include at least a device identifier and a timestamp. The candidate feature vector filtering module is used to filter out several candidate feature vectors from the feature vectors of all data according to the data label. The several candidate feature vectors correspond to the same device identifier, and the difference between the timestamps of any two candidate feature vectors is less than a preset time threshold. The cross-modal association score calculation module is used to calculate the cross-modal association score of the plurality of candidate feature vectors, wherein the cross-modal association score is used to characterize the confidence that the plurality of candidate feature vectors jointly point to the same industrial event; The final privacy budget calculation module is used to calculate the final privacy budget of any candidate feature vector based on the cross-modal association score and the basic privacy budget of the candidate feature vector. The noise vector generation module is used to add noise corresponding to the final privacy budget to the candidate feature vector to obtain a noise vector.

[0091] Optionally, the system further includes: a structured knowledge graph data generation module, used to identify entity information corresponding to each of the noise vectors that correspond one-to-one with the candidate feature vectors; to predict the relationship between the entity information of any two noise vectors to obtain candidate triples; the triples include the entity information of the two noise vectors and the association relationship between the entity information; when the cross-modal association score of the candidate feature vectors is greater than a preset association score threshold, a corresponding data aggregation event entity is created in the knowledge graph; all candidate triples obtained through the entity information of the noise vectors are reconstructed and integrated with the data aggregation event entity as the core to form structured knowledge graph data.

[0092] Optionally, the cross-modal association score calculation module is further configured to determine the modal type corresponding to the plurality of candidate feature vectors and obtain the preset association weights between different modal types; calculate the feature similarity between each group of candidate feature vectors for the combination of candidate feature vectors belonging to different modal types among the plurality of candidate feature vectors; calculate the association score corresponding to each group of candidate feature vectors based on the feature similarity and association weights corresponding to each group of candidate feature vectors; and sum the association scores corresponding to all groups to obtain the cross-modal association score corresponding to the plurality of candidate feature vectors.

[0093] Optionally, the final privacy budget calculation module is further configured to preset a monotonically increasing privacy budget mapping function, wherein the privacy budget mapping function uses cross-modal association scores as the only input variable; input the cross-modal association scores corresponding to the plurality of candidate feature vectors into the privacy budget mapping function to obtain budget adjustment coefficients; and perform correction calculations on the basic privacy budget of the candidate feature vectors based on the budget adjustment coefficients to obtain the final privacy budget corresponding to the candidate feature vectors.

[0094] Optionally, the noise vector generation module is further configured to calculate the corresponding noise scale parameter based on the preset data sensitivity and the final privacy budget; generate random noise according to the noise scale parameter; and add the random noise to the values ​​of each dimension of the candidate feature vector to obtain the perturbed noise vector.

[0095] Optionally, the structured knowledge graph data generation module is further used to preload an industrial domain ontology containing multiple entity information and pre-train a classification model. The entity information includes at least one of the following: equipment, components, fault modes, and operation and maintenance types. The classification model is obtained by training a mixture of standard feature vectors and multiple preset noise vectors. The multiple preset noise vectors are generated by adding random noise of different intensities to the standard feature vectors. The standard feature vectors are extracted based on sample data corresponding to each entity information in the industrial domain ontology. For any noise vector, the noise vector is input into the classification model, and the probability distribution result of the noise vector in the multiple entity information in the industrial domain ontology is output. Based on the probability distribution result, the entity information corresponding to the highest probability is selected as the target entity information for matching the noise vector.

[0096] Optionally, the structured knowledge graph data generation module is also used to predict the semantic relationship between the entity information of any two noise vectors to obtain the semantic association between the entity information; and to generate corresponding candidate triples based on the semantic association and the entity information of the any two noise vectors.

[0097] The differential privacy protection system based on multimodal correlation provided in the above embodiments of this application and the differential privacy protection method based on multimodal correlation provided in the embodiments of this application are based on the same inventive concept and have the same beneficial effects as the methods adopted, run or implemented by the applications stored therein.

[0098] This application also provides a computer device for executing the above-described differential privacy protection method based on multimodal correlation. Please refer to... Figure 3 This illustrates a schematic diagram of a computer device provided by some embodiments of this application. For example... Figure 3 As shown, the computer device 3 includes a processor 300, a memory 301, a bus 302, and a communication interface 303. The processor 300, the communication interface 303, and the memory 301 are connected via the bus 302. The memory 301 stores computer instructions that can be executed on the processor 300. When the processor 300 executes the computer instructions, it performs the differential privacy protection method based on multimodal association provided in any of the foregoing embodiments of this application.

[0099] The memory 301 may include high-speed random access memory (RAM) or non-volatile memory, such as at least one disk storage device. Communication between this system network element and at least one other network element is achieved through at least one communication interface 303 (which can be wired or wireless), such as the Internet, wide area network, local area network, or metropolitan area network.

[0100] Bus 302 can be an ISA bus, PCI bus, or EISA bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. Memory 301 is used to store programs. After receiving an execution instruction, processor 300 executes the program. The differential privacy protection method based on multimodal correlation disclosed in any of the foregoing embodiments can be applied to processor 300, or implemented by processor 300.

[0101] The processor 300 may be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method can be completed by the integrated logic circuitry in the hardware of the processor 300 or by instructions in software form. The processor 300 may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), an off-the-shelf programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software modules may reside in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in memory 301. The processor 300 reads the information in memory 301 and, in conjunction with its hardware, completes the steps of the above method.

[0102] The computer device provided in this application embodiment and the differential privacy protection method based on multimodal correlation provided in this application embodiment are based on the same inventive concept and have the same beneficial effects as the methods they adopt, operate or implement.

[0103] This application also provides a computer-readable storage medium corresponding to the differential privacy protection method based on multimodal correlation provided in the foregoing embodiments. Please refer to [link / reference]. Figure 4 The computer-readable storage medium shown is an optical disc 30, on which computer instructions (i.e., program products) are stored. When the computer instructions are executed by a processor, they perform the differential privacy protection method based on multimodal association provided in any of the foregoing embodiments.

[0104] It should be noted that examples of the computer-readable storage medium may also include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other optical and magnetic storage media, which will not be elaborated here.

[0105] The computer-readable storage medium provided in the above embodiments of this application and the differential privacy protection method based on multimodal correlation provided in the embodiments of this application are based on the same inventive concept and have the same beneficial effects as the methods adopted, run or implemented by the applications stored therein.

[0106] It should be noted that: Numerous specific details are set forth in the specification provided herein. However, it will be understood that embodiments of this application may be practiced without these specific details. In some instances, well-known structures and techniques have not been shown in detail so as not to obscure the understanding of this specification.

[0107] Similarly, it should be understood that, for the sake of brevity and to aid in understanding one or more of the various inventive aspects, in the above description of exemplary embodiments of this application, various features of this application are sometimes grouped together in a single embodiment, figure, or description thereof. However, this disclosure should not be construed as reflecting a schematic diagram in which the claimed application requires more features than expressly recited in each claim. Rather, as reflected in the following claims, inventive aspects lie in fewer than all features of a single foregoing disclosed embodiment. Therefore, the claims following the detailed description are hereby expressly incorporated into that detailed description, wherein each claim itself is a separate embodiment of this application.

[0108] Furthermore, those skilled in the art will understand that although some embodiments described herein include certain features but not others included in other embodiments, combinations of features from different embodiments are intended to be within the scope of this application and form different embodiments. For example, in the following claims, any of the claimed embodiments can be used in any combination.

[0109] The above description is merely a preferred embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A differential privacy protection method based on multimodal correlation, characterized in that, The method includes: Acquire multi-source heterogeneous data and configure data tags for all data in the multi-source heterogeneous data, wherein the data tags include at least a device identifier and a timestamp; Based on the data tag, several candidate feature vectors are selected from the feature vectors of all data. The several candidate feature vectors correspond to the same device identifier, and the difference between the timestamps of any two candidate feature vectors is less than a preset time threshold. Calculate the cross-modal association score of the candidate feature vectors, whereby the cross-modal association score is used to characterize the confidence that the candidate feature vectors jointly point to the same industrial event; For any candidate feature vector, the final privacy budget of the candidate feature vector is calculated based on the cross-modal association score and the basic privacy budget of the candidate feature vector. Add noise corresponding to the final privacy budget to the candidate feature vector to obtain a noise vector.

2. The method according to claim 1, characterized in that, After obtaining the noise vector, the method further includes: For each of the noise vectors that correspond one-to-one with the candidate feature vectors, the entity information corresponding to each noise vector is identified. Candidate triples are obtained by predicting the relationship between the entity information of any two noise vectors; the triples include the entity information of the two noise vectors and the relationship between the entity information. When the cross-modal association score of the aforementioned candidate feature vectors is greater than a preset association score threshold, a corresponding data aggregation event entity is created in the knowledge graph; All candidate triples are obtained through the entity information of the aforementioned noise vectors. The data aggregation event entities are then used as the core for reconstruction and integration to form structured knowledge graph data.

3. The method according to claim 1 or 2, characterized in that, Calculating the cross-modal correlation score of the aforementioned candidate feature vectors includes: Determine the modality type corresponding to the plurality of candidate feature vectors, and obtain the preset correlation weights between different modality types; For the combinations of candidate feature vectors belonging to different modal types among the several candidate feature vectors, calculate the feature similarity between each group of candidate feature vectors; Based on the feature similarity and association weights of each group of candidate feature vectors, the association score of each group of candidate feature vectors is calculated. The cross-modal association scores corresponding to all groups are summed to obtain the cross-modal association scores corresponding to the candidate feature vectors.

4. The method according to claim 1 or 2, characterized in that, Based on the cross-modal association score and the basic privacy budget of the candidate feature vector, the final privacy budget of the candidate feature vector is calculated, including: A pre-defined, monotonically increasing privacy budget mapping function is used, with cross-modal association scores as the sole input variable. The cross-modal association scores corresponding to the candidate feature vectors are input into the privacy budget mapping function to obtain the budget adjustment coefficient; The basic privacy budget of the candidate feature vector is corrected and calculated based on the budget adjustment coefficient to obtain the final privacy budget corresponding to the candidate feature vector.

5. The method according to claim 1 or 2, characterized in that, Add noise corresponding to the final privacy budget to the candidate feature vector to obtain a noise vector, including: Based on the preset data sensitivity and the final privacy budget, the corresponding noise scale parameters are calculated. Random noise is generated based on the noise scale parameters; The random noise is added to the values ​​of each dimension of the candidate feature vector to obtain the perturbed noise vector.

6. The method according to claim 2, characterized in that, Identify the entity information corresponding to each noise vector, including: An industrial domain ontology library containing multiple entity information is preloaded, and a classification model is pre-trained. The entity information includes at least one of the following: equipment, components, fault modes, and operation and maintenance types. The classification model is obtained by training a mixture of standard feature vectors and multiple preset noise vectors. The multiple preset noise vectors are generated by adding random noise of different intensities to the standard feature vectors. The standard feature vectors are extracted based on sample data corresponding to each entity information in the industrial domain ontology library. For any noise vector, the noise vector is input into the classification model, and the probability distribution result of the noise vector in multiple entity information in the industrial domain ontology is output. Based on the probability distribution results, the entity information corresponding to the highest probability is selected as the target entity information for noise vector matching.

7. The method according to claim 2, characterized in that, Relationship prediction is performed on the entity information of any two noise vectors to obtain candidate triples, including: Semantic relationship prediction is performed on the entity information of any two noise vectors to obtain the semantic association between the entity information; Based on the semantic association and the entity information of any two noise vectors, corresponding candidate triples are generated.

8. A differential privacy protection system based on multimodal correlation, characterized in that, The system includes: The data acquisition module is used to acquire multi-source heterogeneous data and configure data tags for all data in the multi-source heterogeneous data. The data tags include at least a device identifier and a timestamp. The candidate feature vector filtering module is used to filter out several candidate feature vectors from the feature vectors of all data according to the data label. The several candidate feature vectors correspond to the same device identifier, and the difference between the timestamps of any two candidate feature vectors is less than a preset time threshold. The cross-modal association score calculation module is used to calculate the cross-modal association score of the plurality of candidate feature vectors, wherein the cross-modal association score is used to characterize the confidence that the plurality of candidate feature vectors jointly point to the same industrial event; The final privacy budget calculation module is used to calculate the final privacy budget of any candidate feature vector based on the cross-modal association score and the basic privacy budget of the candidate feature vector. The noise vector generation module is used to add noise corresponding to the final privacy budget to the candidate feature vector to obtain a noise vector.

9. A computer device, characterized in that, include: A memory and a processor are communicatively connected, the memory stores computer instructions, and the processor executes the computer instructions to perform the differential privacy protection method based on multimodal correlation as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions for causing the computer to execute the differential privacy protection method based on multimodal correlation as described in any one of claims 1 to 7.

Images