A Multimodal Privacy Computing Fusion Method and System
By using multi-dimensional feature quantization and robust optimization, dynamic selection and cross-mode collaboration of privacy computing modes are achieved, solving the problems of non-optimal mode selection and resource waste in existing technologies, improving computational efficiency and model accuracy, while ensuring security and compliance.
Patent Information
- Application Number
- CN202611628715.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-10-21
- Publication Date
- 2026-11-17
AI Technical Summary
The lack of optimality in the selection of existing privacy computing models, the complete isolation between models and the inability to coordinate, and the absence of multi-objective optimization mechanisms under security and compliance constraints lead to waste of computing resources and loss of model accuracy.
Multimodal privacy computing fusion is achieved through multi-dimensional feature quantization, multi-source sub-bar optimization pattern recommendation, unified abstraction of privacy computing patterns and cross-modal intermediate state transformation, and submodal multi-objective optimization and adaptive balance penalty orchestration decision-making.
It enables dynamic mode selection based on the actual statistical distribution of data and the status of computing resources, cross-mode collaborative computing, optimization of resource utilization and model accuracy, and ensures security and compliance.
Smart Images

Figure CN123116343A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of privacy computing and trusted data space technology, specifically to a multimodal privacy computing fusion method and system. Background Technology
[0002] With the increasing demands for data privacy protection and compliance in fields such as healthcare, financial credit reporting, and government data sharing, secure computation that keeps data within its domain has become a fundamental technological support for cross-institutional data collaboration. Currently, the mainstream privacy-preserving computation models include three categories: federated learning, secure multi-party computation, and trusted execution environments. Federated learning allows participating parties to train models locally and exchange only encrypted gradient parameters, suitable for scenarios with sufficient and relatively evenly distributed data. Secure multi-party computation enables direct computation within the encrypted domain, suitable for small-scale statistical queries and simple analysis. Trusted execution environments perform plaintext computation within a secure, hardware-isolated enclave, suitable for computationally intensive tasks but with a strong dependence on the hardware platform.
[0003] The most relevant prior art to this invention is a privacy-preserving computation mode selection framework based on a rule engine. This framework predefines a set of static rules, extracts the sensitivity labels of the data and the computation task type after receiving a task request, matches and searches in the rule table, and assigns the matched privacy-preserving computation mode to the corresponding task for execution; the three modes run independently and do not communicate with each other, remain fixed after the task is assigned, and intermediate computation results cannot be transferred across modes.
[0004] However, the existing technologies mentioned above have the following drawbacks: First, rule matching relies heavily on manually preset static rules and cannot be dynamically adjusted according to the actual statistical distribution characteristics of the data and the real-time status of computing resources, resulting in a lack of optimality in the mode selection results. Second, the three privacy computing modes are completely isolated, lacking a cross-mode intermediate data conversion and collaboration mechanism. When the optimal execution mode of a subtask is inconsistent with the mode allocated by the global task, flexible switching at the subtask level cannot be achieved, resulting in wasted computing resources and loss of model accuracy. Third, there is a lack of a multi-objective optimization mechanism under hard constraints of security and compliance. Existing technologies treat security and compliance as a pre-filtering condition rather than an optimization constraint, failing to finely balance computational efficiency and model accuracy while ensuring security. Therefore, a multimodal privacy computing fusion scheme that can dynamically perceive, uniformly represent, collaborate across modes, and optimize multiple objectives under security and compliance constraints is needed. Summary of the Invention
[0005] To address the technical problems in existing technologies, such as the lack of optimality in privacy computing mode selection, complete isolation between modes preventing collaboration, and the absence of multi-objective optimization mechanisms under security and compliance constraints, this invention provides a multimodal privacy computing fusion method and system. Through multi-dimensional feature quantization, multi-source sub-bar optimization mode recommendation, unified abstraction of privacy computing modes and cross-mode intermediate state transformation, and orchestration decision-making of sub-modal multi-objective optimization and adaptive balance penalty, it achieves intelligent orchestration and organic fusion of multiple privacy computing modes.
[0006] This invention provides a multimodal privacy computing fusion method, comprising:
[0007] Obtain the data asset metadata of each participant, and perform multi-dimensional feature quantization on the data asset metadata to obtain the data feature vector;
[0008] The data feature vector is decomposed into each feature dimension and input into a multi-source robust optimization model. The reliability parameters of each feature dimension are dynamically adjusted based on historical feedback to construct a set of distribution uncertainties. Robust pattern recommendation is performed on the set of distribution uncertainties, and a set of candidate pattern combinations is output.
[0009] According to the preset abstract mapping rules, each privacy computing mode adopted in the candidate mode combination set is mapped to obtain the corresponding unified computing unit abstract model, and the unified computing unit abstract model set is formed by summarizing them.
[0010] Based on the unified computing unit abstract model set, a cross-mode intermediate state data structure is constructed and a state transition function is defined. An intermediate state transition protocol is generated by the cross-mode intermediate state data structure and the state transition function.
[0011] The candidate mode combination set and the unified computing unit abstract model set are input into the submodal multi-objective optimization model. The constrained Pareto optimal solution is solved through an adaptive balance penalty mechanism. The constrained Pareto optimal solution is determined as the multimodal computing orchestration scheme.
[0012] Resource orchestration and cross-modal computation are performed according to the multimodal computation orchestration scheme. When the mode switching conditions are met, the intermediate state conversion protocol is invoked to complete the intermediate state data conversion and obtain the privacy computation result.
[0013] Furthermore, the step of obtaining data asset metadata of each participant and performing multi-dimensional feature quantization on the data asset metadata to obtain a data feature vector includes: receiving data asset configuration files uploaded by each participant through a preset interface, parsing to obtain the data asset metadata; and extracting and calculating sensitivity index, computational complexity index, Non-IID distribution index, and communication capacity index from the data asset metadata.
[0014] The sensitivity index, computational complexity index, Non-IID distribution index, and communication capacity index are each normalized and mapped into feature vectors, and the feature vectors are concatenated to form the data feature vector.
[0015] Further, the step of decomposing the data feature vector into feature dimensions and inputting them into a multi-source sub-Bruker optimization model, dynamically adjusting the reliability parameters of each feature dimension based on historical feedback to construct a set of distributional uncertainties, performing robust pattern recommendation on the set of distributional uncertainties, and outputting a set of candidate pattern combinations includes: decomposing the data feature vector into feature dimensions according to quantified attribute categories, inputting each feature dimension into the multi-source sub-Bruker optimization model, calculating the accuracy rate based on the obtained historical recommendation correctness feedback, dynamically adjusting the Wasserstein trust radius of the corresponding feature dimension based on the accuracy rate, and using the adjusted Wasserstein trust radius as the reliability parameter, wherein each feature dimension includes a sensitivity index, a computational complexity index, a Non-IID distribution index, and a communication capacity index; setting a fluctuation range for the empirical probability distribution of each feature dimension based on the reliability parameter to generate the set of distributional uncertainties; finding the worst-case probability distribution in the set of distributional uncertainties, performing robust pattern recommendation on the worst-case probability distribution, and outputting the set of candidate pattern combinations.
[0016] Further, the step of mapping each privacy computing mode adopted in the candidate mode combination set according to the preset abstract mapping rules to obtain the corresponding unified computing unit abstract model, and summarizing them to form a unified computing unit abstract model set, includes: traversing the candidate mode combination set and extracting the privacy computing mode adopted by each candidate mode combination in the candidate mode combination set; determining the operator type and interface definition corresponding to each privacy computing mode according to the preset abstract mapping rules; constructing a unified computing unit abstract model corresponding to each privacy computing mode based on the operator type and the interface definition; and summarizing all the unified computing unit abstract models to form the unified computing unit abstract model set.
[0017] Furthermore, determining the operator type and interface definition corresponding to each privacy computing mode according to the preset abstract mapping rules includes: parsing the underlying protocol and execution logic of each privacy computing mode according to the preset abstract mapping rules; matching the underlying protocol and execution logic with a preset operator library to determine the corresponding operator type; and extracting the data input and output specifications of each privacy computing mode and standardizing them into a unified interface definition.
[0018] Furthermore, the step of constructing a cross-mode intermediate state data structure and defining a state transition function based on the unified computing unit abstract model set, and generating an intermediate state transition protocol from the cross-mode intermediate state data structure and the state transition function, includes: traversing the unified computing unit abstract model set and extracting the operator attributes and data flow relationships of each unified computing unit abstract model; constructing a standardized cross-mode intermediate state data structure based on the operator attributes and the data flow relationships; determining state triggering conditions and state transition logic using the execution dependencies between each unified computing unit abstract model, mapping the state triggering conditions to the state transition logic, and defining the state transition function; and combining and encapsulating the cross-mode intermediate state data structure and the state transition function to generate the intermediate state transition protocol.
[0019] Further, the step of inputting the candidate mode combination set and the unified computing unit abstract model set into the submodal multi-objective optimization model, solving for the constrained Pareto optimal solution through an adaptive balance penalty mechanism, and determining the constrained Pareto optimal solution as the multimodal computing orchestration scheme includes: inputting the candidate mode combination set and the unified computing unit abstract model set into the submodal multi-objective optimization model, constructing a multi-objective optimization function and corresponding constraints, the constraints including differential privacy budget constraints, security level constraints, and resource upper limit constraints; setting and initializing adaptive penalty weights, calculating the degree of violation of the differential privacy budget constraint, the degree of violation of the security level constraint, and the degree of violation of the resource upper limit constraint in the current optimization iteration process, obtaining the comprehensive violation degree, adaptively updating the adaptive penalty weights using the comprehensive violation degree, and constructing the adaptive balance penalty mechanism; introducing the adaptive balance penalty mechanism into the submodal optimization algorithm, iteratively solving the multi-objective optimization function with added penalty terms, obtaining the constrained Pareto optimal solution that satisfies the constraints, and determining the constrained Pareto optimal solution as the multimodal computing orchestration scheme.
[0020] Furthermore, the iterative solution of the multi-objective optimization function with added penalty terms to obtain the constrained Pareto optimal solution that satisfies the constraints includes: setting an orchestration set and initializing it as an empty set; for each candidate mode combination in the candidate mode combination set, calculating the minimum marginal gain for selecting the candidate mode combination into the orchestration set; under the differential privacy budget constraint, introducing an exponential mechanism to calculate the selection probability of each candidate mode combination based on the minimum marginal gain; and iteratively performing a probabilistic greedy selection to gradually add the candidate mode combination into the orchestration set according to the selection probability.
[0021] When the orchestration set satisfies the constraints and the minimum marginal gain of the candidate pattern combination is lower than a preset threshold, the orchestration set is output as the constrained Pareto optimal solution.
[0022] Furthermore, before inputting the candidate pattern combination set and the unified computing unit abstract model set into the sub-modal multi-objective optimization model, the method further includes: obtaining the semantic annotation of each attribute column in the data asset metadata, calculating the correlation degree between each attribute column based on conditional mutual information, evaluating the inferability between each attribute column based on the correlation degree to conduct inference risk assessment, and obtaining attribute association risk parameters.
[0023] Combining the attribute-related risk parameters and the preset initial privacy budget, a privacy budget allocation model is established with the privacy budget allocation ratio of each attribute column as the optimization variable and the data utility maximization as the objective function. The privacy budget allocation model is solved by Lagrange duality to obtain the optimal solution of the optimization variable, and the optimal solution is used as a differentiated adaptive privacy budget allocation scheme. The adaptive privacy budget allocation scheme is used as a constraint condition and introduced into the submodal multi-objective optimization model for solution to obtain the multimodal computation orchestration scheme.
[0024] Further, the step of performing resource orchestration and cross-modal computation according to the multimodal computation orchestration scheme, and calling the intermediate state conversion protocol to complete the intermediate state data conversion when the mode switching conditions are met, to obtain the privacy computation result, includes: determining the first privacy computation mode in the multimodal computation orchestration scheme as the current privacy computation mode, performing resource orchestration on the current privacy computation mode and starting the computation task of the model to be processed, wherein the model to be processed is a machine learning model constructed based on the data asset metadata;
[0025] The execution status of the computation task is monitored, and the accuracy of the model to be processed is obtained. When the accuracy improvement of the model to be processed is lower than the minimum threshold in a series of preset iterations, or the communication latency of the computation task exceeds a preset latency, the mode switching condition is determined to be met and a mode switching is triggered. The intermediate state data output by the current privacy computation mode is obtained, and the intermediate state conversion protocol is called to parse and map the intermediate state data, converting it into the data structure required by the next privacy computation mode, thus completing the intermediate state data conversion. Based on the data structure, the computation task of the next privacy computation mode is started to complete the cross-mode computation. The steps of monitoring the execution status of the computation task until the cross-mode computation is completed are repeated until the computation task of the last privacy computation mode in the multimodal computation orchestration scheme is completed, and the data output by the last privacy computation mode is output as the privacy computation result.
[0026] Furthermore, after invoking the intermediate state conversion protocol to complete the intermediate state data conversion, the method further includes:
[0027] Obtain the parameters of the model to be processed corresponding to the intermediate state data before and after switching between the current privacy computing mode and the next privacy computing mode, and use them as parameters to be verified; verify the consistency of the parameters to be verified, and use Chebyshev inequality probability constraints to keep the deviation of the parameters to be verified within the tolerance range.
[0028] The present invention also provides a multimodal privacy computing fusion system, comprising:
[0029] The feature quantization module is used to acquire data asset metadata of each participant, perform multi-dimensional feature quantization on the data asset metadata, and obtain data feature vectors.
[0030] The pattern recommendation module is used to decompose the data feature vector into each feature dimension and input it into the multi-source robust optimization model. Based on historical feedback, it dynamically adjusts the reliability parameters of each feature dimension to construct a set of distribution uncertainty, performs robust pattern recommendation on the set of distribution uncertainty, and outputs a set of candidate pattern combinations.
[0031] The abstract mapping module is used to map each privacy computing mode adopted in the candidate mode combination set according to the preset abstract mapping rules, so as to obtain the corresponding unified computing unit abstract model and summarize them to form a unified computing unit abstract model set.
[0032] The protocol generation module is used to construct a cross-mode intermediate state data structure and define a state transition function based on the unified computing unit abstract model set, and to generate an intermediate state transition protocol from the cross-mode intermediate state data structure and the state transition function;
[0033] The scheme optimization module is used to input the candidate mode combination set and the unified computing unit abstract model set into the submodal multi-objective optimization model, solve the constrained Pareto optimal solution through an adaptive balance penalty mechanism, and determine the constrained Pareto optimal solution as the multimodal computing orchestration scheme.
[0034] The fusion computing module is used to perform resource orchestration and cross-modal computing according to the multimodal computing orchestration scheme. When the mode switching conditions are met, the intermediate state conversion protocol is invoked to complete the intermediate state data conversion and obtain the privacy computing result.
[0035] The beneficial effects of this invention are as follows:
[0036] 1. This invention quantifies the data asset metadata of each participant into four dimensions: sensitivity, computational complexity, Non-IID distribution, and communication capacity, and concatenates them into a data feature vector. At the same time, it regards the four dimensions as independent information sources with dynamic changes in reliability, constructs a Wasserstein fuzzy set centered on the experience distribution, and dynamically adjusts the Wasserstein trust radius of each dimension through feedback on the correctness of historical recommendations. This enables the selection of privacy computing mode to be based on the actual statistical distribution characteristics of the data and the real-time status of computing resources, and to provide robust mode recommendations under the worst-case probability distribution. This overcomes the shortcomings of existing rule engines that rely on manually preset static rules, cannot be dynamically adjusted, and are sensitive to feature noise and distribution drift.
[0037] 2. This invention constructs a unified computational unit abstract model that includes input interfaces, operator descriptions, output interfaces, and security states. It maps three heterogeneous privacy-preserving computation modes—federated learning, secure multi-party computation, and trusted execution environment—into a unified model. Based on this abstract model set, it constructs cross-mode intermediate state data structures and state transition functions, and generates intermediate state transition protocols. This enables the computational pipeline to flexibly switch between different modes and semantically retain intermediate computation results. It overcomes the shortcomings of existing technologies, such as complete isolation of the three modes, inability to transfer intermediate results across modes, and resource waste and accuracy loss caused by inconsistency between the subtask-level optimal mode and the global allocation mode.
[0038] 3. This invention proposes a scheme to model mode selection and resource allocation as a multi-objective submodular maximization problem. An adaptive balance penalty mechanism dynamically drives the constraint penalty weights. Before this, an allocation model is established based on conditional mutual information reasoning risk assessment, with the goal of maximizing data utility and the total privacy budget as a constraint. Differentiated adaptive privacy budget allocation schemes are obtained through Lagrange duality solution. Furthermore, before and after mode switching, Chebyshev inequality probability constraints keep the model parameter deviation within the tolerance range, ensuring that constraints such as security level, differential privacy budget, and resource upper limit are not violated during the optimization process. It also coordinates the balance between security compliance and performance availability at both the data level and algorithm level, overcoming the shortcomings of existing technologies that use security compliance as a pre-filter condition and cannot finely balance efficiency and accuracy under the premise of security. Attached Figure Description
[0039] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0040] Figure 1This is a schematic diagram of the overall process of a multimodal privacy computing fusion method according to the present invention;
[0041] Figure 2 This is a schematic diagram of the privacy computing mode recommendation process based on multi-source split-bar optimization of the present invention;
[0042] Figure 3 This is a schematic diagram illustrating the generation and invocation process of the cross-mode intermediate state conversion protocol;
[0043] Figure 4 This is a schematic diagram of the orchestration decision-making process based on submodal multi-objective optimization and adaptive balance penalty;
[0044] Figure 5 This is a schematic diagram of the structure of a multimodal privacy computing fusion system according to the present invention. Detailed Implementation
[0045] To make the objectives, technical solutions, and advantages of the present invention clearer, the embodiments of the present invention will be described in further detail below with reference to the accompanying drawings.
[0046] Example 1
[0047] like Figure 1 As shown, the present invention provides a multimodal privacy computing fusion method, comprising:
[0048] S1: Obtain the data asset metadata of each participant, perform multi-dimensional feature quantization on the data asset metadata, and obtain the data feature vector.
[0049] Each participating party's local computing node encapsulates the descriptive information of local data assets into data asset metadata and reports it through a preset interface, provided that the data does not leave the domain. The data asset metadata does not contain any original records themselves, but only the data volume, field type, sensitivity classification labels of each attribute column, distribution statistical summary of each numerical attribute, and historical communication bandwidth records. This ensures that the original data does not leave the local computing node, providing the minimum necessary descriptive information for cross-organizational collaboration in decision-making. Edge computing nodes are deployed near each participant. After receiving data asset metadata reported by local computing nodes, they quantify multi-dimensional characteristics from four aspects: sensitivity, computational load, distribution heterogeneity, and communication conditions. The sensitivity index of the overall data is characterized by the mean of the quantified weights of the sensitivity levels of each attribute column, reflecting the urgency of data protection. The computational complexity index is characterized by the relative logarithm of the computational load estimate, reflecting the task's consumption of computing power. The Non-IID distribution index is characterized by a relative measure of the differences in label distribution among participants, reflecting the heterogeneity of data distribution among parties. The communication capacity index is characterized by the relative relationship between participant size and available bandwidth, reflecting the capacity for cross-party communication. These four dimensions, from the orthogonal perspectives of security requirements, computational power requirements, data heterogeneity, and communication conditions, respectively, depict the computing task and together constitute a complete representation of the adaptability to the privacy computing model. Edge computing nodes normalize the four indices, compressing the values of each index to the range of zero to one to eliminate dimensional differences. They then concatenate the values in the order of sensitivity, computational complexity, Non-IID distribution, and communication capacity into a four-dimensional data feature vector, which is then uploaded to the cloud orchestration server as a quantitative input for subsequent privacy computing mode decisions.
[0050] S2: Decompose the data feature vector into each feature dimension and input it into the multi-source robust optimization model. Based on historical feedback, dynamically adjust the reliability parameters of each feature dimension to construct a set of distribution uncertainties. Perform robust mode recommendation on the set of distribution uncertainties and output a set of candidate mode combinations.
[0051] After obtaining the data feature vectors, reliable pattern recommendations need to be made under conditions of noise and distribution drift. The cloud orchestration server decomposes the data feature vectors into four feature dimensions: sensitivity, computational complexity, Non-IID distribution, and communication capacity. These four feature dimensions are then input into a multi-source robust optimization model. The multi-source robust optimization model treats each of the four feature dimensions as an independent information source. The reliability of each information source dynamically changes with the scenario, and a reliability parameter driven by historical recommendation correctness feedback characterizes the confidence level of each feature dimension. Based on the empirical probability distribution and reliability parameter of each feature dimension, a fluctuation range is set for the empirical probability distribution, constituting a set of distribution uncertainties for each feature dimension. Pattern recommendation searches for the worst-case probability distribution that maximizes the expected loss of pattern fit within the intersection of the distribution uncertainty sets of each feature dimension. Under the worst-case probability distribution, it selects the privacy-preserving computation pattern with the minimum expected loss. The core criterion for robust pattern recommendation using the privacy-preserving computation pattern is:
[0052]
[0053] in, For the recommended privacy-preserving computation model, Q represents the candidate probability distribution within the uncertainty set. Let F be the set of distribution uncertainties for the m-th feature dimension, and let F be the data feature vector following the distribution Q. The mode fit loss of the data feature vector under the mode. For expectation operators.
[0054] The core of the worst-case criterion lies in no longer assuming that the feature distribution is completely consistent with the empirical distribution. Instead, it evaluates the suitability of each privacy computation mode under the worst-case distribution in the neighborhood of the empirical distribution, thus making the recommendation results resistant to feature noise and distribution drift. The cloud orchestration server performs robust recommendations on the combination of each participant and each computation task, obtaining the recommended privacy computation mode for each participant's task. Through combination enumeration, the recommended modes of each participant's task are aggregated into several feasible global mode allocation schemes. Each global mode allocation scheme provides the correspondence between each computational subtask and three modes: federated learning, secure multi-party computation, and trusted execution environment. It also includes a comprehensive robustness confidence score calculated by combining the reliability parameters of each feature dimension and the worst-case expected loss, used to characterize the robustness of the scheme under the worst-case distribution. The cloud orchestration server filters and sorts the global mode allocation schemes according to the comprehensive robustness confidence score, eliminating obviously infeasible schemes and retaining several high-confidence schemes, obtaining the candidate mode combination set for further processing in the subsequent unified abstraction and multi-objective optimization orchestration stages.
[0055] S3: According to the preset abstract mapping rules, each privacy computing mode adopted in the candidate mode combination set is mapped to obtain the corresponding unified computing unit abstract model, and the unified computing unit abstract model set is formed by summarizing them.
[0056] After obtaining the candidate pattern combination set, it is necessary to eliminate the heterogeneity in data format, computational logic, and security semantics among the three modes: federated learning, secure multi-party computation, and trusted execution environment. The cloud orchestration server traverses the candidate pattern combination set, extracts the privacy-preserving computation modes used, and maps each privacy-preserving computation mode to a unified computational unit abstract model according to preset abstract mapping rules. The unified computational unit abstract model consists of four parts: input interface, operator description, output interface, and security state. The input interface defines the input data type encoding, data format descriptor, and data volume estimation of the mode; the operator description defines the operator category, operator parameters, and computational complexity estimation of the mode; the output interface defines the output data format of the mode; and the security state defines the security level, integrity proof requirements, and side-channel risk score of the mode. By unifying heterogeneous privacy computing patterns into a four-tuple abstract model with the same structure, the three patterns, which were originally incompatible in terms of data format, computational logic, and security semantics, acquire a unified descriptive form that is alignable, comparable, and composable: the standardization of input and output interfaces eliminates the heterogeneity of data format, the standardization of operator descriptions eliminates the heterogeneity of computational logic, and the standardization of security states eliminates the heterogeneity of security semantics. The cloud orchestration server generates a corresponding unified computational unit abstract model for each privacy computing pattern appearing in the candidate pattern combination set. For multiple occurrences of the same pattern in different subtasks, it instantiates the pattern based on the operator parameters and data volume estimation, and adds each unified computational unit abstract model to the set, summarizing them to form a unified computational unit abstract model set.
[0057] S4: Based on the unified computing unit abstract model set, construct a cross-mode intermediate state data structure and define a state transition function, and generate an intermediate state transition protocol from the cross-mode intermediate state data structure and the state transition function.
[0058] Based on a unified set of abstract models for computing units, a universal carrier and transformation rules are needed to migrate intermediate computation results between different modes. The cloud orchestration server traverses the unified set of abstract models for computing units, extracts the operator attributes and data flow relationships of each model, and constructs a standardized cross-mode intermediate state data structure. This structure includes three types of fields: encrypted gradient vectors in federated learning mode, secret share matrix in secure multi-party computation mode, and enclave session handles in trusted execution environment mode, to carry intermediate computation states in different modes. The cross-mode intermediate state data structure uses a unified encapsulation format to carry intermediate computation states in different modes, enabling the standardized writing and reading of intermediate results generated during computation in any mode, thus providing a universal carrier for the cross-mode migration of intermediate results. The cloud orchestration server further leverages the execution dependencies between the abstract models of each unified computing unit to determine state triggering conditions and state transition logic. State triggering conditions describe the timing of migration from the source mode to the target mode, while state transition logic describes the correspondence between intermediate data in the source mode and target mode representations during migration. The state triggering conditions and state transition logic are mapped to state transition functions that take the source mode, target mode, source intermediate data, and context as inputs and the target intermediate data as output. These state transition functions cover all migration directions between federated learning, secure multi-party computation, and trusted execution environments, ensuring semantic preservation and controllable accuracy of intermediate computation results during migration. Finally, the cloud orchestration server encapsulates the cross-mode intermediate data structure and state transition functions to generate the intermediate state transition protocol, which is then invoked in subsequent fusion computing phases when mode switching conditions are met.
[0059] S5: Input the candidate mode combination set and the unified computing unit abstract model set into the submodal multi-objective optimization model, solve the constrained Pareto optimal solution through the adaptive balance penalty mechanism, and determine the constrained Pareto optimal solution as the multimodal computing orchestration scheme.
[0060] Having obtained the candidate pattern combination set and the unified computing unit abstract model set, optimal decisions regarding pattern selection and resource allocation need to be made under hard constraints of security and compliance. The cloud orchestration server inputs the candidate pattern combination set and the unified computing unit abstract model set into the submodal multi-objective optimization model, constructing monotonic submodal utility functions for the three patterns: Federated Learning (FL), Secure Multi-Party Computation (MPC), and Trusted Execution Environment (TEE). The optimization objective is to maximize the minimum value of the utility functions for the three patterns. The optimization objective is as follows:
[0061]
[0062] in, , , , , These are the sets of disjoint subtasks assigned to federated learning, secure multi-party computation, and trusted execution environments, respectively. , , These are the utility functions corresponding to the three modes. The robust criterion of minimizing the value is adopted, which requires the optimizer to improve the utility level of the three modes simultaneously, without sacrificing safety-related utility.
[0063] The solution process incorporates differential privacy budget constraints, security level constraints, and resource ceiling constraints into the objective function as penalty terms through an adaptive balancing penalty mechanism. This mechanism dynamically adjusts the penalty weights based on the distance of the current solution from the feasible region. Penalties are relaxed when the solution approaches the feasible region to focus on objective optimization, and penalties are strengthened when the solution deviates from the feasible region to pull the search back to the feasible region, thus ensuring that security and compliance constraints are not violated throughout the optimization process. The cloud orchestration server utilizes the diminishing marginal utility property of the submodal utility function under cardinality constraints, employing a greedy iterative selection method to approximate the optimal solution. This yields a constrained Pareto optimal solution that satisfies differential privacy budget constraints, security level constraints, and resource ceiling constraints while maximizing the minimum utility of the three modes. This constrained Pareto optimal solution is then determined as the multimodal computation orchestration scheme.
[0064] S6: Perform resource orchestration and cross-modal computation according to the multimodal computation orchestration scheme. When the mode switching conditions are met, call the intermediate state conversion protocol to complete the intermediate state data conversion and obtain the privacy computation result.
[0065] After obtaining the multimodal computing orchestration scheme, the cloud orchestration server determines the first privacy computing mode in the scheme as the current privacy computing mode. It then orchestrates the computing resources, communication bandwidth, and storage resources of the current privacy computing mode and initiates the computing tasks of the model to be processed on the local computing nodes and edge computing nodes of each participant. The model to be processed is a machine learning model built based on data asset metadata. During execution, the execution status of the computing tasks is continuously monitored, and the accuracy of the model to be processed is obtained. When the accuracy improvement of the model to be processed falls below a minimum threshold in a preset number of iterations, or when the communication latency of the computing task exceeds a preset latency, the mode switching condition is met, and a mode switch is triggered. At this time, the intermediate state data output by the current privacy computing mode is extracted, and an intermediate state conversion protocol is called to parse and map the intermediate state data, converting it into the data structure required by the next privacy computing mode. Computation continues in the next privacy computing mode based on the converted data structure, completing cross-mode computing. During cross-mode switching, intermediate data is losslessly migrated to the next privacy computing mode after being parsed and mapped by the intermediate state conversion protocol. This avoids the resource waste and accuracy loss caused by recalculating from scratch due to mode isolation, enabling the computing pipeline to flexibly select the most suitable privacy computing mode at the subtask level. The process of monitoring, switching, and continuing computation is executed repeatedly. Each switch is triggered by mode switching conditions and the semantic consistency of intermediate results is guaranteed by the intermediate state conversion protocol, until the computation task of the last privacy computing mode in the orchestration scheme is completed. The cloud orchestration server outputs the data from the last privacy computing mode as the privacy computing result, thereby completing a complete closed loop from data asset awareness, robust mode recommendation, unified abstraction and protocol generation, multi-objective optimization orchestration to cross-mode fusion computation, realizing intelligent orchestration and organic integration of multiple privacy computing modes.
[0066] Example 2
[0067] Based on Example 1, the implementation method for obtaining data feature vectors through multi-dimensional feature quantization is further specified, refining the implementation method into three steps: data asset metadata parsing, four-dimensional quantization index calculation, and normalization concatenation. The step of obtaining data asset metadata from each participant and performing multi-dimensional feature quantization on the data asset metadata to obtain data feature vectors includes:
[0068] S1.1: Receive the data asset configuration file uploaded by each participant through a preset interface, and parse it to obtain the data asset metadata.
[0069] Edge computing nodes receive data asset configuration files uploaded by local computing nodes of participating parties through a pre-defined interface. These configuration files record descriptive information about local data assets using structured fields, including data volume, set of field types, sensitivity rating labels for each attribute column, distribution statistical summaries for each numerical attribute, and historical communication bandwidth records. Edge computing nodes perform format validation and field integrity checks on the data asset configuration files, eliminating those missing key fields or with invalid formats. For configuration files that pass validation, they parse each field, restoring the recorded descriptive information into structured data asset metadata. This metadata does not contain any original records; it only retains the minimum necessary summary describing the statistical characteristics and communication conditions of the data assets. During parsing, edge computing nodes perform validity reduction on the sensitivity rating labels, standardizing the sensitivity rating labels for each attribute column to one of four levels: low, medium, high, or very high. They also perform numerical validity checks on the distribution statistical summaries to exclude outliers and aggregate historical communication bandwidth records using time windows to obtain bandwidth estimates reflecting recent communication conditions. For field conflicts or out-of-bounds statistical summaries detected during the parsing process, the edge computing nodes correct or mark them according to the preset reduction rules, and roll back the configuration files that cannot be corrected to the corresponding local computing nodes for re-reporting, thereby ensuring that the data asset metadata entering subsequent calculations meets the same standards in terms of field integrity and value legality.
[0070] S1.2: Extract and calculate the sensitivity index, computational complexity index, Non-IID distribution index, and communication capacity index from the data asset metadata.
[0071] After obtaining the data asset metadata, the edge computing nodes extract the corresponding fields and calculate four quantitative indices to characterize the computing task profile from four orthogonal aspects: security requirements, computing power requirements, data heterogeneity, and communication conditions. The formula for calculating the sensitivity index is:
[0072]
[0073] in, Here, |T| represents the sensitivity index, T is the set of field types, and |T| represents the number of field types. For the sensitivity rating label of the j-th attribute column, The quantization weight function is used to assign sensitivity levels. The values of the quantization weight function are 0.1, 0.4, 0.8, and 1.0 for low, medium, high, and very high sensitivity levels, respectively.
[0074] The formula for calculating the computational complexity index is:
[0075]
[0076] in, To calculate the complexity index, To estimate the floating-point operations required to complete the computational task, This is the preset upper limit of the logarithmic computational load.
[0077] The formula for calculating the Non-IID distribution index is:
[0078]
[0079] in, The index of the Non-IID distribution. and Let be the label distributions of the i-th and j-th participants, respectively. Let KL (Kourbeck-Leibler) be the KL divergence between the two, and N be the data volume.
[0080] The formula for calculating the communication capacity index is:
[0081]
[0082] in, This is a communication capacity index. Here, BW represents the number of participants, SNR represents the channel signal-to-noise ratio, and BW represents the available communication bandwidth. The sensitivity index characterizes the urgency of data protection, the computational complexity index characterizes the computational power consumption of the task, the Non-IID distribution index characterizes the heterogeneity of data distribution among the parties, and the communication capacity index characterizes the cross-party communication carrying capacity. These four indices, from mutually orthogonal perspectives, together constitute a complete representation of the adaptability of the privacy computing model. The absence of any dimension will lead to blind spots in the model profile.
[0083] S1.3: Normalize and map the sensitivity index, computational complexity index, Non-IID distribution index and communication capacity index into feature vectors respectively, and concatenate the feature vectors to form the data feature vector.
[0084] Since the four quantization indices have different dimensions and value ranges, directly concatenating them would lead to the indices with larger value ranges having a disproportionate influence in subsequent decisions. Therefore, the edge computing nodes perform min-max normalization on each of the four indices, linearly mapping each index to a unified range of zero to one based on its historical value range, eliminating the differences in dimensions and making the four dimensions comparable in numerical scale. The normalization mapping uses min-max normalization for the i-th quantization index. The normalization calculation method is as follows:
[0085]
[0086] in, The i-th quantization index is the normalized feature component. Let be the original value of the i-th quantization index. and These are the lower and upper bounds of the historical value range of the i-th quantitative index, respectively, and the normalized result falls within the range of zero to one.
[0087] After normalization, the edge computing nodes expand the four normalization indices into corresponding feature vector components, and then concatenate them in a fixed order of sensitivity, computational complexity, Non-IID distribution, and communication capacity to form a four-dimensional data feature vector. The four-dimensional data feature vector is constructed by sequentially concatenating the four normalized feature components, and its concatenation expression is as follows: ,in, For four-dimensional data feature vectors, the superscript... It is the transpose operator. , , , These are the sensitivity index, computational complexity index, Non-IID distribution index, and communication capacity index, respectively. A fixed concatenation order ensures that subsequent decomposition steps can accurately reconstruct the semantics of each feature dimension based on its position. Edge computing nodes upload the concatenated data feature vectors to the cloud orchestration server as the quantization input for robust mode recommendations. During normalization, edge computing nodes dynamically maintain the historical value ranges of each index. When a new quantization index exceeds the existing range, the mapping boundary is expanded accordingly, ensuring that the normalization result always falls within the zero-to-one range and maintains numerical scale stability as the data scale grows. For the cold start phase, when historical value ranges are insufficient, edge computing nodes use pre-set empirical ranges for transitional mapping, gradually replacing them with measured ranges as feedback samples accumulate. This ensures that the normalization mapping provides comparable feature components in both the initial and stable operation phases of the system.
[0088] Example 3
[0089] like Figure 2 As shown, based on Example 1, the data feature vector is decomposed into various feature dimensions and input into a multi-source robust optimization model. The reliability parameters of each feature dimension are dynamically adjusted based on historical feedback to construct a set of distributional uncertainties. Robust pattern recommendation is then performed on this set of distributional uncertainties, outputting a set of candidate pattern combinations, including:
[0090] S2.1: The data feature vector is decomposed into feature dimensions according to the quantified attribute categories. Each feature dimension is input into the multi-source sub-Bruker optimization model. The accuracy is calculated by combining the obtained historical recommendation correctness feedback. Based on the accuracy, the Wasserstein trust radius of the corresponding feature dimension is dynamically adjusted. The adjusted Wasserstein trust radius is used as the reliability parameter. The feature dimensions include sensitivity index, computational complexity index, Non-IID distribution index, and communication capacity index.
[0091] The cloud orchestration server decomposes the data feature vector into feature dimensions according to four quantified attribute categories: sensitivity index, computational complexity index, Non-IID distribution index, and communication capacity index. Each feature dimension is then input into the multi-source sub-Brussels bar optimization model. The Wasserstein trust radius dynamic adjustment mechanism treats each feature dimension as an independent information source whose reliability dynamically changes with the scenario. Based on the accuracy driven by the historical recommendation correctness feedback of the dimension, the radius of the uncertainty set of the feature dimension is adjusted inversely, so that dimensions with higher reliability obtain a more concentrated distribution estimate. The technical principle of the mechanism is that the historical performance of a feature dimension directly reflects the credibility of the feature dimension for the current decision. Quantifying the credibility into accuracy and shrinking or expanding the uncertainty radius accordingly allows the decision to adaptively tilt towards dimensions with better historical performance without manually setting the weights of each dimension. In implementation, the multi-source sub-Brussels bar optimization model maintains a historical recommendation correctness feedback sequence for each feature dimension and calculates the accuracy of the feature dimension using an exponentially weighted moving average. The accuracy update method for the feature dimension is as follows:
[0092]
[0093] in, Let m be the accuracy of the m-th feature dimension in the t-th recommendation. For the m-th feature dimension in the th... Accuracy rate at the second recommendation stage It is a smoothing coefficient and its value is in the range of zero to one. Let be the indicator value for the m-th feature dimension at the t-th recommendation, set to 1 for correct and 0 for incorrect. The Wasserstein confidence radius for the corresponding feature dimension is dynamically adjusted based on accuracy. The adjustment method for the feature dimension is as follows:
[0094]
[0095] in, Let the Wasserstein trust radius of the m-th feature dimension be the value at the t-th recommendation. Based on the radius of uncertainty, The trust scaling factor is used. Higher accuracy results in a smaller exponent term, a smaller Wasserstein trust radius, a more concentrated distribution estimate of the feature dimension, less space for amplification in worst-case assessment, and a higher actual weight in decision-making. Conversely, dimensions with poor historical performance receive larger radii and more conservative distribution estimates. The adjusted Wasserstein trust radius is used as the reliability parameter for the corresponding feature dimension to obtain the reliability parameter.
[0096] S2.2: Based on the reliability parameters, set the fluctuation range for the empirical probability distribution of each feature dimension, and generate the distribution uncertainty set.
[0097] After obtaining the reliability parameters for each feature dimension, the cloud orchestration server constructs a Wasserstein fuzzy set for each feature dimension, centered on the empirical probability distribution and with the reliability parameters of the feature dimension as the radius. This fuzzy set represents the fluctuation range of the empirical probability distribution of the feature dimension. The fuzzy set for the feature dimension is as follows:
[0098]
[0099] in, Let Q be the set of uncertainties in the distribution of the m-th feature dimension at the t-th recommendation, and let Q be the candidate probability distribution. Let m be the empirical probability distribution of the m-th feature dimension. Let be the second-order Wasserstein distance between the candidate probability distribution and the empirical probability distribution. Let be the Wasserstein trust radius of the m-th feature dimension at the t-th recommendation.
[0100] The reason for using Wasserstein distance instead of other distribution divergences to define the fluctuation range is that Wasserstein distance characterizes the minimum cost required to transform one distribution into another. It is sensitive to the translation and deformation of the distribution support set, thus the constructed fuzzy set can cover the neighborhood shift caused by sampling noise and distribution drift in empirical distributions. Furthermore, the radius of the fuzzy set is directly linked to the aforementioned reliability parameter; the higher the reliability of the dimension, the tighter the fuzzy set. The setting of the fuzzy set radius also needs to balance the requirements of coverage sufficiency and search compactness. If the radius is too large, the worst-case distribution will be overly pessimistic, leading to conservative pattern recommendations. If the radius is too small, it may not be able to cover the real distribution drift and weaken robustness. This mechanism achieves a balance between coverage sufficiency and search compactness by binding the radius to the reliability parameter driven by historical accuracy, thus achieving a scenario-adaptive balance. The cloud orchestration server constructs Wasserstein fuzzy sets for each of the four feature dimensions and aggregates the Wasserstein fuzzy sets of each feature dimension to generate a distribution uncertainty set, which serves as the distribution search space for evaluating the adaptability of each privacy computing mode in the worst case, resulting in the distribution uncertainty set.
[0101] S2.3: Find the worst-case probability distribution in the set of uncertain distributions, perform robust mode recommendation on the worst-case probability distribution, and output the set of candidate mode combinations.
[0102] After obtaining the set of distributional uncertainties, the cloud orchestration server searches for the worst-case probability distribution that maximizes the expected loss of pattern fit within the intersection of the distributional uncertainty sets of each feature dimension. Under this worst-case probability distribution, it selects the privacy-preserving computation pattern with the minimum expected loss. The cloud orchestration server performs worst-case solution and pattern selection for each combination of participants and computational tasks, obtaining a recommended privacy-preserving computation pattern for each participant's task. Through combination enumeration, the recommended patterns for each participant's task are aggregated into several global pattern allocation schemes. A comprehensive robustness confidence score, calculated by combining the reliability parameters of each feature dimension and the expected loss of the worst-case scenario, is added to each global pattern allocation scheme. The schemes are then sorted by their comprehensive robustness confidence scores, and high-confidence schemes are retained to obtain the candidate pattern combination set.
[0103] Taking a cross-hospital consultation scenario as an example, three hospitals hold pathology, imaging, and laboratory data, respectively. Two of the hospitals exhibit strong data heterogeneity, with the Kullback-Leibler divergence of their label distributions reaching 0.43, exceeding the preset heterogeneity threshold of 0.3. The robust mode recommendation provides each hospital with an allocation scheme: the first hospital uses federated learning, the second uses a trusted execution environment, and the third uses a hybrid of federated learning and trusted execution environments. The corresponding comprehensive robustness confidence r = 0.87 indicates that the scheme maintains high robustness even under the most unfavorable distribution within the empirical distribution neighborhood. Compared to a unified federated learning scheme, the second hospital's highly sensitive pathology data is processed separately within the trusted execution environment and does not participate in gradient exchange, meeting ethical compliance requirements. The heterogeneous data from the third hospital is identified and assigned a hybrid mode, thus avoiding the accuracy degradation of pure federated learning under Non-IID conditions, and outputting the candidate mode combination set.
[0104] Example 4
[0105] Based on Example 1, the step of mapping each privacy computing mode used in the candidate mode combination set according to a preset abstract mapping rule to obtain the corresponding unified computing unit abstract model, and summarizing them to form a unified computing unit abstract model set, includes:
[0106] S3.1: Traverse the candidate pattern combination set and extract the privacy computing mode adopted by each candidate pattern combination in the candidate pattern combination set.
[0107] The cloud orchestration server traverses each candidate pattern combination in the candidate pattern combination set, reading the privacy computing patterns assigned to each computational subtask by each candidate pattern combination, and deduplicating and merging the extracted privacy computing patterns to obtain a list of privacy computing patterns that need to be abstracted and mapped. During the traversal, the cloud orchestration server simultaneously records the context in which each privacy computing pattern appears in the candidate pattern combination set, including the operator semantics, data volume estimation, and security level requirements of the computational subtask undertaken by the privacy computing pattern, so as to distinguish instances of the same pattern appearing multiple times in different subtasks. Since different schemes in the candidate pattern combination set may repeatedly use the same privacy computing pattern, deduplication and merging avoid repeated abstract mapping of the same pattern, reducing the computational workload of subsequent abstract modeling; while recording the context ensures that the subsequent instantiation stage can model separately according to their respective operator parameters and data volume estimation, without losing subtask-level difference information due to deduplication. The traversal process proceeds sequentially according to the order of candidate pattern combinations. For each candidate pattern combination, the privacy computing patterns assigned to each subtask are read and registered in the list. When a registered pattern is encountered, only the new occurrence context of the pattern is added without re-registering the pattern itself. Thus, pattern extraction, deduplication and merging, and context recording are completed simultaneously in one traversal, avoiding the additional overhead caused by scanning the candidate pattern combination set multiple times.
[0108] S3.2: Determine the operator type and interface definition corresponding to each privacy computing mode according to the preset abstract mapping rules.
[0109] For each privacy computing mode in the privacy computing mode list, the cloud orchestration server determines the corresponding operator type and interface definition according to preset abstract mapping rules. The operator type describes the category of computational operators for the privacy computing mode, covering four categories: aggregation, transformation, inference, and training, corresponding to four typical computational semantics: parameter aggregation, data format transformation, model forward computation, and model iterative update. The interface definition describes the input and output data interfaces of the privacy computing mode, including three elements: data type encoding, data format descriptor, and data volume estimation. Data type encoding uses integers to identify different data formats; plaintext, federated learning encrypted gradients, secure multi-party computation shares, and trusted execution environment contexts correspond to different encoding values. The data format descriptor records the tensor dimension, encoding method, and quantization precision. The data volume estimation provides an estimate of the data size corresponding to the interface. These three elements together constitute a complete description of the interface definition; the absence of any one element will prevent downstream modes from correctly parsing the output data of upstream modes. The cloud orchestration server determines a unique operator type for each privacy computing mode based on the protocol features and operator semantic mapping relationships registered in the abstract mapping rules, and reduces the heterogeneous input and output specifications of the privacy computing mode to an interface definition with a consistent field organization form.
[0110] S3.3: Based on the operator type and the interface definition, construct a unified computing unit abstract model corresponding to each privacy computing mode, and summarize the unified computing unit abstract models to form the unified computing unit abstract model set.
[0111] Based on the defined operator types and interfaces, the cloud orchestration server constructs a unified computational unit abstract model for each privacy-preserving computation mode. This unified computational unit abstract model consists of four parts: input interface, operator description, output interface, and security state. The input and output interfaces directly adopt the aforementioned interface definitions. The operator description is composed of the aforementioned operator type along with operator parameters and computational complexity estimates. The security state records the mode's security level, integrity proof requirements, and side-channel risk score. Specifically, the security level for the federated learning mode is the computational security level, the security level for the secure multi-party computation mode is the information theory security level, and the security level for the trusted execution environment mode is the hardware-level security level, incorporating the corresponding reduction for side-channel risk. For multiple occurrences of the same privacy-preserving computation mode in different subtasks, the cloud orchestration server instantiates the unified computational unit abstract model based on the aforementioned context of occurrence, using the respective operator parameters and data volume estimates. This ensures that the abstract model maintains both the consistency of the four-tuple structure and the differences at the subtask level. The cloud orchestration server adds the unified computational unit abstract models corresponding to each privacy-preserving computation mode to a set, summarizing them to form a unified computational unit abstract model set.
[0112] Example 5
[0113] Based on implementation 1 and embodiment 4, the step of determining the operator type and interface definition corresponding to each privacy computing mode according to the preset abstract mapping rule includes:
[0114] S3.3.1: According to the preset abstract mapping rules, parse the underlying protocol and execution logic of each privacy computing mode.
[0115] The cloud orchestration server parses the underlying protocol and execution logic of each privacy computation mode according to preset abstract mapping rules. The underlying protocol describes the cryptographic protocols or hardware isolation mechanisms upon which the mode implements data protection, while the execution logic describes the computational process and data flow sequence involved in completing a computation. For the federated learning mode, the underlying protocol is secure aggregation of cryptographic gradients, and the execution logic is an alternating iteration of local training and parameter aggregation. For the secure multi-party computation mode, the underlying protocol is secure computation based on secret sharing, and the execution logic is performing circuit operations on the share domain. For the trusted execution environment mode, the underlying protocol is hardware isolation based on secure enclaves, and the execution logic is loading data within the enclave and performing plaintext operations. The parser outputs a set of protocol feature descriptions and a set of logical feature descriptions for each privacy computation mode. The protocol feature descriptions record the cryptographic primitive categories or hardware isolation levels relied upon by the mode, while the logical feature descriptions record the sequence of computational stages and the data flow patterns at each stage. Both are used together as structured criteria for subsequent comparison with the operator library's discrimination conditions. The cloud orchestration server analyzes the underlying protocols and execution logic item by item, extracting the protocol and logical features of each mode in terms of cryptographic mechanisms, operation processes and data flow order, which serve as the criteria for matching the operator library.
[0116] S3.3.2: Based on the underlying protocol and the execution logic, match with the preset operator library to determine the corresponding operator type.
[0117] After obtaining the underlying protocol and execution logic characteristics of each mode, the cloud orchestration server matches these characteristics with a pre-defined operator library. The operator library pre-registers four types of operators: aggregation, transformation, inference, and training, along with applicable protocol and logic characteristics. Each type of operator is associated with a set of discrimination conditions to characterize which protocol mechanism and execution logic corresponds to which operator type. The cloud orchestration server compares the parsed protocol and logic characteristics one by one with the discrimination conditions of each type of operator in the operator library. For example, if the execution logic of alternating local training and parameter aggregation matches a secure aggregation protocol, it is determined to be a training or aggregation operator; if the execution logic of share-domain circuit operations matches a secret sharing protocol, it is determined to be a transformation or inference operator; and if the execution logic of plaintext operations within an enclave matches a hardware isolation protocol, it is determined to be an inference or training operator based on the computational semantics undertaken by the trusted execution environment mode. When the protocol features and logical features of the same pattern simultaneously match the discrimination conditions of multiple types of operators, the cloud orchestration server preferentially determines the operator type of the pattern based on the dominant computational semantics undertaken by the pattern in the corresponding subtask, and records the secondary computational semantics as the additional attributes of the operator, thereby ensuring that each privacy computation pattern is assigned a unique and clear operator type in the context of each subtask. If the protocol features or logical features fail to match any discrimination conditions, the cloud orchestration server classifies the unmatched logic into the transformation operator as a fallback category and marks it for manual review to avoid the interruption of abstract modeling due to matching failure.
[0118] S3.3.3: Extract the data input and output specifications of each privacy computing mode and standardize them into a unified interface definition.
[0119] After determining the operator type, the cloud orchestration server extracts the data input and output specifications of the privacy computing model, including the type, format, and volume constraints of the input data, as well as the type and format constraints of the output data. It then standardizes the input and output specifications of different models into a unified interface definition. The interface definition uses a consistent field organization to describe the input and output data interfaces. Data type encoding uses integers to identify different data formats such as plaintext, federated learning cryptographic gradients, secure multi-party computation shares, and trusted execution environment contexts. Data format descriptors record tensor dimensions, encoding methods, and quantization precision. Data volume estimation provides an estimate of the size of the data corresponding to the interface. Through standardization, the cryptographic gradients of the federated learning model, the secret shares of the secure multi-party computation model, and the enclave context of the trusted execution environment model are incorporated into the same descriptive framework at the interface level. This ensures that different privacy computing models have a consistent descriptive form at both the input and output interface levels, and the output of any model can be recognized and parsed by downstream models based on the interface definition. During standardization, the cloud orchestration server performs field mapping and dimension alignment on each item of the native input and output specifications of each mode, grouping specification items with the same semantics into the same field of the interface definition, and completing the default items with the default specification of the mode, so that the interface definitions of different modes are completely aligned in terms of field composition and value semantics.
[0120] Example 6
[0121] like Figure 3 As shown, based on Example 1, the step of constructing a cross-mode intermediate state data structure and defining a state transition function based on the unified computing unit abstract model set, and generating an intermediate state transition protocol from the cross-mode intermediate state data structure and the state transition function, includes:
[0122] S4.1: Traverse the set of unified computing unit abstract models and extract the operator attributes and data flow relationships of each unified computing unit abstract model.
[0123] The cloud orchestration server traverses the unified computing unit abstract model set, extracting the operator attributes and data flow relationships of each abstract model. Operator attributes include operator type, operator parameters, and computational complexity estimates, characterizing the computational semantics and resource consumption features of the privacy-preserving computing mode corresponding to the abstract model. Data flow relationships describe the input source and output destination of the abstract model, i.e., which upstream abstract model generates the input data and which downstream abstract model consumes the output data. During the traversal, the cloud orchestration server establishes the input-output connection relationships between abstract models based on the data type encoding and data format descriptors of each abstract model's input and output interfaces, thereby reconstructing the dependency topology of candidate mode combinations in the computation pipeline. During the extraction process, the cloud orchestration server establishes an attribute record for each unified computing unit abstract model, which includes operator category, operator parameters, computational complexity estimate, input source identifier, and output destination identifier. The attribute records are then linked together into a directed dependency graph according to the data flow relationship. The nodes of the directed dependency graph correspond to the unified computing unit abstract model, and the edges correspond to the transmission direction of intermediate results, providing a topological basis for subsequent determination of migration direction and migration timing.
[0124] S4.2: Based on the operator attributes and the data flow relationship, construct the standardized cross-modal intermediate data structure.
[0125] Based on the extracted operator attributes and data flow relationships, the cloud orchestration server constructs a standardized cross-modal intermediate state data structure. This structure uses unified fields to represent intermediate computation states across different modes, including a cryptographic gradient vector field for federated learning, a secret share matrix field for secure multi-party computation, and an enclave session handle field for trusted execution environment. These three types of fields correspond to the native representation of the intermediate results generated during computation in each of the three modes. In addition to these three core fields representing intermediate results, the cross-modal intermediate state data structure also includes metadata fields describing the source mode of the current intermediate state, the data format descriptor, and the integrity check value. This allows intermediate computation results from any mode to be standardized, encapsulated, and parsed, with the source mode and format of the intermediate data determined during parsing based on the metadata fields. The three core fields of the cross-modal intermediate data structure correspond one-to-one with the three privacy computing modes. At any given time, only the core field corresponding to the current source mode is filled, while the other core fields are left empty. The source mode identifier in the metadata field is used to indicate the currently valid core field. The data format descriptor records the tensor dimension, encoding method, and quantization precision of the core field. The integrity check value is used to verify whether the intermediate result has undergone unexpected changes before and after migration. The three types of metadata fields together ensure that the heterogeneous intermediate result is encapsulated in a standardized manner and can be parsed unambiguously.
[0126] S4.3: Utilize the execution dependencies between the abstract models of each unified computing unit to determine the state triggering conditions and state transition logic, map the state triggering conditions to the state transition logic, and define the state transition function.
[0127] The cloud orchestration server utilizes the execution dependencies between the unified computing unit abstract models to determine state triggering conditions and state transition logic. These execution dependencies are represented by a directed dependency graph: nodes in the directed dependency graph correspond to the unified computing unit abstract model, edges correspond to the direction of intermediate result propagation, and the two nodes connected by an edge naturally form a transition candidate pair where the source and target modes are adjacent. The cloud orchestration server traverses each edge of the directed dependency graph, obtains candidate transition directions from the mode labels of the two abstract models connected by the edge, and then determines the six mode transition scenarios for which state transition functions need to be defined based on all candidate transition directions.
[0128] The cloud orchestration server determines state triggering conditions for each candidate migration direction based on observable runtime signals. These conditions are inputted from the four-dimensional data feature vector and runtime monitoring values, and specifically include four judgment dimensions: First, convergence and accuracy dimension: when the accuracy improvement rate falls below a preset lower limit for several consecutive rounds (i.e., the accuracy stagnation criterion is met), migration from the current mode to a more expressive mode is triggered. Second, communication dimension: when the communication capacity index falls below a preset threshold and the communication latency exceeds a preset upper limit, migration to a mode with lower communication overhead is triggered. Third, sensitivity and privacy dimension: when the sensitivity index rises and exceeds a preset protection threshold, migration to a mode with stronger privacy protection is triggered. Fourth, computing resource dimension: when the computing complexity index exceeds the edge-side resource limit, migration to a mode with lower computing power requirements is triggered. These four judgment dimensions are independent of each other; satisfying any one dimension constitutes the state triggering condition for the corresponding direction.
[0129] Based on the cross-modal intermediate state data structure, the cloud orchestration server determines the state transition logic for each candidate migration direction. The state transition logic describes how the core fields of the source modality (encrypted gradient vector field, secret share matrix field, or enclave session handle field) should be transformed into the core fields required by the target modality during migration, and what transformation operator should be used to implement the transformation. The state transition logic is constrained by data format descriptors and integrity verification values to ensure that the semantics of intermediate results are not lost and the quantization accuracy is within a controllable range during the transformation process.
[0130] The cloud orchestration server maps state triggering conditions to state transition logic, forming a correspondence between triggering conditions, target modes, and transformation operators. When a certain judgment dimension meets the triggering condition, the cloud orchestration server selects the target mode according to the correspondence and calls the transformation operator specified by the determined state transition logic under the candidate migration direction to transform the source intermediate data into the target intermediate data. The cloud orchestration server unifies the correspondence and six sets of state transition logic into a unified abstraction of a state transition function with source mode, target mode, source intermediate data, and migration context as inputs and target intermediate data as output. The state transition function covers six migration directions between three modes: federated learning, secure multi-party computation, and trusted execution environment. Since the state transition function must ensure semantic preservation and controllable accuracy of intermediate computation results during migration, corresponding state transition logic and accuracy verification rules are defined for each of the six migration directions, and the state transition function is defined accordingly.
[0131] The cloud orchestration server defines the state transition function as follows:
[0132]
[0133] Where τ is the state transition function. and These are the source pattern label and the target pattern label, respectively (values are taken from the set {FL, MPC, TEE}). C represents the core field of the source intermediate data, which is the cross-mode intermediate data structure corresponding to the source mode. C represents the migration context (which includes the data format descriptor, integrity check value, and quantization precision parameter). The target intermediate state data refers to the core field corresponding to the target mode in the cross-mode intermediate state data structure. The state transition function covers bidirectional conversion between any two of the federated learning mode, secure multi-party computation mode, and trusted execution environment mode.
[0134] The six migration directions are as follows:
[0135] Direction 1: Migration from Federated Learning Mode to Secure Multi-Party Computation Mode: The state transition function first involves each participating party performing threshold decryption on the encrypted gradient vector field in the cross-mode intermediate data structure locally to recover gradient parameter fragments. Then, the gradient parameters are distributed and reconstructed into a secret share matrix field through secret sharing. During the migration process, no single party can obtain the complete plaintext gradient. The reconstruction accuracy of the secret share matrix field is determined by the numerical accuracy of the interpolation operation. The accuracy verification rule is that after the migration is completed, the source encrypted gradient vector field and the gradient parameters restored by inverse sharing are compared based on the integrity verification value. If the relative error does not exceed the preset accuracy threshold, the migration is deemed valid.
[0136] Direction 2: Migration from Federated Learning Mode to Trusted Execution Environment Mode: The state transition function first decrypts the encrypted gradient vector field through a threshold to restore the gradient parameters, and then sends the gradient parameters as input to the enclave session handle field into the trusted execution environment enclave. The enclave completes subsequent calculations under the protection of the enclave session handle field. The accuracy verification rule is to compare the consistency digest of the calculation results in the enclave with the local gradient parameters. If the consistency digest matches, the migration is deemed valid.
[0137] Direction 3: Migration from Secure Multi-Party Computation Mode to Federated Learning Mode: The state transition function first involves the participating parties performing a secret sharing inverse operation on the secret share matrix field to recover the gradient parameters, and then re-encrypting and encapsulating the gradient parameters into an encrypted gradient vector field; the accuracy verification rule is to compare the consistency between the reconstructed gradient parameters and the original fragments of the secret share matrix field, and if they are consistent, the migration is deemed valid.
[0138] Fourthly, the migration from the secure multi-party computation mode to the trusted execution environment mode: The state transition function first performs share reconstruction on the secret share matrix field within the trusted execution environment enclave, recovers the plaintext intermediate result, and loads it into the enclave session handle field to avoid share leakage due to reconstruction outside the enclave; the accuracy verification rule is to compare the share reconstruction result with the original fragment digest within the enclave, and if the digest is consistent, the migration is deemed valid.
[0139] Direction 5: Migration from Trusted Execution Environment Mode to Federated Learning Mode: The state transition function first exports the plaintext intermediate results protected by the enclave session handle field from the enclave, and then encapsulates them into an encrypted gradient vector field for consumption by the federated learning mode; the accuracy verification rule is that the intermediate results are compared with the integrity verification value before and after export, and if no change occurs, the migration is deemed valid.
[0140] Direction 6: Migration from Trusted Execution Environment Mode to Secure Multi-Party Computation Mode: The state transition function first exports the plaintext intermediate result in the enclave session handle field, and then distributes it as a secret share matrix field through secret sharing; the accuracy verification rule is to compare the inverse restoration result of the share matrix with the intermediate result exported from the enclave, and if they are consistent, the migration is deemed valid.
[0141] The state transition logic and precision verification rules for the six migration directions together constitute the complete definition of the state transition function. This allows the fusion calculation stage to directly call the corresponding instantiated migration logic based on the source mode and target mode identifiers when the mode switching conditions are met. The semantically preserved and precision-controllable intermediate state data conversion is completed on the cross-mode intermediate state data structure, resulting in the instantiated definition of the state transition function in each migration direction.
[0142] S4.4: Combine and encapsulate the cross-mode intermediate state data structure and the state transition function to generate the intermediate state transition protocol.
[0143] The cloud orchestration server combines and encapsulates cross-mode intermediate state data structures and state transition functions covering six migration directions to generate an intermediate state transition protocol. During this encapsulation, the cloud orchestration server uses the cross-mode intermediate state data structure as the protocol's data carrier specification and the state transition functions as the protocol's migration behavior specification, establishing a unified calling interface for both. This allows the fusion computing phase to, when the mode switching conditions are met, input the intermediate state data output from the source mode and the target mode identifier through the calling interface. Internally, the protocol selects the corresponding state transition function based on the source and target modes, parses the source intermediate state data according to the cross-mode intermediate state data structure, and generates the target intermediate state data. The intermediate state transition protocol also encapsulates integrity checks and exception rollback logic before and after migration, triggering rollback and retries when the migration result fails the integrity check. The combined and encapsulated intermediate state conversion protocol exposes only a unified calling interface to the outside world, while hiding the specific implementation of each migration direction internally. During the fusion computing phase, the target intermediate state data can be obtained simply by passing in the source intermediate state data and the target mode identifier through the interface. The selection of state conversion functions, intermediate state data parsing, target data generation and integrity verification within the protocol are transparent to the caller, thereby decoupling the implementation details of cross-mode migration from the scheduling logic of fusion computing.
[0144] Example 7
[0145] like Figure 4 As shown, based on Example 1, the step of inputting the candidate mode combination set and the unified computing unit abstract model set into the submodal multi-objective optimization model, solving for the constrained Pareto optimal solution through an adaptive balance penalty mechanism, and determining the constrained Pareto optimal solution as the multimodal computing orchestration scheme includes:
[0146] S5.1: Input the candidate mode combination set and the unified computing unit abstract model set into the sub-modal multi-objective optimization model to construct a multi-objective optimization function and corresponding constraints, including differential privacy budget constraints, security level constraints and resource upper limit constraints.
[0147] The cloud orchestration server inputs the candidate pattern combination set and the unified computing unit abstract model set into the submodal multi-objective optimization model, and takes the set of all unassigned computing subtasks as the whole set to construct the utility function of monotonic submodals for the three modes of federated learning, secure multi-party computation and trusted execution environment respectively.
[0148] The utility function for federated learning is:
[0149]
[0150] The utility function for secure multi-party computation is:
[0151]
[0152] The trusted execution environment utility function is:
[0153]
[0154] in, , , These are the utility functions corresponding to the three modes, where v is the subtask. , , The execution effectiveness of subtasks in the three modes are respectively: , , These are communication overhead, computational overhead, and hardware constraints, respectively. , , These are the corresponding cost weight coefficients. The utility function satisfies the submodular property of diminishing marginal returns, i.e. Here, X and Y are two sets of subtasks where X is contained within Y, z is a newly added subtask not belonging to Y, and f is the utility function of any mode. The multi-objective optimization function aims to maximize the minimum of the three utility functions. The robust criterion of minimization requires the optimizer to simultaneously improve the utility level of the three modes. The corresponding constraints include differential privacy budget constraints, security level constraints, and resource limit constraints. The differential privacy budget constraint requires that the privacy budget consumed during the orchestration process does not exceed the total budget. The security level constraint requires that the security level provided by the mode assigned to each subtask is not lower than the security level required by the subtask. The resource limit constraint requires that the sum of the resource consumption of all modes does not exceed the resource budget limit.
[0155] S5.2: Set and initialize adaptive penalty weights, calculate the degree of violation of the differential privacy budget constraint, the degree of violation of the security level constraint, and the degree of violation of the resource upper limit constraint in the current optimization iteration process, obtain the comprehensive violation degree, and use the comprehensive violation degree to adaptively update the adaptive penalty weights to construct the adaptive balance penalty mechanism.
[0156] In this embodiment, the adaptive balancing penalty mechanism introduces safety and compliance-related constraints into the objective function as penalty terms, and dynamically adjusts the weight of the penalty terms based on the distance of the current solution to the feasible region. When the solution is close to the feasible region, the penalty is relaxed to focus on objective optimization; when the solution deviates from the feasible region, the penalty is strengthened to pull the search back to the feasible region, thus ensuring that constraints are not violated throughout the optimization process. Fixed-weight penalties struggle to balance the dual needs of a large exploration space in the early stages of optimization and strict constraint compliance in the later stages. Linking the penalty weight to a computable lower bound of feasibility allows the penalty intensity to adaptively increase or decrease with the feasibility of the solution, eliminating the need for repeated manual parameter tuning. The cloud orchestration server sets and initializes the adaptive penalty weights, summarizing constraint violations as penalty terms.
[0157]
[0158] in, To constrain violations and penalties, B represents the resource consumption for each mode, and B represents the upper limit of the resource budget. The required security level for subtask v The actual security level provided by the mode assigned to subtask v; the first penalty term characterizes the degree of violation of the resource limit constraint, and the second term characterizes the degree of violation of the security level constraint. Both are measured by the square of the excess amount. The degree of violation of the differential privacy budget constraint is measured by the square of the excess amount of privacy budget consumed by the orchestration process exceeding the total budget. The sum of the three constitutes the comprehensive degree of violation.
[0159] The cloud orchestration server updates the adaptive penalty weights based on the overall degree of violation in the following way:
[0160]
[0161] in, Let be the adaptive penalty weight for the t-th iteration. and These are the lower and upper bounds of the penalty weight, respectively. It is an S-shaped function. Let be the lower bound of the computability of the t-th iteration. As a feasibility threshold, For smooth transition parameters; the lower bound of computable feasibility is defined as follows: ,in, For the current solution The objective function value after adding the penalty term, Let be the current solution in the t-th iteration. To constrain the upper bound estimation of the optimal value of the objective function in the feasible region, the lower bound can be directly calculated from the current solution without explicitly solving the original optimization problem. When the feasibility lower bound is greater than the threshold, the sigmoid function approaches 1 and the penalty weight approaches the upper bound; conversely, the penalty weight approaches the lower bound. This achieves adaptive adjustment, increasing the penalty when the solution deviates from the feasible region and relaxing the penalty when the solution approaches the feasible region, thus constructing the adaptive balance penalty mechanism.
[0162] S5.3: Introduce the adaptive balance penalty mechanism into the submodal optimization algorithm, iteratively solve the multi-objective optimization function with added penalty term, obtain the constrained Pareto optimal solution that satisfies the constraint conditions, and determine the constrained Pareto optimal solution as the multimodal computation orchestration scheme.
[0163] The cloud orchestration server introduces an adaptive balancing penalty mechanism into the submodular optimization algorithm. The penalty term is weighted by the current adaptive penalty weight and incorporated into the multi-objective optimization function, forming the objective function with the penalty term. Utilizing the diminishing marginal utility property of the submodular utility function under cardinality constraints, a greedy iterative selection is employed to approximate the optimal solution. In each iteration, the cloud orchestration server first calculates the computable feasibility lower bound based on the current solution and updates the adaptive penalty weight accordingly. Then, a greedy selection step is performed on the updated objective function with the penalty term, allowing the solution to gradually approach the feasible region as the iteration progresses, with the penalty weight adaptively increasing or decreasing. When the solution falls into the feasible region, the penalty weight tends to the lower bound, and the optimization process focuses on minimizing the utility of the three modes. This iteration is repeated until the solution is stable and satisfies the differential privacy budget constraint, security level constraint, and resource upper limit constraint. The cloud orchestration server obtains a constrained Pareto optimal solution that satisfies all constraints and maximizes the minimum utility of the three modes. The diminishing marginal utility property of the submodular function ensures that the solution has a controllable approximate quality relative to the global optimal solution. This embodiment uses security and compliance as optimization constraints and provides adaptive penalties throughout the optimization process, so that the solution precisely balances computational efficiency and model accuracy while ensuring security. The cloud orchestration server determines the constraint Pareto optimal solution as the multimodal computation orchestration scheme.
[0164] Example 8
[0165] Based on Example 7, the iterative solution of the multi-objective optimization function with added penalty terms to obtain the constrained Pareto optimal solution that satisfies the constraints includes:
[0166] S5.3.1: Set the orchestration set and initialize it to an empty set. For each candidate pattern combination in the candidate pattern combination set, calculate the minimum marginal gain for selecting the candidate pattern combination into the orchestration set.
[0167] The cloud orchestration server sets up an orchestration set and initializes it to an empty set. In each iteration, for each candidate pattern combination that has not yet been selected in the candidate pattern combination set, the cloud orchestration server calculates the minimum marginal gain resulting from selecting the candidate pattern combination into the orchestration set. The minimum marginal gain is given by the following formula:
[0168] in, Let v be the minimum marginal gain of the candidate pattern combination v relative to the current orchestration set S, where S is the current orchestration set. , , , , These are the sets of disjoint subtasks assigned to federated learning, secure multi-party computation, and trusted execution environments, respectively. , , These are the utility functions corresponding to the three modes. Specifically, at the beginning of each iteration, the cloud orchestration server calculates the increments of the federated learning utility function, the secure multi-party computation utility function, and the trusted execution environment utility function for each candidate mode combination that has not yet been selected in the candidate mode combination set. The minimum value among these three is taken as the minimum marginal gain. All three utility functions satisfy monotonicity and submodularity, and each increment is non-negative and decreases as the orchestration set expands. Taking the minimum of the utility increments of the three modes corresponds to a robust selection strategy prioritizing the worst-case scenario. This ensures that each selection prioritizes improving the utility of the weakest mode, preventing the utility of any mode from being continuously ignored during the selection process, thus maintaining consistency with the optimization objective of maximizing the minimum value.
[0169] S5.3.2: Under the differential privacy budget constraint, an exponential mechanism is introduced to calculate the selection probability of each candidate mode combination based on the minimum marginal gain.
[0170] To meet the differential privacy budget constraint during the selection process, the cloud orchestration server introduces an exponential mechanism to calculate the selection probability of each candidate pattern combination based on the minimum marginal gain:
[0171]
[0172] in, For differential privacy budget parameters, Let v be the minimum marginal gain of the candidate pattern combination v relative to the current orchestration set S. The sensitivity bound is the global maximum marginal gain, and k is the maximum number of selections under the cardinality constraint. The exponential mechanism is introduced to increase the probability of selecting candidate pattern combinations with larger marginal gains, thus approximating the effect of greedy selection in an expected sense. Simultaneously, it provides differential privacy protection for the selection process itself through a probabilistic selection method, preventing the selection result from directly leaking sensitive information of individual subtasks. The sensitivity bound in the exponential mechanism is taken as the global maximum marginal gain, used to characterize the maximum impact of changes in individual subtasks on the marginal gain. A larger sensitivity bound results in a smoother distribution of selection probability and stronger randomness under the same privacy budget. The maximum number of selections under the cardinality constraint is used to distribute the limited privacy budget across multiple rounds of selection, ensuring that the privacy budget consumed in each round matches the total number of rounds, thereby guaranteeing that the sum of the privacy budget consumed in the entire iterative selection process does not exceed the total amount of the differential privacy budget constraint.
[0173] S5.3.3: By iteratively performing probabilistic greedy selection, the candidate pattern combinations are gradually added to the orchestration set according to the selection probability.
[0174] The cloud orchestration server performs a probabilistic greedy selection based on the selection probability. It extracts a candidate pattern combination according to the selection probability distribution and adds it to the orchestration set. Then, it updates the utility function values of each pattern, the remaining differential privacy budget, and the current occupancy of each constraint term before proceeding to the next iteration. Each iteration recalculates the minimum marginal gain and selection probability of the unselected candidate pattern combinations, thus dynamically responding to changes in the orchestration set: as the orchestration set expands, the marginal gain of each candidate pattern combination gradually decreases due to submodular properties, and the selection probability distribution adjusts accordingly. Subsequent selections still focus on candidate pattern combinations that can still bring significant marginal gains. After each round of candidate pattern combinations is extracted, the cloud orchestration server immediately removes the candidate pattern combination from the candidate set and adds it to the orchestration set. Simultaneously, it deducts the differential privacy budget consumed by the combination, updates the current value of each pattern utility function, and accumulates the current occupancy of each constraint term. Then, based on the updated orchestration set, it recalculates the minimum marginal gain and selection probability of the candidate pattern combinations that have not yet been selected. For candidate pattern combinations that would cause any constraint to exceed the limit after being added, the cloud orchestration server sets the selection probability of the candidate pattern combination to zero in the current round to exclude it from the candidate range, thereby ensuring that each step of selection does not violate the feasibility of the constraints.
[0175] S5.3.4: When the orchestration set satisfies the constraints and the minimum marginal gain of the candidate pattern combination is lower than a preset threshold, the orchestration set is output as the constrained Pareto optimal solution.
[0176] After each round of probabilistic greedy selection, the cloud orchestration server checks the termination condition. The iteration terminates when the orchestration set has satisfied the differential privacy budget constraint, security level constraint, and resource limit constraint, and the minimum marginal gain of all remaining candidate pattern combinations is below a preset threshold, meaning further selection will not yield any effective benefit. Setting a minimum marginal gain threshold aims to avoid consuming additional differential privacy budget and resources for minor utility improvements, thus achieving a balance between gains and costs in the orchestration set.
[0177] The termination condition considers both constraint satisfaction and marginal benefit. Constraint satisfaction requires the orchestration set to simultaneously meet differential privacy budget constraints, security level constraints, and resource cap constraints. Marginal benefit requires that the minimum marginal gain of all candidate mode combinations not yet selected is below a preset threshold. Iteration terminates only when both conditions are met simultaneously, thus avoiding premature stopping due to unmet constraints and preventing the continuous consumption of budget and resources for small gains. Upon termination, the cloud orchestration server outputs the current orchestration set as the constrained Pareto optimal solution. The diminishing marginal returns of the submodular function ensure that the solution has a controllable approximate quality relative to the global optimal solution. If the minimum marginal gain of each candidate mode combination is generally below the threshold before the constraints are satisfied, the cloud orchestration server relaxes the constraints near the threshold to prioritize satisfying them, ensuring the feasibility of the output solution, and outputs the orchestration set as the constrained Pareto optimal solution.
[0178] Example 9
[0179] Before inputting the candidate pattern combination set and the unified computing unit abstract model set into the submodal multi-objective optimization model, the method further includes:
[0180] S4.1': Obtain the semantic annotations of each attribute column in the data asset metadata, calculate the correlation degree between each attribute column based on conditional mutual information, evaluate the inferability between each attribute column based on the correlation degree to conduct inference risk assessment, and obtain attribute association risk parameters.
[0181] The adaptive privacy budget allocation mechanism refers to the differentiated allocation of a limited total privacy budget to each attribute column based on the risk differences inferred from other attribute columns. This ensures that attribute columns with higher inferred risk receive stricter protection, while attribute columns with higher value receive more adequate availability. The technical principle behind this mechanism is that uniformly allocating the privacy budget can lead to insufficient protection for high-risk attributes and excessive protection for low-risk attributes. By quantifying the inferability between attribute columns using conditional mutual information, the budget can be allocated differentially, improving the overall balance between privacy protection and data utility under the same total budget. In implementation, the cloud orchestration server obtains the semantic annotations of each attribute column from the data asset metadata. These semantic annotations identify the category and value of each attribute column in terms of business semantics. Subsequently, the correlation between each attribute column is calculated based on conditional mutual information. Measurement, in which, Given that the other attribute columns Z are known, the attribute column... With attribute column Mutual information between conditions For the a-th attribute column, Let Z be the b-th attribute column, and Z be the set of all other attribute columns besides the two attribute columns. The greater the conditional mutual information, the stronger the inferability of the two attribute columns when the other attributes are known. The cloud orchestration server evaluates the inferability of each attribute column based on the correlation between each attribute column, performs inference risk assessment, and uses the sum of the conditional mutual information between each attribute column and the other attribute columns as the attribute association risk parameter to characterize the inference risk of the attribute column.
[0182] S4.2': Combining the attribute-related risk parameters and the preset initial privacy budget, establish a privacy budget allocation model with the privacy budget allocation ratio of each attribute column as the optimization variable and the data utility maximization as the objective function.
[0183] The cloud orchestration server combines attribute-related risk parameters and a preset initial privacy budget to establish a privacy budget allocation model. The privacy budget allocation model uses the privacy budget allocation ratio of each attribute column as the optimization variable, maximizing data utility as the objective function, and the total privacy budget as a constraint. The optimization objective is: And satisfy ,in, For the privacy budget allocated to the a-th attribute column, List the a-th attribute in the privacy budget The data utility below The total privacy budget is calculated. The data utility function is shaped based on the aforementioned attribute-associated risk parameters, making the data utility of attribute columns with high inferred risk less sensitive to budget increases, thus obtaining a relatively limited budget and being subject to stricter protection during optimization. Conversely, the data utility of attribute columns with high value and controllable inferred risk is more sensitive to budget increases, thus obtaining a relatively sufficient budget and maintaining high availability during optimization.
[0184] S4.3': Solve the privacy budget allocation model using Lagrange duality to obtain the optimal solution for the optimization variables, and use the optimal solution as a differentiated adaptive privacy budget allocation scheme.
[0185] The cloud orchestration server solves the privacy budget allocation model using Lagrange duality. During the solution process, the cloud orchestration server incorporates the total privacy budget constraint into the objective function using Lagrange multipliers to construct the corresponding Lagrange function, which is:
[0186]
[0187] in, To list privacy budgets by attribute With Lagrange multipliers The Lagrange function of the variable. The Lagrange multiplier corresponding to the total privacy budget constraint is equal to the marginal value of the data utility of each attribute column, i.e., the shadow price, at the optimal solution.
[0188] Taking the partial derivative of the privacy budget with respect to each attribute column yields the optimality condition. The optimality condition states that at the optimal solution, the marginal utility of each attribute column's data with respect to the budget is equal and equal to the Lagrange multiplier, i.e.:
[0189]
[0190] in, For the a-th attribute column in the optimal budget The marginal value of the data utility at that location relative to the budget. For the optimal privacy budget of the a-th attribute column, The shadow price is given; the condition indicates that the budget is allocated to a point where the global marginal utility is balanced. Subsequently, the cloud orchestration server performs a one-dimensional search of the Lagrange multipliers to ensure that the total budget constraint is satisfied, thereby efficiently finding the optimal solution to the original problem in the dual space and avoiding direct search in the high-dimensional original variable space.
[0191] The values of the Lagrange multipliers reflect the marginal data utility that a unit privacy budget can bring under optimal allocation. During one-dimensional search, the cloud orchestration server starts with the initial values of the multipliers and adjusts the multiplier values based on the difference between the sum of the optimal budgets for each attribute column under the current multiplier and the total budget. If the difference is positive, the multiplier is increased to tighten the budget; if the difference is negative, the multiplier is decreased to loosen the budget. This adjustment is repeated until the sum of the budgets converges to the total privacy budget, thus obtaining the optimal solution that satisfies the constraints. The optimal solution obtained ensures that the critical attribute columns with high sensitivity and value receive a more sufficient privacy budget, and the general attribute columns receive stricter privacy protection. The cloud orchestration server uses the optimal solution as a differentiated adaptive privacy budget allocation scheme.
[0192] S4.4': The adaptive privacy budget allocation scheme is used as a constraint and introduced into the submodal multi-objective optimization model for solution to obtain the multimodal computation orchestration scheme.
[0193] The cloud orchestration server introduces an adaptive privacy budget allocation scheme as a constraint into the submodal multi-objective optimization model. This ensures that the privacy budget consumed by each attribute column during orchestration does not exceed the budget share determined by the allocation scheme for each attribute column, thus transforming data-level differentiated privacy protection requirements into executable constraints in the optimization solution phase. After its introduction, the submodal multi-objective optimization model, while satisfying differential privacy budget constraints, security level constraints, and resource ceiling constraints, is additionally subject to attribute-by-attribute budget constraints from the adaptive privacy budget allocation scheme. The total amount of differential privacy budget constraints is determined by the sum of the budget shares of each attribute in the adaptive privacy budget allocation scheme, while the security level constraints and resource ceiling constraints remain unchanged. With the introduction of the adaptive privacy budget allocation scheme, the differential privacy budget constraint of the submodal multi-objective optimization model is refined from a global single budget constraint to an attribute-by-attribute budget constraint. The privacy budget consumed by each attribute column during orchestration must not exceed the budget share allocated to each attribute column, thus precisely implementing data-level differentiated protection requirements into every step of the optimization solution. The cloud orchestration server performs submodal multi-objective optimization under constraints to obtain an orchestration scheme that satisfies differentiated privacy budget allocation, security level and resource limits, and maximizes the minimum utility of the three modes. At the data level, differentiated privacy budget allocation ensures the availability of critical attributes and the protection strength of general attributes, while at the algorithm level, submodal multi-objective optimization ensures a trade-off between security compliance and efficiency accuracy. This two-level collaboration enables the orchestration scheme to take into account both data availability and security compliance. The submodal multi-objective optimization model is introduced for solving to obtain the multimodal computation orchestration scheme.
[0194] Example 10
[0195] Based on Example 1, the step of performing resource orchestration and cross-modal computation according to the multimodal computation orchestration scheme, and calling the intermediate state conversion protocol to complete intermediate state data conversion when the mode switching conditions are met, to obtain privacy computation results, includes:
[0196] S6.1: Determine the first privacy computing mode in the multimodal computing orchestration scheme as the current privacy computing mode, perform resource orchestration on the current privacy computing mode and start the computing task of the model to be processed, wherein the model to be processed is a machine learning model constructed based on the data asset metadata.
[0197] The cloud orchestration server identifies the first privacy-preserving computing mode in the multimodal computing orchestration scheme as the current privacy-preserving computing mode. It then orchestrates resources for this mode, allocating computing resources, communication bandwidth, and storage resources to the local and edge computing nodes of each participant according to the scheme. During resource orchestration, the cloud orchestration server determines the computing resource quotas for each local computing node, the reserved communication bandwidth between participants, and the storage space for intermediate results based on the operator attributes and data volume estimates of the subtasks undertaken by the current privacy-preserving computing mode. It then issues orchestration instructions to the corresponding nodes. After the resource orchestration instructions are issued, each local and edge computing node completes the reservation of computing resources, the establishment of communication links, and the allocation of storage space according to its quota, and reports its readiness status to the cloud orchestration server. The cloud orchestration server only starts the first round of computing tasks for the model to be processed after confirming that all relevant nodes are ready, avoiding computing task startup failure or mid-process blockage due to the inability of individual nodes to have ready resources. After the resource orchestration is completed, the cloud orchestration server starts the computing tasks of the models to be processed on the local computing nodes and edge computing nodes of each participant. The models to be processed are machine learning models built based on the metadata of data assets. The model structure and training objectives of the models to be processed are pre-set according to the joint analysis task.
[0198] S6.2: Monitor the execution status of the computing task and obtain the accuracy of the model to be processed. When the accuracy of the model to be processed is lower than the minimum threshold in a series of preset iterations, or the communication delay of the computing task exceeds the preset delay, determine that the mode switching condition is met and trigger the mode switching.
[0199] The cloud orchestration server continuously monitors the execution status of computational tasks and periodically obtains the accuracy of the model to be processed. During the monitoring process, the cloud orchestration server, on the one hand, calculates the accuracy improvement of the model to be processed over a preset number of iterations, and on the other hand, measures the communication latency of the computational tasks. These two criteria can be formalized as follows: the moving average of the accuracy increment over H consecutive iterations is lower than a minimum threshold, i.e. ; and the round-trip delay of a single round of parameter exchange exceeds the preset delay limit, i.e. A switch is triggered immediately upon the fulfillment of any criterion, denoted as... Where H is the size of the consecutive round window used to determine when the accuracy stalls. For the first The accuracy of the model to be processed after rounds of iteration. Let w be the iteration number of the w-th round within the window. To improve accuracy, the minimum threshold is required. The round-trip communication delay for a single round of parameter exchange between the participating parties. To preset the upper limit of communication latency, For precision stall trigger indication, the value is 1 when it is true. The communication bottleneck trigger indicator is set to 1 when it is true. The accuracy improvement is characterized by the sliding statistics of the accuracy increment of the model under test in a preset number of iterations. When the sliding statistics are continuously lower than the minimum threshold, the model convergence is considered to be slowing down. The communication latency is measured by the round-trip latency of parameter exchange between each participant in a single round. When the latency exceeds the preset latency, communication is considered to be a bottleneck. The two criteria are independent of each other. The success of either criterion triggers a switch, so that the switching decision can respond to both convergence stagnation and communication limitation. When the accuracy improvement of the model under test is lower than the minimum threshold in a preset number of iterations, it indicates that the model convergence is slowing down in the current mode and continuing to calculate in the current mode will not bring effective accuracy gains; or the communication latency of the calculation task exceeds the preset latency, it indicates that communication has become a bottleneck in the current mode and the communication cost of continuing to calculate in the current mode is too high. The cloud orchestration server determines that the mode switching condition is met and triggers the mode switch.
[0200] S6.3: Obtain the intermediate state data output by the current privacy computing mode, call the intermediate state conversion protocol to parse and map the intermediate state data, convert it into the data structure required by the next privacy computing mode, and complete the intermediate state data conversion.
[0201] After the switch is triggered, the cloud orchestration server pauses the computation pipeline of the current privacy computation mode and saves a computation snapshot. It then obtains the intermediate state data output by the current privacy computation mode and calls the intermediate state conversion protocol to parse and map the intermediate state data. During the call, the cloud orchestration server passes the intermediate state data output by the current privacy computation mode and the identifier of the next privacy computation mode to the intermediate state conversion protocol. The intermediate state conversion protocol selects the corresponding state transition function based on the source and target modes, parses the source intermediate state data according to the cross-mode intermediate state data structure, and maps the source intermediate state data to the data structure required by the next privacy computation mode. Taking the switch from federated learning mode to secure multi-party computation mode as an example, the intermediate state conversion protocol decrypts the encrypted gradient vector output by the federated learning mode through threshold decryption to restore the gradient parameters, and then distributes them into a secret share matrix through secret sharing. After mapping, the integrity check value of the metadata field verifies the conversion result. When the current privacy computing mode is paused, the cloud orchestration server saves a calculation snapshot containing the current model parameters, iteration rounds, and intermediate statistics so that it can roll back to the state before the switch if the transition fails. When the intermediate state transition protocol is called, the protocol automatically selects the corresponding state transition function from the six transition directions based on the passed source mode and target mode identifiers, without the caller needing to be aware of the specific transition implementation.
[0202] S6.4: Based on the data structure, start the computation task of the next privacy computing mode to complete the cross-mode computation.
[0203] Based on the transformed data structure, the cloud orchestration server restores the computing pipeline and starts computing tasks in the next privacy computing mode, determines the next privacy computing mode as the new current privacy computing mode, and re-orchestrates the resources of the mode according to the operator attributes and data volume estimation of the subtasks undertaken by the mode. Then, the computing tasks of the model to be processed continue to be executed on the local computing nodes and edge computing nodes of each participant, completing cross-mode computing.
[0204] Taking a provincial regional medical platform's joint statistical analysis of 500,000 de-identified medical records of diabetic complications conducted by eight hospitals as an example, the initial training of the logistic regression model was performed using a federated learning model. Due to the significant difference in the number of cases between tertiary and primary hospitals and the extremely uneven data distribution, the accuracy of the federated learning aggregation gradient stagnated after the fifteenth iteration. Monitoring determined that the mode switching conditions were met and triggered the switch. An intermediate state conversion protocol was invoked to convert the encrypted gradient vector output by the federated learning model into a secret share matrix via threshold decryption and secret sharing with Shamir (3,8). Then, under a secure multi-party computation mode reorganized into a trusted multi-party execution mode, the odds ratio (OR) values and 95% confidence intervals of each risk factor were calculated using a secure comparison circuit. The deviation of model parameters before and after the switch was much smaller than the tolerance threshold, verifying the parameter consistency of cross-mode migration. After resuming computation, the secure multi-party computation mode continued to complete subsequent risk factor analysis in the share domain without reloading the original medical record data. The data structure based on lossless migration continued computation in the next privacy computation mode, enabling the computation pipeline to flexibly select the most suitable privacy computation mode at the sub-task level to complete cross-mode computation.
[0205] S6.5: Repeat the step of monitoring the execution status of the computing task until the cross-modal computing is completed, until the computing task of the last privacy computing mode in the multimodal computing orchestration scheme is completed, and output the data output by the last privacy computing mode as the privacy computing result.
[0206] The cloud orchestration server repeatedly monitors the execution status of computation tasks, determines mode switching conditions, completes intermediate data transformation, and continues computation for the new current privacy computation mode. Each switch is triggered by mode switching conditions, and the semantic consistency of intermediate results is guaranteed by the intermediate state transformation protocol. This monitoring, determination, transformation, and continued computation process is repeated, allowing the computation pipeline to switch between federated learning, secure multi-party computation, and trusted execution environment modes as needed according to the orchestration scheme, completing the sub-tasks undertaken by each privacy computation mode in the scheme one by one. After each mode switch, the cloud orchestration server incorporates the new current privacy computation mode into the monitoring loop, repeatedly monitoring accuracy and latency, determining switching conditions, and transforming intermediate data until all privacy computation modes in the orchestration scheme have been executed sequentially. If adjacent sub-tasks in the orchestration scheme are assigned to the same privacy computation mode, they can be executed continuously without triggering a switch, thus incurring switching overhead only when a mode change is necessary. When the computation task of the last privacy computation mode in the multimodal computation orchestration scheme is completed, the cloud orchestration server outputs the data from the last privacy computation mode as the privacy computation result.
[0207] Example 11
[0208] After the intermediate state data conversion is completed by calling the intermediate state conversion protocol, the process also includes:
[0209] Step A: Obtain the parameters of the model to be processed corresponding to the intermediate state data before and after switching between the current privacy computing mode and the next privacy computing mode, and use them as parameters to be verified.
[0210] After completing the intermediate data transformation, the cloud orchestration server obtains the model parameters to be processed corresponding to the intermediate data output by the current privacy computing mode before the switch, and the model parameters to be processed recovered from the transformation result by the next privacy computing mode after the switch. These two sets of model parameters are used as parameters to be verified. When obtaining parameters before the switch, the cloud orchestration server reads the model parameters at the time of the switch from the computation snapshot saved by the current privacy computing mode; when obtaining parameters after the switch, the cloud orchestration server reads the corresponding parameters from the initial model parameters obtained by recovering the computation pipeline based on the transformed data structure of the next privacy computing mode. To ensure that the two sets of model parameters have alignable semantics during comparison, the cloud orchestration server maps the model parameters before and after the switch to the same arrangement order in the same parameter space according to the interface definition of the unified computing unit abstract model, ensuring that the same model component is located in the corresponding position in both sets of parameters. For parameter components that are inconsistent in representation granularity due to mode differences, they are aligned with precision according to the data format descriptor in the interface definition before being included in the comparison, thereby eliminating spurious biases caused by differences in parameter arrangement or precision representation.
[0211] Step B: Verify the consistency of the parameter to be verified by using Chebyshev's inequality to probabilistically constrain the deviation of the parameter to be verified to be within the tolerance range.
[0212] The cloud orchestration server calculates the deviation of the parameters to be verified before and after the switch, and performs consistency verification on the deviation. Chebyshev's inequality is used to constrain the probability distribution of the deviation, ensuring that the probability of the deviation falling outside the tolerance range is controlled below a preset level. The reason for using Chebyshev's inequality is that the specific probability distribution of the model parameter deviation before and after a cross-mode switch is often difficult to model precisely. Chebyshev's inequality only relies on the mean and variance of the deviation to provide an upper bound on the probability of the deviation falling outside the tolerance range, without requiring additional assumptions about the shape of the deviation distribution. Therefore, it is suitable for scenarios with unknown distributions, such as cross-mode transitions. The verification process uses the mean and variance of the deviation as the only required statistics, making the parameter consistency determination feasible and stable even when the distribution shape is unknown. The cloud orchestration server estimates the mean and variance of the deviation based on deviation samples accumulated from multiple transitions before and after the switch, and substitutes them into the probability constraints to calculate the upper bound on the probability of the deviation exceeding the tolerance threshold.
[0213] Taking the accumulated deviation sample from 50 consecutive cross-mode switchings as an example, the sample shows a mean of 0.010 and a standard deviation of 0.015, with the actual deviation distribution interval (i.e., deviation range) being [-0.038, 0.041]. Let the tolerance threshold be 0.050, the tolerance range (the allowable deviation interval for consistent judgment parameters) be [-0.050, 0.050], and the preset level (the upper bound of the allowed probability) be 0.200. According to Chebyshev's inequality, the upper bound of the probability that the deviation exceeds the tolerance range is (0.015). 2 / (0.050-0.010) 2 =0.000225 / 0.0016≈0.140625, and the upper bound of 0.141 is lower than the preset level of 0.200, satisfying the consistency criterion. When the probability of the deviation falling outside the tolerance range is not higher than the preset level, it is determined that the parameters to be verified before and after the switch are consistent, and the cross-mode conversion has not introduced unacceptable parameter drift. The computation pipeline continues to execute in the next privacy computation mode. Otherwise, it is determined that the current intermediate state data conversion has introduced parameter drift exceeding the tolerance, triggering a rollback and re-conversion. The intermediate state conversion protocol is called again and the conversion parameters are adjusted until the probability of the deviation falling outside the tolerance range is constrained to below the preset level. The deviation of the parameters to be verified is probabilistically constrained within the tolerance range by Chebyshev inequality.
[0214] Example 12
[0215] The multimodal privacy computing fusion system provided in this embodiment includes a feature quantization module, a pattern recommendation module, an abstract mapping module, a protocol generation module, a scheme optimization module, and a fusion computing module. Each module is deployed in an implementation environment that includes local computing nodes, edge computing nodes, and cloud orchestration servers, and the modules are connected sequentially by data flow.
[0216] like Figure 5 As shown, the present invention provides a multimodal privacy computing fusion system, comprising:
[0217] The feature quantization module 501 is used to acquire data asset metadata from each participant and perform multi-dimensional feature quantization on the data asset metadata to obtain a data feature vector. The feature quantization module 501 receives data asset metadata reported by the local computing nodes of each participant through a preset interface. This data asset metadata only includes descriptive content such as data volume, field type, sensitivity classification labels for each attribute column, statistical summaries of numerical attribute distribution, and historical communication bandwidth records, and does not contain any original records themselves. The feature quantization module 501 characterizes the overall data sensitivity index, the task's computational complexity index, the Non-IID distribution index of the heterogeneity of data distribution among parties, and the communication capacity index of cross-party communication carrying capacity from four orthogonal aspects: sensitivity, computational load, distribution heterogeneity, and communication conditions. Subsequently, the four indices are normalized to eliminate dimensional differences and concatenated in a fixed order into a four-dimensional data feature vector. This data feature vector is then uploaded to a cloud orchestration server as the quantization input for subsequent robust mode recommendations.
[0218] The pattern recommendation module 502 is used to decompose the data feature vector into various feature dimensions and input them into a multi-source robust optimization model. Based on historical feedback, it dynamically adjusts the reliability parameters of each feature dimension to construct a set of distributed uncertainty. Robust pattern recommendation is then performed on this set of distributed uncertainty, outputting a candidate pattern combination set. The pattern recommendation module 502 takes the data feature vector output by the feature quantization module as input, decomposes it into four feature dimensions: sensitivity, computational complexity, Non-IID distribution, and communication capacity, and inputs these four feature dimensions into the multi-source robust optimization model. The multi-source robust optimization model treats each of the four feature dimensions as a dynamically changing information source, characterizes its reliability parameters based on the historical recommendation correctness feedback of each information source, and sets a fluctuation range for the empirical probability distribution of each information source, thus forming a set of distributed uncertainty. Subsequently, robust pattern recommendation is performed within this set of distributed uncertainty based on the worst-case probability distribution criterion, obtaining the privacy computation patterns recommended by each participating party's task one by one. These patterns are then combined and enumerated into several feasible global pattern allocation schemes, and after screening and sorting, a candidate pattern combination set is obtained and output.
[0219] The abstract mapping module 503 is used to map each privacy computing mode used in the candidate mode combination set according to a preset abstract mapping rule, thereby obtaining a corresponding unified computing unit abstract model, which is then aggregated to form a unified computing unit abstract model set. The abstract mapping module 503 takes the candidate mode combination set as input, traverses each privacy computing mode used within it, and maps each privacy computing mode to a unified computing unit abstract model according to the preset abstract mapping rule. The unified computing unit abstract model consists of four parts: input interface, operator description, output interface, and security state. It standardizes the input data type and format, operator category and complexity, output data format, security level, and side-channel risk score for each mode. Through this mapping, the three originally unrelated modes obtain a unified description form that is alignable, comparable, and combinable. The abstract mapping module 503 generates a corresponding unified computing unit abstract model for each privacy computing mode appearing in the candidate mode combination, and then aggregates and outputs the unified computing unit abstract model set.
[0220] The protocol generation module 504 is used to construct a cross-mode intermediate state data structure and define state transition functions based on the unified computing unit abstract model set. An intermediate state transition protocol is generated from the cross-mode intermediate state data structure and the state transition functions. The protocol generation module 504 takes the unified computing unit abstract model set as input, traverses the operator attributes and data flow relationships of each unified computing unit abstract model, and constructs a standardized cross-mode intermediate state data structure. This cross-mode intermediate state data structure includes three types of fields: a cryptographic gradient vector in the federated learning mode, a secret share matrix in the secure multi-party computation mode, and an enclave session handle in the trusted execution environment mode. These fields are encapsulated in a unified format to carry the intermediate computation states of different modes. The protocol generation module 504 further determines the state triggering conditions and state transition logic based on the execution dependencies between the unified computing unit abstract models, and maps and defines state transition functions that cover all migration directions between the three modes, ensuring semantic preservation and controllable precision. Finally, the cross-mode intermediate state data structure and the state transition functions are combined and encapsulated to generate the intermediate state transition protocol and output it.
[0221] The scheme optimization module 505 is used to input the candidate mode combination set and the unified computing unit abstract model set into the submodal multi-objective optimization model, solve for the constrained Pareto optimal solution through an adaptive balance penalty mechanism, and determine the constrained Pareto optimal solution as the multimodal computing orchestration scheme. The scheme optimization module 505 takes the candidate mode combination set and the unified computing unit abstract model set as input, constructs monotonic submodal utility functions for the three modes of federated learning, secure multi-party computation, and trusted execution environment, and aims to maximize the minimum value of the utility functions of the three modes. This ensures that the optimization process must simultaneously improve the utility levels of the three modes without sacrificing security-related utilities. During the solution process, the scheme optimization module 505 introduces the differential privacy budget constraint, security level constraint, and resource upper limit constraint into the objective function as penalty terms through an adaptive balancing penalty mechanism. It dynamically adjusts the penalty weight based on the distance of the current solution to the feasible region. Under the premise that the constraints are not violated, it uses the marginal diminishing property of the submodal function to greedily iterate and approximate the optimum. The solution obtains a constrained Pareto optimal solution that satisfies all constraints and maximizes the minimum utility of the three modes. The constrained Pareto optimal solution is then output as the multimodal computation orchestration scheme.
[0222] The fusion computing module 506 is used to perform resource orchestration and cross-modal computing according to the multimodal computing orchestration scheme. When the mode switching conditions are met, it calls the intermediate state conversion protocol to complete the intermediate state data conversion and obtain the privacy computing result. The fusion computing module 506 takes the multimodal computing orchestration scheme and the intermediate state conversion protocol as input, allocates computing tasks on the local computing nodes and edge computing nodes of each participant according to the orchestration scheme, and starts the computing pipeline. During the computing process, it continuously monitors the execution status of each task. When mode switching conditions such as accuracy stagnation, communication bottlenecks, or sensitivity exceeding limits are met, it pauses the current mode and saves a computing snapshot. It then calls the intermediate state conversion protocol to seamlessly migrate the intermediate state data of the current mode to the data structure required by the next mode, and then resumes computing in the new mode. Driven by mode switching conditions and ensuring semantic consistency of migration through the intermediate state conversion protocol, the fusion computing module 506 switches between the three modes as needed, completing each sub-task in the orchestration scheme one by one until the computing task of the last privacy computing mode is completed, and uses the final output data as the privacy computing result.
[0223] Example 13
[0224] This embodiment uses a specific application scenario of medical insurance big data fraud detection to provide a supporting explanation of the overall effect of the multimodal privacy computing fusion method under large-scale, highly sensitive, and highly heterogeneous data, and gives a comparison with the single privacy computing mode.
[0225] In one embodiment, this application scenario involves ten provincial medical insurance centers. A joint fraud detection model is trained and analyzed on approximately three billion pieces of medical insurance settlement data from the past three years. The data from each provincial center includes highly sensitive information such as insured individuals' medical records, drug prescriptions, and expense details. A cloud orchestration server performs multi-dimensional feature quantification based on the data asset metadata reported by each provincial center. The sensitivity of the expense detail attribute column is categorized as extremely high, while other attribute columns are categorized as medium or low, resulting in a data feature vector. A multi-source robust optimization model, combined with historical recommendation correctness feedback, dynamically adjusts the Wasserstein trust radius of each feature dimension to perform robust pattern recommendation for the tasks of the ten provincial centers, obtaining a set of candidate pattern combinations. The abstract mapping module maps the privacy-preserving computation modes used in the candidate pattern combinations to a unified computational unit abstract model set. Based on this, the protocol generation module constructs a cross-pattern intermediate data structure and state transition function. The scheme optimization module inputs the candidate mode combination set and the unified computing unit abstract model set into the submodal multi-objective optimization model. It solves for the constrained Pareto optimal solution through an adaptive balance penalty mechanism, obtaining a multimodal computing orchestration scheme: six provincial centers adopt a federated learning mode, two provincial centers adopt a secure multi-party computation mode, and two provincial centers adopt a trusted execution environment mode. Cost detail data is always processed within the hardware isolation boundary of the trusted execution environment. The fusion computing module performs resource orchestration and cross-modal computation according to the orchestration scheme. When switching conditions are met, it calls the intermediate state conversion protocol to complete the intermediate state data conversion and outputs privacy-preserving computation results.
[0226] In another embodiment, if all ten provincial centers adopt a single federated learning mode, the fraud detection AUC (area under the receiver operating characteristic curve) is only 0.88 due to the highly heterogeneous data distribution among the provincial centers and the risk of leakage of highly sensitive attributes such as cost details during gradient exchange, and it fails to meet the ethical compliance requirements of cost details. If all adopt a single secure multi-party computation mode, although security can be met, full-scale secure computation significantly increases the total training time, and the fraud detection AUC of the single secure multi-party computation mode is only 0.79. Using the multimodal computation orchestration scheme of this embodiment, the fraud detection AUC reaches 0.94, and the total training time is reduced by 41% compared with the single secure multi-party computation mode. The results show that, through the multimodal fusion of this invention, under the conditions of highly heterogeneous and sensitive data, the three modes of federated learning, secure multi-party computation, and trusted execution environment are reasonably allocated to their respective most suitable sub-tasks based on data characteristics. Cost details are processed within the hard boundary of the trusted execution environment, heterogeneous data is trained in federated learning and hybrid modes, and statistical query sub-tasks are executed in secure multi-party computation mode. Thus, computational efficiency and model accuracy are balanced under the hard constraints of security and compliance.
[0227] This invention achieves intelligent orchestration and organic integration of three privacy computing modes—federated learning, secure multi-party computation, and trusted execution environment—by quantifying multi-dimensional features of data asset metadata of each participant, robust pattern recommendation based on multi-source multi-component robust optimization, unified abstraction of privacy computing modes and generation of cross-mode intermediate state conversion protocols, and orchestration decision-making based on sub-modal multi-objective optimization and adaptive balance penalty mechanism. Under the hard constraints of security and compliance, it balances computational efficiency and model accuracy, overcoming the shortcomings of existing technologies such as lack of optimality in mode selection, complete isolation between modes and inability to coordinate, and lack of multi-objective optimization mechanism under security and compliance constraints.
[0228] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and such modifications or substitutions should all be covered within the scope of protection of the present invention.
Claims
1. A multi-modal privacy computing fusion method, characterized in that, include: Obtain the data asset metadata of each participant, and perform multi-dimensional feature quantization on the data asset metadata to obtain the data feature vector; The data feature vector is decomposed into each feature dimension and input into a multi-source robust optimization model. The reliability parameters of each feature dimension are dynamically adjusted based on historical feedback to construct a set of distribution uncertainties. Robust pattern recommendation is performed on the set of distribution uncertainties, and a set of candidate pattern combinations is output. According to the preset abstract mapping rules, each privacy computing mode adopted in the candidate mode combination set is mapped to obtain the corresponding unified computing unit abstract model, and the unified computing unit abstract model set is formed by summarizing them. Based on the unified computing unit abstract model set, a cross-mode intermediate state data structure is constructed and a state transition function is defined. An intermediate state transition protocol is generated by the cross-mode intermediate state data structure and the state transition function. The candidate mode combination set and the unified computing unit abstract model set are input into the submodal multi-objective optimization model. The constrained Pareto optimal solution is solved through an adaptive balance penalty mechanism. The constrained Pareto optimal solution is determined as the multimodal computing orchestration scheme. Resource orchestration and cross-modal computation are performed according to the multimodal computation orchestration scheme. When the mode switching conditions are met, the intermediate state conversion protocol is invoked to complete the intermediate state data conversion and obtain the privacy computation result.
2. The method of claim 1, wherein, The process of acquiring data asset metadata from each participant and performing multi-dimensional feature quantization on the data asset metadata to obtain a data feature vector includes: Receive data asset configuration files uploaded by each participant through a preset interface, and parse them to obtain the data asset metadata. The sensitivity index, computational complexity index, Non-IID distribution index, and communication capacity index are extracted and calculated from the data asset metadata. The sensitivity index, computational complexity index, Non-IID distribution index, and communication capacity index are each normalized and mapped into feature vectors, and the feature vectors are concatenated to form the data feature vector.
3. The method according to claim 1, characterized in that, The process involves decomposing the data feature vector into various feature dimensions and inputting them into a multi-source robust optimization model. Based on historical feedback, the reliability parameters of each feature dimension are dynamically adjusted to construct a set of distributional uncertainties. Robust pattern recommendation is then performed on this set of distributional uncertainties, outputting a set of candidate pattern combinations, including: The data feature vector is decomposed into feature dimensions according to the quantified attribute categories. Each feature dimension is input into the multi-source sub-Brussels optimization model. The accuracy is calculated by combining the obtained historical recommendation correctness feedback. Based on the accuracy, the Wasserstein trust radius of the corresponding feature dimension is dynamically adjusted. The adjusted Wasserstein trust radius is used as the reliability parameter. The feature dimensions include sensitivity index, computational complexity index, Non-IID distribution index, and communication capacity index. Based on the reliability parameters, a fluctuation range is set for the empirical probability distribution of each feature dimension, thereby generating the distribution uncertainty set; Find the worst-case probability distribution in the set of uncertain distributions, perform robust pattern recommendation on the worst-case probability distribution, and output the set of candidate pattern combinations.
4. The method according to claim 1, characterized in that, The process involves mapping each privacy computing mode used in the candidate mode combination set according to a preset abstract mapping rule to obtain a corresponding unified computing unit abstract model, which is then aggregated to form a unified computing unit abstract model set, including: Traverse the candidate pattern combination set and extract the privacy computing mode adopted by each candidate pattern combination in the candidate pattern combination set; According to the preset abstract mapping rules, determine the operator type and interface definition corresponding to each privacy computing mode; Based on the operator type and the interface definition, a unified computing unit abstract model corresponding to each privacy computing mode is constructed, and the unified computing unit abstract models are summarized to form the unified computing unit abstract model set.
5. The method according to claim 4, characterized in that, The step of determining the operator type and interface definition corresponding to each privacy computing mode according to the preset abstract mapping rules includes: According to the preset abstract mapping rules, the underlying protocol and execution logic of each privacy computing mode are parsed. Based on the underlying protocol and the execution logic, the corresponding operator type is determined by matching with a preset operator library. Extract the data input and output specifications of each privacy computing mode and standardize them into a unified interface definition.
6. The method according to claim 1, characterized in that, The process of constructing a cross-modal intermediate state data structure and defining a state transition function based on the unified computing unit abstract model set, and generating an intermediate state transition protocol from the cross-modal intermediate state data structure and the state transition function, includes: Traverse the set of unified computing unit abstract models and extract the operator attributes and data flow relationships of each unified computing unit abstract model; Based on the operator attributes and the data flow relationship, a standardized cross-modal intermediate data structure is constructed; By utilizing the execution dependencies between the abstract models of each unified computing unit, the state triggering conditions and state transition logic are determined, the state triggering conditions are mapped to the state transition logic, and the state transition function is defined. The cross-mode intermediate state data structure and the state transition function are combined and encapsulated to generate the intermediate state transition protocol.
7. The method according to claim 1, characterized in that, The step of inputting the candidate mode combination set and the unified computing unit abstract model set into the submodal multi-objective optimization model, solving for the constrained Pareto optimal solution through an adaptive balance penalty mechanism, and determining the constrained Pareto optimal solution as the multimodal computing orchestration scheme includes: The candidate mode combination set and the unified computing unit abstract model set are input into the submodal multi-objective optimization model to construct a multi-objective optimization function and corresponding constraints, including differential privacy budget constraints, security level constraints and resource upper limit constraints. Set and initialize adaptive penalty weights, calculate the degree of violation of the differential privacy budget constraint, the degree of violation of the security level constraint, and the degree of violation of the resource upper limit constraint in the current optimization iteration process, obtain the comprehensive violation degree, and use the comprehensive violation degree to adaptively update the adaptive penalty weights to construct the adaptive balance penalty mechanism; The adaptive balance penalty mechanism is introduced into the submodal optimization algorithm. The multi-objective optimization function with added penalty term is iteratively solved to obtain the constrained Pareto optimal solution that satisfies the constraint conditions. The constrained Pareto optimal solution is determined as the multimodal computation orchestration scheme.
8. The method according to claim 7, characterized in that, The iterative solution of the multi-objective optimization function with added penalty terms to obtain the constrained Pareto optimal solution that satisfies the constraints includes: Set an orchestration set and initialize it to an empty set. For each candidate pattern combination in the candidate pattern combination set, calculate the minimum marginal gain that selects the candidate pattern combination into the orchestration set. Under the differential privacy budget constraint, an exponential mechanism is introduced to calculate the selection probability of each candidate mode combination based on the minimum marginal gain; By iteratively executing a probabilistic greedy selection, the candidate pattern combinations are gradually added to the orchestration set according to the selection probability; When the orchestration set satisfies the constraints and the minimum marginal gain of the candidate pattern combination is lower than a preset threshold, the orchestration set is output as the constrained Pareto optimal solution.
9. The method according to claim 1, characterized in that, Before inputting the candidate pattern combination set and the unified computing unit abstract model set into the submodal multi-objective optimization model, the method further includes: Semantic annotations of each attribute column in the data asset metadata are obtained, the correlation degree between each attribute column is calculated based on conditional mutual information, the inferability between each attribute column is evaluated based on the correlation degree to conduct inference risk assessment, and attribute association risk parameters are obtained. By combining the attribute-related risk parameters and the preset initial privacy budget, a privacy budget allocation model is established with the privacy budget allocation ratio of each attribute column as the optimization variable and the data utility maximization as the objective function. The privacy budget allocation model is solved by Lagrange duality to obtain the optimal solution of the optimization variables, and the optimal solution is used as a differentiated adaptive privacy budget allocation scheme. The adaptive privacy budget allocation scheme is used as a constraint and introduced into the submodal multi-objective optimization model for solution to obtain the multimodal computation orchestration scheme.
10. The method according to claim 1, characterized in that, The process involves performing resource orchestration and cross-modal computation according to the multimodal computation orchestration scheme. When the mode switching conditions are met, the intermediate state conversion protocol is invoked to complete the intermediate state data conversion and obtain the privacy computation result, including: The first privacy computing mode in the multimodal computing orchestration scheme is determined as the current privacy computing mode. Resource orchestration is performed on the current privacy computing mode and the computing task of the model to be processed is started. The model to be processed is a machine learning model built based on the data asset metadata. Monitor the execution status of the computing task and obtain the accuracy of the model to be processed. When the accuracy improvement of the model to be processed is lower than the minimum threshold in a series of preset iterations, or when the communication latency of the computing task exceeds the preset latency, determine that the mode switching condition is met and trigger mode switching. Obtain the intermediate state data output by the current privacy computing mode, call the intermediate state conversion protocol to parse and map the intermediate state data, convert it into the data structure required by the next privacy computing mode, and complete the intermediate data conversion; Based on the data structure, the computation task of the next privacy computation mode is initiated to complete the cross-mode computation; Repeat the steps of monitoring the execution status of the computing task until the cross-modal computing is completed, until the computing task of the last privacy computing mode in the multimodal computing orchestration scheme is completed, and output the data of the last privacy computing mode as the privacy computing result.
11. The method according to claim 10, characterized in that, After the intermediate state data conversion is completed by calling the intermediate state conversion protocol, the process also includes: Obtain the parameters of the model to be processed corresponding to the intermediate state data before and after switching between the current privacy computing mode and the next privacy computing mode, and use them as parameters to be verified; The consistency of the parameter to be verified is verified by using Chebyshev's inequality to probabilistically constrain the deviation of the parameter to be verified to be within the tolerance range.
12. A multimodal privacy computing fusion system, characterized in that, include: The feature quantization module is used to acquire data asset metadata of each participant, perform multi-dimensional feature quantization on the data asset metadata, and obtain data feature vectors. The pattern recommendation module is used to decompose the data feature vector into each feature dimension and input it into the multi-source robust optimization model. Based on historical feedback, it dynamically adjusts the reliability parameters of each feature dimension to construct a set of distribution uncertainty, performs robust pattern recommendation on the set of distribution uncertainty, and outputs a set of candidate pattern combinations. The abstract mapping module is used to map each privacy computing mode adopted in the candidate mode combination set according to the preset abstract mapping rules, so as to obtain the corresponding unified computing unit abstract model and summarize them to form a unified computing unit abstract model set. The protocol generation module is used to construct a cross-mode intermediate state data structure and define a state transition function based on the unified computing unit abstract model set, and to generate an intermediate state transition protocol from the cross-mode intermediate state data structure and the state transition function; The scheme optimization module is used to input the candidate mode combination set and the unified computing unit abstract model set into the submodal multi-objective optimization model, solve the constrained Pareto optimal solution through an adaptive balance penalty mechanism, and determine the constrained Pareto optimal solution as the multimodal computing orchestration scheme. The fusion computing module is used to perform resource orchestration and cross-modal computing according to the multimodal computing orchestration scheme. When the mode switching conditions are met, the intermediate state conversion protocol is invoked to complete the intermediate state data conversion and obtain the privacy computing result.
