A privacy computing driven data flow method and system

Abstract

The present application relates to the technical field of data processing, in particular to a privacy calculation driven data flow method and system. The method comprises: obtaining the fields of a first participant and a second participant, constructing a candidate mapping relationship graph according to the semantic vector and the similarity of the distribution characteristics of the fields of the two participants, selecting a target field of the first participant and a candidate field similar to the target field in the second participant, and generating a field combination; determining the reconstruction deviation according to the mutual information amount of the field combination and the target field and the information entropy of the target field, and judging whether there is an equivalent relationship of the composite split type between the field combination and the target field; judging whether there is an equivalent relationship of the coding heterogeneous type between the target field of the first participant and the candidate field of the second participant based on the candidate transformation function set; and generating corresponding instructions according to the type of the equivalent relationship. The present application improves the success rate of starting and the calculation accuracy of the privacy calculation task in the heterogeneous data source scene.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and more specifically to a privacy-computing-driven data flow method and system. Background Technology

[0002] Currently, privacy-preserving computation technology is widely used as a core driving force in the data circulation field to address the issues of secure sharing and value extraction of data elements under the principle of "usable but not visible." Existing solutions, such as data circulation systems based on homomorphic encryption, secure multi-party computation, or federated learning, typically include the following standard processes: data preprocessing, privacy-preserving computation task distribution, multi-party secure computation execution, and computation result output. In scenarios involving joint modeling or joint querying of multiple data sources, to ensure the accuracy and security of the computation logic, the system usually performs consistency verification on the data features or data tags provided by each participant before executing the privacy-preserving computation task. Existing technologies generally use pre-set metadata mapping tables or field matching algorithms based on character edit distance to directly establish the correspondence between data fields of the participants, and then proceed to the privacy-preserving computation task distribution stage.

[0003] However, in real-world data circulation environments, data among different participants often exhibits significant heterogeneity. On one hand, the representation of the same business entity may differ across systems. For example, one party might store user identity information as a single user ID field, while another might split it into multiple sub-fields such as "document type" and "document number." On the other hand, different systems may employ different encoding logics. For instance, one party might use a plaintext mobile phone number, while another, for security reasons, might use the hash value of that number. In these scenarios, existing methods relying on explicit field name matching or simple character similarity calculations have significant limitations. Specifically, when single-field and multi-field combinations are logically equivalent or when encoding methods differ between participants, existing methods cannot identify this explicit field name mismatch but implicit business logic equivalence. This makes it impossible to accurately determine during task initialization whether the field mismatch is caused by differences in naming rules or by the composite splitting of data structures or heterogeneous encoding logic. Therefore, existing technologies struggle to accurately output a directional correction instruction with field mapping reconstruction capabilities during critical data alignment stages, which in turn affects the success rate and computational accuracy of privacy computing tasks in heterogeneous data source scenarios. Summary of the Invention

[0004] To address the issues of low success rate and computational accuracy of privacy-preserving computation tasks in existing heterogeneous data source scenarios, the present invention aims to provide a privacy-preserving computation-driven data circulation method and system, the specific technical solution of which is as follows:

[0005] In a first aspect, the present invention provides a privacy-computing-driven data circulation method, the method comprising the following steps:

[0006] Obtain the fields of the first and second participants, and extract the semantic vectors and distribution features of the fields;

[0007] A comprehensive similarity is obtained based on the similarity of the semantic vectors and distribution features of the fields of the first participant and the second participant; initial field pairs are selected based on the comprehensive similarity, and candidate mapping relationship graphs are constructed using the initial field pairs;

[0008] Based on the candidate mapping relationship graph, the target field of the first participant and the candidate fields similar to the target field in the second participant are selected to generate a field combination; the reconstruction deviation is determined according to the mutual information between the field combination and the target field and the information entropy of the target field; and the reconstruction deviation is used to determine whether there is an equivalence relationship of composite split type between the field combination and the target field.

[0009] Based on the candidate mapping relationship graph, the target field of the first participant and the candidate field of the second participant are selected to construct a candidate transformation function set; based on the candidate transformation function set, it is determined whether there is an equivalence relationship of encoding heterogeneous types between the target field and the candidate field.

[0010] Based on the type of equivalence relation, generate the corresponding instruction and execute it.

[0011] Preferably, the step of obtaining a comprehensive similarity based on the similarity of the semantic vectors and distribution features of the fields of the first participant and the second participant includes:

[0012] Calculate the cosine similarity between the semantic vectors of the fields of the first participant and the fields of the second participant, and use it as the semantic similarity.

[0013] The statistical feature values ​​of the fields of the first participant and the fields of the second participant are extracted and normalized respectively. The distribution similarity is obtained based on the difference between the normalized statistical feature values ​​of the fields of the first participant and the normalized statistical feature values ​​of the fields of the second participant. The statistical feature values ​​are used to characterize the distribution features.

[0014] The semantic similarity and the distributional similarity are weighted and summed to obtain the comprehensive similarity.

[0015] Preferably, the initial field pair filtering based on comprehensive similarity includes:

[0016] Sort all the comprehensive similarities in ascending order to obtain the comprehensive similarity sequence;

[0017] Calculate the first difference between adjacent data in the comprehensive similarity sequence, and use the largest first difference as the dividing point to divide the comprehensive similarity sequence into two parts; use the field pairs corresponding to the right part as the initial field pairs.

[0018] Preferably, determining the reconstruction deviation based on the mutual information between the field combination and the target field and the information entropy of the target field includes: calculating a first ratio of the mutual information to the information entropy; and using the difference between the constant 1 and the first ratio as the reconstruction deviation.

[0019] Preferably, the step of determining whether there is an equivalence relationship of composite split type between the field combination and the target field based on the reconstruction deviation includes:

[0020] The reconstruction deviation of all field combinations corresponding to the same target field is sorted in ascending order to obtain the reconstruction deviation sequence;

[0021] Calculate the second difference between adjacent reconstruction deviations in the reconstruction deviation sequence, and use the largest second difference as the dividing point to extract the low deviation cluster in the reconstruction deviation sequence;

[0022] Based on the number of combinations within the low deviation cluster, the value of the reconstruction deviation, and the second difference, determine whether there is an equivalence relationship of composite split type between the optimal equivalent combination and the target field.

[0023] Preferably, the step of determining whether there is an equivalence relationship of composite split type between the optimal equivalent combination and the target field based on the number of combinations within the low deviation cluster, the value of the reconstruction deviation, and the second difference includes:

[0024] The combination with the smallest reconstructed deviation in the low deviation cluster is taken as the optimal equivalent combination;

[0025] If the preset conditions are met, it is determined that there is no equivalence relationship of composite split type between the optimal equivalent combination and the target field;

[0026] If the preset conditions are not met, it is determined that there is an equivalence relationship of composite split type between the optimal equivalent combination and the target field;

[0027] The preset conditions are: the largest second difference is less than a preset difference threshold, or the number of combinations contained in the low deviation cluster exceeds a preset proportion of the total number of combinations, or the minimum reconstruction deviation in the low deviation cluster is greater than a preset reconstruction deviation threshold.

[0028] Preferably, determining whether there is an equivalence relationship of encoding heterogeneous types between the target field and the candidate fields based on the candidate transformation function set includes:

[0029] Select a preset number of samples, use the candidate transformation functions in the candidate transformation function set to map the value of the target field to the value of the candidate field, and perform inverse transformation restoration. The proportion of samples that are not restored correctly is used as the one-way loss value.

[0030] The same transformation function is repeatedly applied to the value of the target field for a preset number of mapping and inverse transformation restoration processes, and the proportion of the number of times the results are consistent is used as the deterministic confidence level.

[0031] When the forward mapping verification is successful, the one-way loss value is less than or equal to the first preset threshold, and the deterministic confidence is greater than or equal to the second preset threshold, it is determined that there is an equivalence relationship between the target field and the candidate field in terms of encoding heterogeneous types.

[0032] Preferably, generating corresponding instructions based on the type of equivalence relation includes:

[0033] For equivalence relations of composite split types, generate instructions that include scenario type, source participant identifier, source field list, target participant identifier, target field name, and execution action;

[0034] For equivalence relations of heterogeneous encoding types, generate instructions that include scene type, source participant identifier, source field name, target participant identifier, target field name, execution action, and transformation rules.

[0035] Preferably, the instructions include:

[0036] When executing a compound splitting instruction, during the data loading phase, when performing field value concatenation on each record, the delimiters in the preset delimiter dictionary are traversed to generate multiple sets of concatenation results for verification, and the concatenated values ​​are mapped to the specified target fields.

[0037] When executing instructions that encode heterogeneous types, the inverse transformation operation specified in the transformation rules is performed on each record during the data loading phase, mapping the transformed value to the specified target field.

[0038] In a second aspect, the present invention provides a privacy-computing-driven data circulation system for implementing the method of the first aspect described above, the system comprising:

[0039] The acquisition module is used to acquire the fields of the first and second participants, and extract the semantic vectors and distribution features of the fields;

[0040] The relationship building module is used to obtain a comprehensive similarity based on the similarity of the semantic vectors and distribution features of the fields of the first participant and the second participant; it then filters initial field pairs based on the comprehensive similarity and uses these initial field pairs to build a candidate mapping relationship graph.

[0041] The first judgment module is used to select the target field of the first participant and the candidate fields similar to the target field in the second participant based on the candidate mapping relationship graph, and generate a field combination; determine the reconstruction deviation degree based on the mutual information between the field combination and the target field and the information entropy of the target field; and determine whether there is an equivalence relationship of composite split type between the field combination and the target field based on the reconstruction deviation degree.

[0042] The second judgment module is used to select the target field of the first participant and the candidate field of the second participant based on the candidate mapping relationship graph, and construct a candidate transformation function set; based on the candidate transformation function set, it is determined whether there is an equivalence relationship of encoding heterogeneous types between the target field and the candidate field;

[0043] The execution module is used to generate and execute corresponding instructions based on the type of equivalence relation.

[0044] The present invention has at least the following beneficial effects:

[0045] This invention obtains fields from a first participant and a second participant, and extracts the semantic vectors and distribution features of these fields. It then obtains a comprehensive similarity score based on the similarity of the semantic vectors and distribution features, filters initial field pairs based on this score, and constructs a candidate mapping graph. Based on the candidate mapping graph, it selects target fields and candidate fields to generate field combinations. The reconstruction deviation is determined based on the mutual information between the field combinations and the target field, and the information entropy of the target field. The reconstruction deviation is used to determine whether there is an equivalence relationship for composite splitting types. Based on the candidate mapping graph, it selects target fields and candidate fields to construct a candidate transformation function set, and determines whether there is an equivalence relationship for heterogeneous encoding types. Finally, it generates and executes corresponding instructions based on the type of equivalence relationship. This method can automatically identify the equivalence relationships between participants in business logic (single and multiple field combinations) and when there are heterogeneous encoding transformations, without relying on explicit field name matching or simple character similarity calculations. This accurately determines the cause of field mismatches and outputs directional correction instructions with field mapping reconstruction capabilities, improving the success rate and accuracy of privacy computing tasks in heterogeneous data source scenarios. Attached Figure Description

[0046] To more clearly illustrate the technical solutions and advantages in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0047] Figure 1A flowchart illustrating a privacy-driven data flow method provided in an embodiment of the present invention;

[0048] Figure 2 This is a structural block diagram of a privacy-driven data circulation system provided in an embodiment of the present invention. Detailed Implementation

[0049] To further illustrate the technical means and effects adopted by the present invention to achieve the intended purpose, the following detailed description of a privacy-computing-driven data flow method and system based on the present invention, in conjunction with the accompanying drawings and preferred embodiments, is provided below.

[0050] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0051] The following description, in conjunction with the accompanying drawings, details a specific scheme for a privacy-computing-driven data circulation method and system provided by the present invention.

[0052] An example of a privacy-computing-driven data flow method:

[0053] This embodiment proposes a privacy-preserving computation-driven data flow method, such as... Figure 1 As shown, a privacy-computing-driven data flow method according to this embodiment includes the following steps:

[0054] Step S1: Obtain the fields of the first participant and the second participant, and extract the semantic vectors and distribution features of the fields.

[0055] First, a domain ontology library is constructed, covering core business entity categories in data circulation scenarios, including user identifiers (mobile phone number, ID card number, user ID), device fingerprints (IMEI, MAC address), geographic location (latitude and longitude, city code), transaction flow (transaction amount, order number), etc., with standard semantic concepts and a set of synonyms defined under each category.

[0056] Then, each field name of the first and second participants is segmented and replaced with synonyms, thus completing the preprocessing of the fields. The preprocessed field names are then matched with standard semantic concepts in the ontology library, and the standard concepts are converted into semantic vectors using the Word2Vec model. This yields the semantic vectors of each field of the first participant and each field of the second participant.

[0057] Simultaneously, statistical feature values ​​are extracted for each field of the first participant and each field of the second participant. These statistical feature values ​​are used to characterize the distribution characteristics of the fields. In this embodiment, the statistical feature values ​​include the mean, standard deviation, and sample entropy. In specific applications, implementers can set the types of statistical feature values ​​according to specific circumstances, which will not be elaborated further here.

[0058] Thus, the semantic vectors and statistical feature values ​​of the fields of the first and second participants have been obtained through the above methods.

[0059] Step S2: Obtain a comprehensive similarity based on the similarity of the semantic vectors and distribution features of the fields of the first participant and the second participant; filter initial field pairs based on the comprehensive similarity, and construct a candidate mapping relationship graph using the initial field pairs.

[0060] After obtaining the semantic vectors and distribution features of the fields of the first participant and the second participant, it is necessary to evaluate the similarity between the two separately. Based on the evaluation results, the comprehensive similarity between the fields of the first participant and the second participant is determined, and then the corresponding candidate mapping relationship graph is constructed.

[0061] This embodiment uses one field from the first participant and one field from the second participant as examples for illustration. Other fields can be processed using the method provided in this embodiment.

[0062] Specifically, firstly, the cosine similarity between the semantic vectors of the fields of the first participant and the fields of the second participant is calculated as semantic similarity. The greater the semantic similarity, the higher the degree of similarity between the two fields in the semantic space.

[0063] Considering the consistency of the value distribution patterns of business-equivalent fields, although the same business entity may have different names in different systems, the data values ​​it carries originate from the same set of real-world objects. Therefore, the statistical distribution characteristics should exhibit quantifiable similarity. For example, the first participant has a field "user's mobile phone number," and the second participant has a field "mobile phone identifier." The semantic similarity of the two fields is high. In terms of value range distribution, both fields store 11-digit mobile phone numbers, and the values ​​are randomly distributed, meaning the distribution similarity should also be high.

[0064] Based on the above characteristics, the statistical feature values ​​of the fields of the first participant and the second participant are normalized respectively. The distribution similarity is obtained based on the difference between the normalized statistical feature values ​​of the fields of the first participant and the second participant. This embodiment uses the maximum-minimum value normalization method to normalize the statistical feature values. The maximum-minimum value normalization method is existing technology and will not be elaborated further here. As other implementation methods, other existing data normalization methods can also be used to normalize the data.

[0065] As a concrete example, the specific formula for calculating distribution similarity is given. Distribution similarity can be expressed as:

[0066]

[0067] in, Indicates distribution similarity, This represents the number of statistical characteristic values. This represents the normalized k-th statistical feature value of the field representing the first participant. This represents the normalized k-th statistical feature value of the field belonging to the second participant. Indicates the absolute value sign. This indicates the preset zero-prevention parameter. This represents the normalization function.

[0068] In this embodiment, the data normalization process uses the maximum-minimum value normalization method, which is an existing technology and will not be described in detail here. As other implementation methods, other existing data normalization methods can also be used for processing.

[0069] The preset zero-prevention parameter is introduced in the above formula for calculating distribution similarity in order to prevent the denominator of the formula from being 0. In this embodiment, the value of the preset zero-prevention parameter is 0.001. In specific applications, implementers can set it according to specific circumstances.

[0070] The difference between the normalized k-th statistical feature value of the field representing the first participant and the normalized k-th statistical feature value of the field representing the second participant. The average difference between the normalized statistical feature value of the field representing the first participant and the normalized statistical feature value of the field representing the second participant. The smaller the value, the more consistent the distribution characteristics between the two fields; the larger the value, the greater the distribution difference between the two fields, and they should be excluded even if they are semantically similar.

[0071] Considering that relying solely on semantic or distribution features may lead to misjudgments, fields with similar semantics but vastly different distributions (such as "User ID" and "Order Number" being semantically similar identifiers but with significantly different distribution features), as well as fields with similar distributions but unrelated semantics (such as "Amount" and "Temperature" being continuous value distributions but with different business semantics), need to undergo double verification through weighted fusion to improve the accuracy of similarity judgment.

[0072] Therefore, the comprehensive similarity is obtained by weighting and summing the semantic similarity and distribution similarity obtained above.

[0073] As a concrete example, the specific formula for calculating the overall similarity is given. The overall similarity can be expressed as:

[0074]

[0075] in, Indicates the overall similarity. Indicates semantic similarity. Indicates distribution similarity, Weights representing semantic similarity The weights represent the similarity of the distributions.

[0076] The sum of the weights of semantic similarity and distribution similarity is 1. In this embodiment, since semantic similarity and distribution similarity are equally important, the weight of semantic similarity is set to 0.5, and the weight of distribution similarity is also 0.5.

[0077] Using the above method, we can obtain the comprehensive similarity score between each field of the first participant and each field of the second participant. The higher the comprehensive similarity score, the higher the probability of business equivalence between the two fields. Sort all comprehensive similarity scores in ascending order to obtain a comprehensive similarity score sequence.

[0078] Calculate the absolute value of the difference between adjacent data in the comprehensive similarity sequence, and denote this absolute value as the first difference. Use the largest first difference as the split point to divide the comprehensive similarity sequence into two parts. The field pairs corresponding to the data in the right part are used as the initial field pairs. Each field pair consists of one field from the first participant and one field from the second participant. It should be noted that if multiple largest first differences exist, the first first difference in the comprehensive similarity sequence is used as the split point.

[0079] Furthermore, a candidate mapping relationship graph is constructed, which contains all initial field pairs, that is, all highly similar field combinations, to provide data support for subsequent analysis.

[0080] Step S3: Based on the candidate mapping relationship graph, select the target field of the first participant and the candidate fields of the second participant that are similar to the target field to generate a field combination; determine the reconstruction deviation degree based on the mutual information between the field combination and the target field and the information entropy of the target field; and determine whether there is an equivalence relationship of composite split type between the field combination and the target field based on the reconstruction deviation degree.

[0081] Considering that some single fields may be split into multiple subfields, the business equivalence of the two can be verified by calculating the information loss between the joint distribution of the multi-field combination and the single distribution of the target field. If multiple subfields of the second participant are the result of splitting a certain field of the first participant, then the joint distribution of these subfields should contain all the information of the target field, and the information entropy between the two should be an inclusion relationship rather than an intersection relationship. In other words, after knowing the combination of subfields, the uncertainty of the target field should be minimized, that is, the conditional mutual information should be close to the original entropy value of the target field.

[0082] Next, this embodiment will use a field of the first participant in the candidate mapping relationship diagram obtained by the above steps as an example for illustration. Other fields of the first participant in the candidate mapping relationship diagram can be processed using the method provided in this embodiment.

[0083] Specifically, any field of the first participant in the candidate mapping graph obtained in the above steps is denoted as the target field, which can be denoted as C; all fields that form an initial field pair with the target field C are candidate fields similar to the target field C. The candidate fields of the target field C can generate all binary and ternary combinations. Selecting binary combinations as the field combinations for subsequent analysis can cover more than 90% of the splitting scenarios.

[0084] Furthermore, the information entropy of the target field C of the first participant is calculated. This information entropy reflects the total amount of information contained in the target field; the higher the entropy value, the more random and uncertain the field value. Assuming candidate fields similar to the target field C are A and B, the mutual information between the field combination AB and the target field C is calculated. This mutual information is denoted as [formula missing]. ;in, This represents the conditional entropy of C given AB, i.e., the remaining uncertainty; Let the information entropy of the target field C of the first participant be represented. Then, the ratio of mutual information to information entropy is calculated and recorded as the first ratio; the difference between the constant 1 and the first ratio is taken as the reconstruction deviation.

[0085] As a concrete example, the specific formula for calculating the reconstruction deviation is given. The reconstruction deviation can be expressed as:

[0086]

[0087] in, Indicates the degree of reconstruction deviation. This represents the mutual information between field combination AB and target field C. This represents the first ratio.

[0088] The value of the reconstruction deviation ranges from 0 to 1. The closer the value is to 0, the more information the field combination AB can restore from the target field C, and the higher the equivalence. The closer the value of the reconstruction deviation is to 1, the more information the combined field AB can explain from the target field, and other information needs to be obtained from other sources.

[0089] It should be noted that when calculating the reconstruction deviation, if the denominator of the calculation formula is 0, a preset zero-prevention parameter is added to it as a new denominator before recalculation.

[0090] Using the above method, the reconstruction deviation of all field combinations of target field C can be obtained. These reconstruction deviations are then sorted in ascending order to obtain a reconstruction deviation sequence. The absolute value of the difference between adjacent reconstruction deviations in the sequence is calculated and designated as the second difference. Using the largest second difference as the dividing point, the reconstruction deviation sequence is divided into two parts: the left part is designated as the low deviation cluster, and the right part as the high deviation cluster. It should be noted that if multiple largest second differences exist, the first second difference in the reconstruction deviation sequence is used as the dividing point.

[0091] Field combinations within low-deviation clusters are considered potentially equivalent candidate combinations. The combination with the smallest reconstruction deviation within the low-deviation cluster is selected as the optimal equivalent combination, and then an equivalence relationship of composite splitting type is determined between the optimal equivalent combination and the target field. Specifically, preset conditions are first set: the largest second difference is less than a preset difference threshold, or the number of combinations in the low-deviation cluster exceeds a preset proportion of the total number of combinations, or the smallest reconstruction deviation in the low-deviation cluster is greater than a preset reconstruction deviation threshold. When the largest second difference is less than the preset difference threshold, it indicates that the data distribution is uniform and there are no significant jumps. When the number of combinations in the low-deviation cluster exceeds a preset proportion of the total number of combinations, it indicates that the low-deviation cluster is too large and lacks sufficient discriminative power. When the smallest reconstruction deviation in the low-deviation cluster is greater than the preset reconstruction deviation threshold, it indicates that the optimal combination also suffers from significant information loss. If the preset conditions are met, it is determined that there is no equivalence relationship of composite splitting type between the optimal equivalent combination and the target field, and the corresponding field is classified as having low confidence and requires manual review. If the preset conditions are not met, it is determined that there is an equivalence relationship of composite splitting type between the optimal equivalent combination and the target field. In this embodiment, the preset ratio is 30%, the preset reconstruction deviation threshold is 0.15, and the preset difference threshold is 0.08. In specific applications, implementers can set these values ​​according to specific circumstances.

[0092] Step S4: Select the target field of the first participant and the candidate field of the second participant based on the candidate mapping relationship graph, and construct a candidate transformation function set; based on the candidate transformation function set, determine whether there is an equivalence relationship between the target field and the candidate field for heterogeneous encoding types.

[0093] Next, we will determine whether there is an equivalence relationship between the target field of the first participant and the candidate field of the second participant regarding the type of coded heterogeneity. Equivalent coded transformations should satisfy two conditions: first, there exists a deterministic transformation function that maps one code to another; second, the transformation is mathematically invertible, meaning there exists an inverse transformation that can recover the plaintext from the ciphertext. By verifying the one-way loss and deterministic confidence of the transformation function, we can quantitatively determine whether pairs of coded heterogeneous fields are equivalent.

[0094] Next, we will take the target field C of the first participant in the candidate mapping relationship diagram as an example for explanation. For any candidate field of the target field C in the fields of the second participant, we will denote the candidate field as R.

[0095] Based on the format characteristics, enumerate the possible transformation rules to form a candidate transformation function set F; the candidate transformation functions include: hash function, encoding function, encryption function, etc.

[0096] An equivalent encoding transformation should satisfy the following: for a sufficient number of samples, the result of transforming the value of C using the candidate transformation function is completely consistent with the value of R; at the same time, if the candidate transformation function is deterministic (i.e., the same input always produces the same output), then the results of multiple repeated transformations should be consistent.

[0097] Based on the above characteristics, a preset number of samples are selected. The values ​​of the target field C are mapped to the values ​​of the candidate field R using the candidate transformation functions in the candidate transformation function set. Then, the inverse transformation is used to restore the samples. The number of samples that are not restored correctly (the restored values ​​are not equal) is counted. The ratio of the number of samples that are not restored correctly to the preset number is taken as the proportion of samples that are not restored correctly. The closer this proportion is to 0, the better the reversibility of the candidate transformation function. The proportion of samples that are not restored correctly is taken as the one-way loss value. In this embodiment, the preset number is 1000. In specific applications, the implementer can set it according to the specific situation.

[0098] Furthermore, for the target field C, the same transformation function is used to process it a preset number of times, checking whether the results obtained each time are consistent. The ratio of the number of consistent results to the preset number of times is taken as the proportion of consistent results, and the proportion of consistent results is taken as the deterministic confidence level. A deterministic confidence level of 1 indicates that the function is deterministic, that is, the same input always produces the same output; a deterministic confidence level less than 1 indicates that the function may contain random components. In this embodiment, the preset number of times is 100. In specific applications, the implementer can set it according to the specific situation. Therefore, it is necessary to set a judgment rule. Only when the following three conditions are met simultaneously is it determined that there is an equivalence relationship of encoding heterogeneous types:

[0099] Condition 1: Forward mapping verification passes; specifically, forward mapping verification involves mapping the target field C using a candidate transformation function. If the mapped value equals the value of the candidate field R, then the forward mapping verification passes.

[0100] Condition 2: The one-way loss value is less than or equal to the first preset threshold; when the one-way loss value is less than or equal to the first preset threshold, it indicates that the reversibility is good; in this embodiment, the first preset threshold is 0.05, which means that a maximum of 5% of the samples cannot be restored. This is because there may be data quality problems in the actual data (such as a small amount of dirty data, inconsistent format, etc.). The 5% fault tolerance rate can avoid the failure of the entire verification due to a very small number of abnormal data.

[0101] Condition 3: The certainty confidence level is greater than or equal to the second preset threshold; when the certainty confidence level is greater than or equal to the second preset threshold, it indicates that the encoding transformation is stable; in this embodiment, the second preset threshold is 0.95, which requires that more than 95% of the repeated test results are consistent. This is to exclude hash functions containing random salts, because although random salt hashes are often used for data protection, their irreversibility makes them unsuitable for field mapping.

[0102] If all functions in the candidate function set fail the verification, the field pair is marked as low confidence and the reason for failure is recorded (i.e., which of the three conditions mentioned above is not met). Then, the process reverts to the candidate mapping graph, the field pair is no longer processed, and an early warning message is generated, prompting the administrator to manually confirm the coding rules.

[0103] Step S5: Generate and execute the corresponding instruction based on the type of equivalence relation.

[0104] In this embodiment, all equivalence relations were quantified in the above steps. Next, the implicit equivalence relations need to be converted into structured correction instructions that can be executed by the privacy computing platform. This will drive the platform to automatically complete the field mapping reconstruction during the task initialization phase, ensuring that subsequent core tasks such as privacy intersection and federated learning are executed on the basis of correct field alignment.

[0105] The verification results contain three types of information: verified equivalence relations, relations with low confidence requiring manual intervention, and relations that failed verification. Only verified equivalence relations require the generation of correction instructions; relations with low confidence require the generation of warning instructions to prompt manual review; and relations that failed verification do not generate any instructions.

[0106] For equivalence relations of composite split type, generate instructions including scenario type, source participant identifier, source field list, target participant identifier, target field name and execution action; value description: fixed value "composite split", system identifier of data provider, array of field names to be combined, system identifier of data receiver, target field name corresponding to the combination, and fixed value "combination mapping".

[0107] As a concrete example, by introducing a preset delimiter dictionary (containing null characters, spaces, commas, and underscores), multiple sets of concatenation results can be generated through the delimiters for verification, thus solving the problem of overly rigid constraints in direct concatenation.

[0108] For equivalence relations of heterogeneous encoding types, generate instructions that include scenario type, source participant identifier, source field name, target participant identifier, target field name, execution action and transformation rules; value descriptions: fixed value "heterogeneous encoding", system identifier of data provider, encoded field name, system identifier of data receiver, decoded target field name, fixed value "encoding conversion", and specific decoding / inverse transformation method description.

[0109] For low-confidence field pairs, generate warning instructions but do not perform automatic mapping.

[0110] Furthermore, the generated instruction set is sent in batches to the task scheduling module through a standardized interface provided by the privacy computing platform. This interface supports data exchange in JSON format, and the platform returns a confirmation status upon receiving the instruction.

[0111] When executing a composite splitting instruction, the corresponding field in the data source is located based on the source participant identifier and source field list in the instruction; during the data loading phase, a field value concatenation operation is performed on each record, with the combination method being sequential string concatenation and the concatenation operator being an empty string (i.e., direct concatenation); the combined value is mapped to the target field specified in the instruction, replacing the original field mapping relationship; the execution information of this mapping operation is recorded, including the source field, target field, execution time, and number of records affected.

[0112] When executing instructions for heterogeneous encoding types, the corresponding field in the data source is located based on the source participant identifier and source field name in the instruction. During the data loading phase, the inverse transformation operation specified in the transformation rules is executed for each record. For verification and mapping of hash-type encodings, direct reverse decoding is not performed. Instead, a forward collision verification mechanism is adopted. That is, after the verification is passed (the forward mapping verification of condition 1 is passed), the original value of the first participant is forward encrypted according to the corresponding hash algorithm, so that it maintains the same encoding form as the heterogeneous field of the second participant. This allows subsequent privacy calculations to be completed in ciphertext or hash state, and the table is restored by looking up the table using a pre-set rainbow table or dictionary. For reversible encodings (such as Base64), the corresponding decoding function is executed. The transformed value is mapped to the target field specified in the instruction, replacing the original field mapping relationship. The execution information of this transformation operation is recorded, including the source field, target field, transformation rule, execution time, and number of records successfully restored.

[0113] After all correction instructions have been executed, the platform has completed the reconstruction of the field mapping relationship. At this time, the platform starts privacy computing tasks (such as privacy intersection, federated learning, etc.) according to the original process. All tasks will be executed based on the aligned fields. For any exceptions that occur during the execution process (such as source field not existing, transformation failure, record number mismatch, etc.), the platform will interrupt task initialization and return an error report, including the exception type, the instructions involved, and specific error information, for administrators to troubleshoot.

[0114] This embodiment obtains the fields of the first and second participants and extracts their semantic vectors and distribution features. It then obtains a comprehensive similarity score based on the similarity of the semantic vectors and distribution features, filters initial field pairs based on this score, and constructs a candidate mapping graph. Based on the candidate mapping graph, it selects target fields and candidate fields to generate field combinations. The reconstruction deviation is determined based on the mutual information between the field combination and the target field, and the information entropy of the target field. The reconstruction deviation is used to determine whether there is an equivalence relationship for composite splitting types. Based on the candidate mapping graph, it selects target fields and candidate fields to construct a candidate transformation function set and determines whether there is an equivalence relationship for heterogeneous encoding types. Finally, it generates and executes corresponding instructions based on the type of equivalence relationship. This method can automatically identify the equivalence relationships between single and multiple fields in business logic, as well as the equivalence relationships when there are heterogeneous encoding transformations, without relying on explicit field name matching or simple character similarity calculations. This accurately determines the cause of field mismatches and outputs directional correction instructions with field mapping reconstruction capabilities, improving the success rate and accuracy of privacy computing tasks in heterogeneous data source scenarios.

[0115] An example of a privacy-computing-driven data circulation system:

[0116] See Figure 2 The diagram illustrates a structural block diagram of a privacy-driven data circulation system according to an embodiment of the present invention. The system may include an acquisition module, a relationship construction module, a first judgment module, a second judgment module, and an execution module.

[0117] The acquisition module is used to acquire the fields of the first and second participants and extract the semantic vectors and distribution features of the fields.

[0118] The relationship building module is used to obtain a comprehensive similarity based on the similarity of the semantic vectors and distribution features of the fields of the first participant and the second participant; it then filters initial field pairs based on the comprehensive similarity and uses these initial field pairs to build a candidate mapping relationship graph.

[0119] The first judgment module is used to select the target field of the first participant and the candidate fields similar to the target field in the second participant based on the candidate mapping relationship graph, and generate a field combination; determine the reconstruction deviation degree based on the mutual information between the field combination and the target field and the information entropy of the target field; and determine whether there is an equivalence relationship of composite split type between the field combination and the target field based on the reconstruction deviation degree.

[0120] The second judgment module is used to select the target field of the first participant and the candidate field of the second participant based on the candidate mapping relationship graph, and construct a candidate transformation function set; based on the candidate transformation function set, it is determined whether there is an equivalence relationship of encoding heterogeneous types between the target field and the candidate field;

[0121] The execution module is used to generate and execute corresponding instructions based on the type of equivalence relation.

[0122] It should be understood that Figure 2 The block diagram and modules of the privacy-driven data circulation system shown can be implemented in various ways. For example, in some embodiments, the system and its modules can be implemented in hardware, software, or a combination of both. The hardware portion can be implemented using dedicated logic; the software portion can be stored in memory and executed by appropriate instructions, such as a microprocessor or dedicated hardware. Those skilled in the art will understand that the methods and systems described above can be implemented using computer-executable instructions and / or included in processor control code, for example, on a media such as a disk, CD, or DVD-ROM, a programmable memory such as read-only memory (firmware), or a data carrier such as an optical or electronic signal carrier. The systems and modules of this specification can be implemented not only with hardware circuits such as very large-scale integrated circuits or gate arrays, semiconductors such as logic chips, transistors, or programmable hardware devices such as field-programmable gate arrays, programmable logic devices, etc., but also with software, for example, executed by various types of processors, or with a combination of the aforementioned hardware circuits and software (e.g., firmware).

[0123] For more details about the above modules, please refer to other parts of this manual; they will not be repeated here.

[0124] It should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A privacy-preserving computation-driven data flow method, characterized in that, The method includes the following steps: Obtain the fields of the first and second participants, and extract the semantic vectors and distribution features of the fields; A comprehensive similarity is obtained based on the similarity of the semantic vectors and distribution features of the fields of the first participant and the second participant; initial field pairs are selected based on the comprehensive similarity, and candidate mapping relationship graphs are constructed using the initial field pairs; Based on the candidate mapping relationship graph, the target field of the first participant and the candidate fields similar to the target field in the second participant are selected to generate a field combination; the reconstruction deviation is determined according to the mutual information between the field combination and the target field and the information entropy of the target field; and the reconstruction deviation is used to determine whether there is an equivalence relationship of composite split type between the field combination and the target field. Based on the candidate mapping relationship graph, the target field of the first participant and the candidate field of the second participant are selected to construct a candidate transformation function set; based on the candidate transformation function set, it is determined whether there is an equivalence relationship of encoding heterogeneous types between the target field and the candidate field. Based on the type of equivalence relation, generate the corresponding instruction and execute it; The step of determining whether there is an equivalence relationship of composite split type between the field combination and the target field based on the reconstruction deviation includes: The reconstruction deviation of all field combinations corresponding to the same target field is sorted in ascending order to obtain the reconstruction deviation sequence; Calculate the second difference between adjacent reconstruction deviations in the reconstruction deviation sequence, and use the largest second difference as the dividing point to extract the low deviation cluster in the reconstruction deviation sequence; Based on the number of combinations within the low deviation cluster, the value of the reconstruction deviation, and the second difference, determine whether there is an equivalence relationship of composite split type between the optimal equivalent combination and the target field; The step of determining whether there is an equivalence relationship between the target field and the candidate fields based on the candidate transformation function set includes: Select a preset number of samples, use the candidate transformation functions in the candidate transformation function set to map the value of the target field to the value of the candidate field, and perform inverse transformation restoration. The proportion of samples that are not restored correctly is used as the one-way loss value. The same transformation function is repeatedly applied to the value of the target field for a preset number of mapping and inverse transformation restoration processes, and the proportion of the number of times the results are consistent is used as the deterministic confidence level. When the forward mapping verification is successful, the one-way loss value is less than or equal to the first preset threshold, and the deterministic confidence is greater than or equal to the second preset threshold, it is determined that there is an equivalence relationship between the target field and the candidate field in terms of encoding heterogeneous types.

2. The privacy-computing-driven data flow method according to claim 1, characterized in that, The comprehensive similarity is obtained based on the similarity of the semantic vectors and distribution features of the fields of the first participant and the second participant, including: Calculate the cosine similarity between the semantic vectors of the fields of the first participant and the fields of the second participant, and use it as the semantic similarity. The statistical feature values ​​of the fields of the first participant and the fields of the second participant are extracted and normalized respectively. The distribution similarity is obtained based on the difference between the normalized statistical feature values ​​of the fields of the first participant and the normalized statistical feature values ​​of the fields of the second participant. The statistical feature values ​​are used to characterize the distribution features. The semantic similarity and the distributional similarity are weighted and summed to obtain the comprehensive similarity.

3. The privacy-computing-driven data flow method according to claim 1, characterized in that, The initial field pair selection based on comprehensive similarity includes: Sort all the comprehensive similarity scores in ascending order to obtain the comprehensive similarity score sequence; Calculate the first difference between adjacent data in the comprehensive similarity sequence, and use the largest first difference as the dividing point to divide the comprehensive similarity sequence into two parts; use the field pairs corresponding to the right part as the initial field pairs.

4. The privacy-computing-driven data flow method according to claim 1, characterized in that, The step of determining the reconstruction deviation based on the mutual information between the field combination and the target field and the information entropy of the target field includes: calculating a first ratio of the mutual information to the information entropy; and using the difference between the constant 1 and the first ratio as the reconstruction deviation.

5. The privacy-computing-driven data flow method according to claim 1, characterized in that, The step of determining whether there is an equivalence relationship of composite split type between the optimal equivalent combination and the target field based on the number of combinations within the low deviation cluster, the value of the reconstruction deviation, and the second difference includes: The combination with the smallest reconstructed deviation in the low deviation cluster is taken as the optimal equivalent combination; If the preset conditions are met, it is determined that there is no equivalence relationship of composite split type between the optimal equivalent combination and the target field; If the preset conditions are not met, it is determined that there is an equivalence relationship of composite split type between the optimal equivalent combination and the target field; The preset conditions are: the largest second difference is less than a preset difference threshold, or the number of combinations contained in the low deviation cluster exceeds a preset proportion of the total number of combinations, or the minimum reconstruction deviation in the low deviation cluster is greater than a preset reconstruction deviation threshold.

6. A privacy-computing-driven data flow method according to claim 1, characterized in that, The step of generating corresponding instructions based on the type of equivalence relation includes: For equivalence relations of composite split types, generate instructions that include scenario type, source participant identifier, source field list, target participant identifier, target field name, and execution action; For equivalence relations of heterogeneous encoding types, generate instructions that include scene type, source participant identifier, source field name, target participant identifier, target field name, execution action, and transformation rules.

7. A privacy-computing-driven data flow method according to claim 6, characterized in that, the instruction... include: When executing a compound splitting instruction, during the data loading phase, when performing field value concatenation on each record, the delimiters in the preset delimiter dictionary are traversed to generate multiple sets of concatenation results for verification, and the concatenated values ​​are mapped to the specified target fields. When executing instructions that encode heterogeneous types, the inverse transformation operation specified in the transformation rules is performed on each record during the data loading phase, mapping the transformed value to the specified target field.

8. A privacy-preserving computing-driven data circulation system, the system being used to implement the method of claim 1, characterized in that, The system includes: The acquisition module is used to acquire the fields of the first and second participants, and extract the semantic vectors and distribution features of the fields; The relationship building module is used to obtain a comprehensive similarity based on the similarity of the semantic vectors and distribution features of the fields of the first participant and the second participant; it then filters initial field pairs based on the comprehensive similarity and uses these initial field pairs to build a candidate mapping relationship graph. The first judgment module is used to select the target field of the first participant and the candidate fields similar to the target field in the second participant based on the candidate mapping relationship graph, and generate a field combination; determine the reconstruction deviation degree based on the mutual information between the field combination and the target field and the information entropy of the target field; and determine whether there is an equivalence relationship of composite split type between the field combination and the target field based on the reconstruction deviation degree. The second judgment module is used to select the target field of the first participant and the candidate field of the second participant based on the candidate mapping relationship graph, and construct a candidate transformation function set; based on the candidate transformation function set, it is determined whether there is an equivalence relationship of encoding heterogeneous types between the target field and the candidate field; The execution module is used to generate and execute corresponding instructions based on the type of equivalence relation.

Images