Data evaluation method and device based on multi-party secure computation, equipment and medium

Abstract

This disclosure relates to the field of artificial intelligence technology, and more particularly to a data evaluation method, apparatus, device, and medium based on multi-party secure computation. The method includes: obtaining data description information from a platform; sending requirement information to the platform based on the data description information, so that the platform can build an experimental sub-platform according to the requirement information; wherein the experimental sub-platform includes a first sample dataset matching the data description information; sending a preset model to be tested and a preset second sample dataset to the platform, so that the experimental sub-platform on the platform can test the model to be tested based on the first and second sample datasets, and obtain test results; obtaining the test results; analyzing and evaluating the test results to obtain a data evaluation report, wherein the data evaluation report is used to characterize the matching degree of the first sample dataset to the model to be tested. The solution of this application can realize data evaluation and reduce decision-making costs.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a data evaluation method, apparatus, device, and medium based on multi-party secure computation. Background Technology

[0002] Models require extensive training data to achieve the desired results. However, it's often difficult for training providers to offer massive amounts of data for specific models, necessitating the purchase of data from suppliers to meet training requirements. In data transactions, besides being influenced by data ownership and pricing, the compatibility between the data and the specific model plays a crucial role.

[0003] In related technologies, the assessment of the matching degree between data and a specific model involves directly providing the seller's data to the buyer for testing, or directly providing the buyer's model to be trained to the seller for testing. This operation is prone to data leakage from either the seller or the buyer, which is even more detrimental to data transactions. Summary of the Invention

[0004] The main objective of this disclosure is to propose a data evaluation method, apparatus, device, and medium based on multi-party secure computation, so as to achieve data evaluation, reduce decision-making costs, avoid data leakage between buyers and sellers, protect the data privacy of buyers and sellers, and improve data security.

[0005] To achieve the above objectives, a first aspect of this disclosure proposes a data evaluation method based on multi-party secure computation, which is applied to the user end.

[0006] The methods include:

[0007] Data and information are obtained from the platform.

[0008] The system sends requirement information to the platform based on the data description information, so that the platform can build an experimental sub-platform according to the requirement information; wherein, the experimental sub-platform includes a first sample dataset that matches the data description information;

[0009] The preset test model and the preset second sample dataset are sent to the platform so that the experimental sub-platform on the platform can test the test model based on the first sample dataset and the second sample dataset and obtain the test results.

[0010] Obtain test results;

[0011] The test results are analyzed and evaluated to obtain a data evaluation report; the data evaluation report is used to characterize the matching degree of the first sample dataset to the model under test.

[0012] In some embodiments, the data evaluation report includes an information value report;

[0013] The test results are analyzed and evaluated to obtain a data evaluation report, including:

[0014] The relationships are obtained from the platform; these relationships include horizontal federation or vertical federation, and are used to characterize the connection between the first sample dataset and the second sample dataset.

[0015] The test results are grouped according to the correlation to obtain several groups whose information value needs to be evaluated.

[0016] Calculate the information value index for each information value group to be evaluated to obtain the target information value set;

[0017] The target information value set is statistically classified to obtain an information value report.

[0018] In some embodiments, statistical classification processing is performed on the target information value set to obtain an information value report, including:

[0019] The target information value set is classified according to the preset first information threshold and the preset second information threshold to obtain the first information value set, the second information value set and the third information value set;

[0020] Statistical analysis was performed on the first, second, and third information value sets respectively to obtain information value reports.

[0021] In some embodiments, the data evaluation report also includes a model value evaluation report:

[0022] The test results are analyzed and evaluated to obtain a data evaluation report, including:

[0023] The test results are subjected to target operations to obtain a model value assessment report; wherein the target operations include at least one of the following operations: area under the curve calculation, Kolmogorov-Smirnov test calculation, precision calculation, recall calculation, Shapley score calculation, and balanced F-score calculation. To achieve the above objectives, a second aspect of this disclosure proposes a data evaluation method based on multi-party secure computation, applied to a platform.

[0024] The methods include:

[0025] Obtain the first sample dataset from the supply side;

[0026] Data description information is generated based on the first sample dataset, so that the user can generate requirement information based on the data description information;

[0027] The receiving end generates requirement information based on the data description information;

[0028] An experimental sub-platform is built based on the requirements information; the experimental sub-platform includes a first sample dataset that matches the data description information;

[0029] Receive the test model and the second sample dataset sent by the user.

[0030] The test sub-platform is invoked to test the model under test based on the first and second sample datasets, and the test results are obtained.

[0031] The test results are sent to the user so that the user can obtain a data evaluation report based on the test results.

[0032] In some embodiments, testing the model to be tested based on a first sample dataset and a second sample dataset includes:

[0033] A connection is established between the first sample dataset and the second sample dataset based on a preset association relationship to obtain a joint dataset; wherein the association relationship includes horizontal federation or vertical federation.

[0034] The joint dataset is normalized and classified to obtain the training dataset and the adversarial dataset.

[0035] The model to be tested is trained based on the training dataset and the adversarial dataset.

[0036] In some embodiments, the joint dataset is subjected to normalization classification to obtain a training dataset and an adversarial dataset, including:

[0037] The joint dataset is normalized to obtain the dataset to be evaluated;

[0038] The dataset to be evaluated is classified according to a preset weight ratio to obtain the training dataset and the adversarial dataset.

[0039] To achieve the above objectives, a third aspect of this disclosure provides a data evaluation apparatus based on multi-party secure computation, applied at the user end, the apparatus comprising:

[0040] The information acquisition module is used to obtain data description information from the platform.

[0041] The first sending module is used to send demand information to the platform based on the data description information, so that the platform can build an experimental sub-platform according to the demand information; wherein, the experimental sub-platform includes a first sample dataset that matches the data description information;

[0042] The second sending module is used to send the preset test model and the preset second sample dataset to the platform, so that the experimental sub-platform on the platform can test the test model according to the first sample dataset and the second sample dataset and obtain the test results.

[0043] The result acquisition module is used to obtain test results;

[0044] The analysis and evaluation module is used to analyze and evaluate the test results and obtain a data evaluation report. The data evaluation report is used to characterize the matching degree of the first sample dataset to the model under test.

[0045] To achieve the above objectives, a fourth aspect of this disclosure provides an electronic device, comprising:

[0046] At least one memory;

[0047] At least one processor;

[0048] At least one program;

[0049] The program is stored in memory, and the processor executes at least one program to achieve the following:

[0050] The method as described in any one of the embodiments of the first aspect; or,

[0051] The method of any one of the embodiments of the second aspect.

[0052] To achieve the above objectives, a fifth aspect of this disclosure provides a storage medium, which is a computer-readable storage medium storing computer-executable instructions for causing a computer to perform the following:

[0053] The method as described in any one of the embodiments of the first aspect; or,

[0054] The method of any one of the embodiments of the second aspect.

[0055] The data evaluation method, apparatus, device, and medium based on multi-party secure computation proposed in this disclosure allow the user to obtain data description information from a platform and generate and send requirement information based on this information. The platform then builds an experimental sub-platform to test the user's model to be tested and a second sample dataset, obtaining test results. The user analyzes and evaluates the test results to obtain a data evaluation report characterizing the matching degree of the first sample dataset to the model to be tested. This setup facilitates data evaluation by the user, reduces decision-making costs, avoids data leakage between buyers and sellers, protects the data privacy of both parties, improves data security, and safeguards the interests of data providers. Attached Figure Description

[0056] Figure 1 This is a block diagram of a data evaluation system based on multi-party secure computation provided in the embodiments of this application;

[0057] Figure 2 This is a first flowchart of the data evaluation method based on multi-party secure computation provided in the embodiments of this application;

[0058] Figure 3 yes Figure 2 A flowchart illustrating the specific method of step S205;

[0059] Figure 4 yes Figure 3 A flowchart illustrating the specific method of step S304 in the middle section;

[0060] Figure 5 This is a second flowchart of the data evaluation method based on multi-party secure computation provided in the embodiments of this application;

[0061] Figure 6 yes Figure 5 A flowchart illustrating the specific method of step S506;

[0062] Figure 7 yes Figure 6 Flowchart of the specific method for step S602;

[0063] Figure 8 This is a block diagram of a data evaluation device based on multi-party secure computation provided in the embodiments of this application;

[0064] Figure 9 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0065] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0066] It should be noted that although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first," "second," etc., in the specification, claims, and the aforementioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0067] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0068] First, let's analyze some of the terms used in this application:

[0069] Artificial intelligence (AI) is a new branch of computer science that studies, develops, and applies theories, methods, technologies, and systems to simulate, extend, and expand human intelligence. It aims to understand the essence of intelligence and produce intelligent machines that can react in a way similar to human intelligence. Research in this field includes robotics, speech recognition, image recognition, natural language processing, and expert systems. AI can simulate the information processes of human consciousness and thought. Furthermore, AI utilizes digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceiving the environment, acquiring knowledge, and using that knowledge to achieve optimal results.

[0070] Multi-party computation (MPC): As a subfield of cryptography, MPC allows multiple data owners to collaboratively compute data without mutual trust, outputting the results while ensuring that no party receives any information other than the intended computation result. In other words, MPC technology can extract the value of data without revealing its original content. MPC has the following characteristics:

[0071] (1) Homomorphic Encryption (HE)

[0072] Homomorphic encryption is a type of encryption method with special natural properties, capable of performing data operations within the ciphertext domain. Compared to general encryption algorithms, homomorphic encryption, in addition to implementing basic encryption operations, can also perform various computational functions between ciphertexts; that is, computation before decryption is equivalent to decryption before computation.

[0073] (2) Garbled Circuit (GC)

[0074] The concept of obfuscated circuits utilizes computer simulation of integrated circuits to achieve secure multi-party computation. It transforms computational tasks into the form of gate circuits and encrypts each line, thus greatly protecting user privacy and security.

[0075] (3) Oblivious Transfer (OT)

[0076] Unintentional transfer protocol is a privacy-preserving secret protocol that allows the sender and receiver to exchange information unintentionally, thereby protecting privacy. It is a two-party secure computation protocol where the receiver selects a portion of the data from the sender's data. The protocol ensures that the receiver is completely unaware of the remaining data except for the selected content, and the sender is also unaware of the selected content.

[0077] (4) Secret Sharing (SS)

[0078] Secret sharing, also known as secret splitting, is a method of managing secret information. It splits secrets into pieces, each of which is managed by different participants. A single participant cannot recover the secret information; a certain number of people must collaborate to merge the pieces in order to recover the secret file.

[0079] Information value (IV): The IV value measures the degree of influence of a feature on a target. Its basic idea is to compare and calculate the degree of correlation based on the ratio of black and white samples hit by the feature to the total number of black and white samples. Because the calculation uses the proportion of black and white samples hit respectively, it avoids the bias caused by different choices of the number of black and white samples to a certain extent in engineering practice.

[0080] Area under the curve (AUC): AUC is defined as the area under the ROC curve. AUC is often used as a model evaluation metric because the ROC curve often doesn't clearly indicate which classifier performs better, and as a numerical value, a larger AUC corresponds to a better classifier.

[0081] Receiver operating characteristic curve (ROC): ROC is a curve plotted based on a series of different binary classification methods (cutoff values ​​or decision thresholds), with the true positive rate (sensitivity) as the ordinate and the false positive rate (1-specificity) as the abscissa.

[0082] Kolmogorov–Smirnov test (KS test): The KS test method can use sample data to infer whether the population from which the sample comes follows a certain theoretical distribution. It is a goodness-of-fit test method and is suitable for exploring the distribution of continuous random variables.

[0083] Precision: Also known as accuracy, it refers to the prediction result. It means the probability that a sample that is actually positive out of all samples that are predicted to be positive. In other words, how confident are you in predicting the correct result among the samples that are predicted to be positive?

[0084] Recall: Also known as the completeness of the sample, it refers to the original sample and represents the probability that a sample that is actually positive will be predicted as positive.

[0085] Shapley value (SV): The Shapley value reflects the degree of contribution of each party to the overall goal of the cooperation. It avoids egalitarianism in distribution and is more reasonable and fair than any distribution method that only considers the value of resource input, the efficiency of resource allocation, or a combination of the two. It also reflects the process of mutual game among the parties involved in the cooperation.

[0086] The balanced F-score, also known as the F1 score, is a statistical metric used to measure the precision of a binary classification model. It balances the precision and recall of the classification model. The F1 score can be seen as a harmonic average of the model's precision and recall, with a maximum value of 1 and a minimum value of 0.

[0087] The embodiments of this application can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence (AI) refers to the theories, methods, technologies, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.

[0088] Foundational technologies for artificial intelligence generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies mainly encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0089] Models require extensive training data to achieve desired results. However, training providers often struggle to offer the massive amounts of data needed for specific models, necessitating the purchase of data from suppliers to meet training requirements. In data transactions, besides data ownership and pricing, the compatibility between the data and the specific model plays a crucial role. Therefore, prior to a data transaction, the value of the data to be traded must be assessed to determine its suitability for the model. However, simply providing data providers with their data for computation undermines data security and harms the interests of the data providers.

[0090] Based on this, embodiments of this application provide a data evaluation method, system, device, and medium based on multi-party secure computation. This enables data users to evaluate the value of data while protecting the interests of data providers, reducing decision-making costs for data users, preventing data leakage between buyers and sellers, protecting the data privacy of both parties, improving data security, and safeguarding the interests of data providers.

[0091] The embodiments of this application will be further described below with reference to the accompanying drawings.

[0092] This application discloses a data evaluation method, apparatus, device, and medium based on multi-party secure computation, which relates to the field of artificial intelligence technology. The data evaluation method based on multi-party secure computation described in this application can be applied to a terminal, a server, or software running on either a terminal or a server. In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, or smartwatch, etc.; the server can be configured as an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms; the software can be an application implementing an activity classification model training method, a classification method, etc., but is not limited to the above forms.

[0093] This disclosure can be applied to numerous general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. This application can be described in the general context of computer-executable instructions, such as program modules, that are executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In a distributed computing environment, program modules can reside in local and remote computer storage media, including storage devices.

[0094] like Figure 1 As shown in the figure, this application embodiment provides a data evaluation system, which includes a user terminal 102, a supply terminal 103 and a platform terminal 101. Multiple user terminals 102 and supply terminals 103 can be set up and connected to the platform terminal 101 respectively. The system comprises the following components: Supply end 103 uploads data information to platform end 101, including a first sample dataset and data description information matching the first sample dataset; User end 102 obtains the data description information from platform end 101, generates demand information based on the data description information, and sends the demand information to platform end 101; Platform end 101 receives the demand information and builds an experimental sub-platform 104 based on the demand information, wherein the experimental sub-platform 104 includes a first sample dataset matching the data description information; User end 102 inputs a preset model to be tested and a preset second sample dataset into experimental sub-platform 104; Experimental sub-platform 104 tests the model to be tested based on the first and second sample datasets, obtains and outputs test results; User end 102 obtains the test results, analyzes and evaluates the test results, and obtains a data evaluation report, wherein the data evaluation report characterizes the matching degree of the first sample dataset to the model to be tested.

[0095] In the data evaluation system of this application embodiment, the supply end 103 uploads data information to the platform end 101, and the user end 102 can only obtain data description information from the platform end 101. This protects the interests of the supply end 103. When the user end 102 selects certain data, the supply end 103 generates a matching experimental sub-platform 104 based on the user end 102's requirements. This allows the user end 102 to test the model under test using a first data sample dataset and a preset second sample dataset, obtaining test results. The user end 102 then analyzes and evaluates the test results to obtain a data evaluation report characterizing the matching degree of the first sample dataset to the model under test. This setup facilitates data evaluation by the user end 102, reduces user decision-making costs, avoids data leakage between buyers and sellers, protects the data privacy of both parties, improves data security, and safeguards the interests of the data supplier.

[0096] based on Figure 1 The data evaluation system shown is based on... Figure 2 Some embodiments of this application provide a data evaluation method based on multi-party secure computation, applied to the user end. The method includes steps S201, S202, S203, S204, and S205. These five steps are described in detail below. It should be understood that the data evaluation method based on multi-party secure computation in the embodiments of this application includes, but is not limited to, steps S201 to S205.

[0097] Step S201: Obtain data introduction information from the platform.

[0098] In step S201, the platform is connected to multiple suppliers and multiple users. Each supplier can upload multiple data information to the platform. The platform will conduct a preliminary evaluation and review of the data information uploaded by the suppliers. After the preliminary evaluation and review are passed, the platform will only display the relevant data description information and hide the corresponding first sample dataset. The user can only obtain the data description information from the platform and cannot directly obtain the first sample dataset corresponding to the data description information.

[0099] Step S202: Send the requirement information to the platform according to the data description information, so that the platform can build the experimental sub-platform according to the requirement information; wherein, the experimental sub-platform includes a first sample dataset that matches the data description information.

[0100] In step S202, the user browses the data introduction information on the platform using the terminal, selects one or more data introduction information that meets the expected conditions based on multiple data introduction information, applies for trial and purchase to generate corresponding demand information, and sends the demand information to the platform so that the platform can build an experimental sub-platform according to the demand information.

[0101] Step S203: Send the preset test model and the preset second sample dataset to the platform so that the experimental sub-platform on the platform can test the test model based on the first sample dataset and the second sample dataset and obtain the test results.

[0102] In step S203, the experimental sub-platform acts as a secure testing sandbox computing environment. The user terminal is unaware of the specific data content of the first sample dataset; it can only call upon the first sample dataset to perform test calculations and obtain a test result. This effectively provides a multi-party secure computing environment where each party only receives the test result and is unaware of the specific data content. Furthermore, in this embodiment, the model to be tested is not a fixed model and can change according to variations in the user terminal. There is also no specific limit to the number of models to be tested; only one model can be input for testing, or multiple models can be input for testing.

[0103] Step S204: Obtain test results.

[0104] Step S205: Analyze and evaluate the test results to obtain a data evaluation report; wherein, the data evaluation report is used to characterize the matching degree of the first sample dataset with the model to be tested.

[0105] In step S205, the user analyzes and evaluates the test results to obtain a data evaluation report characterizing the matching degree of the first sample dataset to the model under test. If the user is satisfied with the data evaluation report, it means that the first sample dataset meets the user's data requirements. Based on this, the user can purchase data from the supplier through the platform. If not satisfied, the user can request testing from other data suppliers.

[0106] It should be noted that the first sample dataset and the second sample dataset mentioned in the embodiments of this application are both data that have been de-identified. Before entering the experimental sub-platform for testing, all data are homomorphically encrypted. The data processing in the experimental sub-platform is equivalent to multi-party secure collaborative computing.

[0107] The data evaluation method based on multi-party secure computation in this application involves the user obtaining data description information from the platform and generating and sending requirement information based on this information. The platform then builds an experimental sub-platform to test the user's model to be tested and a second sample dataset, obtaining test results. The user analyzes and evaluates the test results to obtain a data evaluation report characterizing the matching degree of the first sample dataset to the model to be tested. This setup facilitates data evaluation by the user, reduces decision-making costs, prevents data leakage between buyers and sellers, protects their data privacy, improves data security, and safeguards the interests of the data provider.

[0108] Reference Figure 3 In some embodiments of this application, the data evaluation report includes an information value report. Step S205 includes steps S301, S302, S303, and S304. These four steps are described in detail below. It should be understood that step S205 includes, but is not limited to, steps S301 to S304.

[0109] Step S301: Obtain the association relationship from the platform; wherein, the association relationship includes horizontal federation or vertical federation, and the association relationship is used to characterize the relationship between the first sample dataset and the second sample dataset.

[0110] In step S301, the experimental sub-platform on the platform side establishes a connection between the first sample dataset and the second sample dataset through a preset association relationship, resulting in a joint dataset. The connection can be established based on data type or data identification number (ID). Obtaining the association relationship from the platform side facilitates subsequent grouping and processing of the test results by the user end.

[0111] Step S302: Group the test results according to the correlation to obtain several information value groups to be evaluated.

[0112] In step S302, the test results are grouped according to the association relationships obtained in the preceding steps to obtain several groups whose information value needs to be evaluated. For example, if the experimental sub-platform establishes the connection between the first sample dataset and the second sample dataset based on ID, then the association relationship is ID. In this embodiment, the test results are divided into several groups whose information value needs to be evaluated based on the different IDs.

[0113] Step S303: Calculate the information value index for each information value group to be evaluated to obtain the target information value set.

[0114] In step S303, the IV value of each information value to be evaluated group is calculated to obtain the target information value set. The IV value is mainly used to encode and evaluate the predictive ability of the input variables. The magnitude of the IV value of a feature variable indicates the strength of its predictive ability. The range of the IV value is [0, positive infinity). If the current group only contains responding customers or non-responding customers, IV = positive infinity. It is calculated using formula (1):

[0115]

[0116] In formula (1), DistributionGood i This represents the percentage of white samples hit in group i. If we use "good"... i Indicates the number of samples that hit the group, good T If the total number of all white samples is represented, then Similarly, we can know that DistributionBad i .

[0117] Step S304: Perform statistical classification processing on the target information value set to obtain an information value report.

[0118] In step S304, the target information value set calculated in step S303 is statistically classified to obtain an information value report. Generally, an IV value less than 0.02 indicates that the model has no predictive ability; an IV value between 0.02 and 0.1 indicates that the model has weak predictive ability; an IV value between 0.1 and 0.3 indicates that the model has moderate predictive ability; an IV value between 0.3 and 0.5 indicates that the model has strong predictive ability; and an IV value greater than 0.5 indicates that the predictive performance is too good and not realistic. By statistically classifying the target information value set, the matching degree of the first sample dataset to the test model is determined.

[0119] Reference Figure 4 In some embodiments of this application, step S304 includes steps S401 and S402. The following describes... Figure 4 The two steps will be described in detail. It should be understood that step S304 includes, but is not limited to, steps S401 and S402.

[0120] Step S401: Classify the target information value set according to the preset first information threshold and the preset second information threshold to obtain the first information value set, the second information value set and the third information value set.

[0121] Step S402: Perform statistics on the first information value set, the second information value set, and the third information value set respectively to obtain an information value report.

[0122] Specifically, in this embodiment, the target information value set is divided into three categories—a first information value set, a second information value set, and a third information value set—by setting a first information threshold and a second information threshold. Then, statistical analysis is performed on these three information value sets to obtain an information value report. For example, if the first information threshold is 0.3 and the second information threshold is 0.5, the first information value set represents the set of information values ​​less than 0.3, the second information value set represents the set of information values ​​in the range of 0.3 to 0.5, and the third information value set represents the set of information values ​​greater than 0.5. Through this classification process, users can more clearly understand the matching degree of the first sample dataset to the model under test.

[0123] In some embodiments of this application, the data evaluation report also includes a model value evaluation report, and step S205 further includes, but is not limited to, the step "perform a target operation on the test results to obtain a model value evaluation report; wherein the target operation includes at least one of the following operations: area under the curve calculation, Kolmogorov-Smirnov test calculation, precision calculation, recall calculation, Shapley value calculation, and balanced F-score calculation".

[0124] Specifically, a model value assessment report is obtained by performing target operations on the test results. This setup, combining the model value assessment report with the IV value to evaluate the first sample dataset, yields more accurate results and facilitates user decision-making. In this embodiment, the target operation can also be: calculating the mean of the area under the curve, calculating the mean KS score, the mean precision, the mean recall, the mean F1-score, the maximum Shapley score, and the mean Shapley score, etc.

[0125] based on Figure 1 The data evaluation system shown is based on... Figure 5 Some embodiments of this application provide a data evaluation method based on multi-party secure computation, applied to a platform, the method including steps S501, S502, S503, S504, S505, S506 and S507.

[0126] Step S501: Obtain the first sample dataset from the supply side.

[0127] Step S502: Generate data description information based on the first sample dataset, so that the user can generate requirement information based on the data description information.

[0128] In steps S501 and S502, there are multiple suppliers. The platform obtains the first sample dataset of each supplier and generates corresponding data description information based on each first sample dataset. The user selects one or more data description information from the multiple data description information to purchase or try it, thereby generating corresponding demand information.

[0129] Step S503: Receive the requirement information generated by the user based on the data description information.

[0130] Step S504: Build an experimental sub-platform based on the requirements information; wherein, the experimental sub-platform includes a first sample dataset that matches the data description information.

[0131] In steps S503 and S504, the platform receives the requirement information generated by the user based on the data description information, and obtains the corresponding first sample dataset according to the data description information selected by the user. An experimental sub-platform is then built based on the requirement information, and the first sample dataset is input into the experimental sub-platform.

[0132] Step S505: Receive the test model and the second sample dataset sent by the user.

[0133] Step S506: Call the experimental sub-platform to test the model to be tested based on the first sample dataset and the second sample dataset, and obtain the test results.

[0134] In steps S505 and S506, the platform receives the model to be tested and the second sample dataset sent by the user, inputs the model to be tested and the second sample dataset into the experimental sub-platform, calls the experimental sub-platform, and tests the model to be tested based on the first sample dataset and the second sample dataset to obtain the test results.

[0135] It's important to note that the experimental sub-platform functions as a secure testing sandbox computing environment. Users cannot know the specific data content of the first sample dataset; they can only use it for testing and obtain a result. This essentially provides a multi-party secure computing environment where each party only receives the test result, without knowing the specific data content. The model under test is not fixed and can change depending on the user. There is also no specific limit to the number of models that can be tested; you can input only one model or multiple models for testing.

[0136] Step S507: Send the test results to the user so that the user can obtain a data evaluation report based on the test results.

[0137] In step S507, the platform sends the test results to the user so that the user can analyze and evaluate the test results and obtain a data evaluation report that characterizes the matching degree of the first sample dataset to the model under test.

[0138] The data evaluation method in this application involves the platform building an experimental sub-platform based on the user's needs. This sub-platform is used to test the user's model to be tested and the second sample dataset, yielding test results. The user then analyzes and evaluates these results to obtain a data evaluation report characterizing the matching degree of the first sample dataset to the model to be tested. This setup not only facilitates data evaluation for the user, reducing decision-making costs, but also prevents data leakage between buyers and sellers, protecting their data privacy, improving data security, and safeguarding the interests of the data provider.

[0139] Reference Figure 6 In some embodiments of this application, step S506 includes steps S601, S602, and S603. The following describes... Figure 6 The three steps will be described in detail. It should be understood that step S506 includes, but is not limited to, steps S601, S602 and S603.

[0140] Step S601: Establish a connection between the first sample dataset and the second sample dataset using a preset association relationship to obtain a joint dataset; wherein the association relationship includes horizontal federation or vertical federation.

[0141] In step S601, the experimental sub-platform on the platform side establishes a connection between the first sample dataset and the second sample dataset through a preset association relationship to obtain a joint dataset. The connection can be established by data type or by the data's identity ID.

[0142] Step S602: Perform normalization classification on the joint dataset to obtain the training dataset and the adversarial dataset.

[0143] Step S603: Train the model to be tested based on the training dataset and the adversarial dataset.

[0144] Reference Figure 7 In some embodiments of this application, step S602 includes steps S701 and S702. The following describes the steps in conjunction with... Figure 7 The two steps will be described in detail. It should be understood that step S602 includes, but is not limited to, steps S701 and S702.

[0145] Step S701: Normalize the joint dataset to obtain the dataset to be evaluated.

[0146] In step S701, normalization is performed to scale the data in the joint dataset to the interval [0,1] to obtain the dataset to be evaluated, thereby reducing the computational load.

[0147] Step S702: Classify the dataset to be evaluated according to the preset weight ratio to obtain the training dataset and the adversarial dataset.

[0148] In step S702, the preset weight ratio can be 8:2 or other weight ratios. Assuming the weight ratio is 8:2, the dataset to be evaluated is randomly classified according to the ratio of training dataset to adversarial dataset = 8:2. This is equivalent to classifying the original data into training and validation sets during model training, which facilitates the training of the model to be tested.

[0149] Reference Figure 8 Some embodiments of this application also provide a data evaluation device based on multi-party secure computation, applied to the user end. The device includes an information acquisition module 801, a first sending module 802, a second sending module 803, a result acquisition module 804, and an analysis and evaluation module 805.

[0150] The information acquisition module 801 is used to acquire data introduction information from the platform.

[0151] The first sending module 802 is used to send demand information to the platform based on the data description information, so that the platform can build an experimental sub-platform based on the demand information; wherein, the experimental sub-platform includes a first sample dataset that matches the data description information.

[0152] The second sending module 803 is used to send the preset test model and the preset second sample dataset to the platform, so that the experimental sub-platform on the platform can test the test model according to the first sample dataset and the second sample dataset and obtain the test results.

[0153] The result acquisition module 804 is used to acquire test results.

[0154] The analysis and evaluation module 805 is used to analyze and evaluate the test results and obtain a data evaluation report. The data evaluation report is used to characterize the matching degree of the first sample dataset to the model under test.

[0155] The data evaluation device based on multi-party secure computation in this application embodiment obtains data description information from the platform and generates and sends demand information based on the data description information. This allows the platform to build an experimental sub-platform based on the demand information, and then tests the model to be tested and a second sample dataset through the experimental sub-platform to obtain test results. By analyzing and evaluating the test results, a data evaluation report characterizing the matching degree of the first sample dataset to the model to be tested is obtained. This setup facilitates data evaluation by the user, reduces user decision-making costs, avoids data leakage between buyers and sellers, protects the data privacy of both parties, improves data security, and safeguards the interests of the data provider.

[0156] It should be noted that the data evaluation device based on multi-party secure computation in this application corresponds to the aforementioned data evaluation method based on multi-party secure computation, and the evaluation steps are similar. For details, please refer to the aforementioned data evaluation method based on multi-party secure computation, which will not be repeated here.

[0157] This application also provides an electronic device, including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the data evaluation method based on multi-party secure computation of this application.

[0158] The following is combined Figure 9 The hardware structure of the electronic device is described in detail. The electronic device includes: a processor 901, a memory 902, an input / output interface 903, a communication interface 904, and a bus 905.

[0159] The processor 901 can be implemented using a general-purpose central processing unit (CPU), microprocessor, application specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application.

[0160] The memory 902 can be implemented as a read-only memory (ROM), static storage device, dynamic storage device, or random access memory (RAM). The memory 802 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 802 and is called and executed by the processor 801 using the network service deployment method of the embodiments of this application.

[0161] The input / output interface 903 is used to implement information input and output;

[0162] Communication interface 904 is used to enable communication and interaction between this device and other devices. Communication can be achieved via wired means (e.g., USB, Ethernet cable) or wireless means (e.g., mobile network, Wi-Fi, Bluetooth).

[0163] Bus 905 transmits information between various components of the device (e.g., processor 901, memory 902, input / output interface 903, and communication interface 904);

[0164] The processor 901, memory 902, input / output interface 903, and communication interface 904 are connected to each other within the device via bus 905.

[0165] This application also provides a computer-readable storage medium storing a program that is executed by a processor to implement the data evaluation method based on multi-party secure computation as described in this application.

[0166] In one embodiment, the computer-readable storage medium stores computer-executable instructions that are executed by one or more control processors, for example, executing... Figure 2 Method steps S201 to S205, Figure 3 Method steps S301 to S304 in the above method Figure 4 Method steps S401 to S402, Figure 5 Method steps S501 to S507 in the above method Figure 6 Method steps S601 to S603, execution Figure 7 The method steps S701 to S702.

[0167] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0168] Those skilled in the art will understand that all or some of the steps and systems in the methods disclosed above can be implemented as software, firmware, hardware, and suitable combinations thereof. Some or all of the physical components can be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit. Such software can be distributed on a computer-readable medium, which can include computer storage media (or non-transitory media) and communication media (or transient media). As is known to those skilled in the art, the term computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, storage device storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible to a computer. Furthermore, as is known to those skilled in the art, communication media typically include computer-readable instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.

[0169] It should also be understood that the various implementation methods provided in this application can be combined arbitrarily to achieve different technical effects.

[0170] The above provides a detailed description of the preferred embodiments of this application. However, this application is not limited to the above-described embodiments. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of this application. All such equivalent modifications or substitutions are included within the scope defined by the claims of this application.

Claims

1. A data evaluation method based on multi-party secure computation, characterized in that, Applied to the user end; The method includes: Obtain data and information from the platform. The platform sends requirement information to the platform based on the data description information, so that the platform can build an experimental sub-platform based on the requirement information; wherein, the experimental sub-platform includes a first sample dataset that matches the data description information; The preset test model and the preset second sample dataset are sent to the platform so that the experimental sub-platform of the platform can test the test model based on the first sample dataset and the second sample dataset and obtain the test results. Obtain the test results; The test results are analyzed and evaluated to obtain a data evaluation report; wherein the data evaluation report is used to characterize the matching degree of the first sample dataset to the model under test.

2. The method according to claim 1, characterized in that, The data evaluation report includes an information value report; The analysis and evaluation of the test results to obtain a data evaluation report includes: The association relationship is obtained from the platform; wherein, the association relationship includes horizontal federation or vertical federation, and the association relationship is used to characterize the relationship between the first sample dataset and the second sample dataset; The test results are grouped according to the aforementioned correlation to obtain several information value groups to be evaluated. Calculate the information value index for each information value group to be evaluated to obtain the target information value set; The target information value set is statistically classified to obtain the information value report.

3. The method according to claim 2, characterized in that, The step of performing statistical classification processing on the target information value set to obtain the information value report includes: The target information value set is classified according to a preset first information threshold and a preset second information threshold to obtain a first information value set, a second information value set and a third information value set. The information value report is obtained by statistically analyzing the first information value set, the second information value set, and the third information value set respectively.

4. The method according to any one of claims 1 to 3, characterized in that, The data evaluation report also includes a model value evaluation report: The analysis and evaluation of the test results to obtain a data evaluation report includes: The test results are subjected to target operations to obtain a model value assessment report; wherein the target operations include at least one of the following operations: area under the curve calculation, Kolmogorov-Smirnov test calculation, precision calculation, recall calculation, Shapley value calculation, and balanced F-score calculation.

5. A data evaluation method based on multi-party secure computation, characterized in that, Applied to the platform side; The method includes: Obtain the first sample dataset from the supply side; Data description information is generated based on the first sample dataset, so that the user can generate requirement information based on the data description information; Receive the requirement information generated by the user terminal based on the data description information; An experimental sub-platform is built based on the aforementioned requirements information; wherein, the experimental sub-platform includes a first sample dataset that matches the aforementioned data description information; Receive the test model and the second sample dataset sent by the user terminal; The test sub-platform is invoked to test the model under test based on the first sample dataset and the second sample dataset, and the test results are obtained. The test results are sent to the user terminal so that the user terminal can obtain a data evaluation report based on the test results.

6. The method according to claim 5, characterized in that, The step of testing the model to be tested based on the first sample dataset and the second sample dataset includes: A connection is established between the first sample dataset and the second sample dataset based on a preset association relationship to obtain a joint dataset; wherein, the association relationship includes horizontal federation or vertical federation; The joint dataset is subjected to normalization classification to obtain the training dataset and the adversarial dataset. The model to be tested is trained using the training dataset and the adversarial dataset.

7. The method according to claim 6, characterized in that, The normalization and classification process performed on the joint dataset to obtain the training dataset and the adversarial dataset includes: The joint dataset is normalized to obtain the dataset to be evaluated; The dataset to be evaluated is classified according to a preset weight ratio to obtain a training dataset and an adversarial dataset.

8. A data evaluation device based on multi-party secure computation, characterized in that, The device, applied to the user end, includes: The information acquisition module is used to obtain data description information from the platform. The first sending module is used to send demand information to the platform terminal according to the data description information, so that the platform terminal can build an experimental sub-platform according to the demand information; wherein, the experimental sub-platform includes a first sample dataset that matches the data description information; The second sending module is used to send a preset test model and a preset second sample dataset to the platform, so that the experimental sub-platform of the platform can test the test model based on the first sample dataset and the second sample dataset and obtain test results. The result acquisition module is used to acquire the test results; An analysis and evaluation module is used to analyze and evaluate the test results to obtain a data evaluation report, wherein the data evaluation report is used to characterize the matching degree of the first sample dataset to the model under test.

9. An electronic device, characterized in that, include: At least one memory; At least one processor; At least one program; The program is stored in the memory, and the processor executes the at least one program to achieve the following: The method as described in any one of claims 1 to 4; or, The method as described in any one of claims 5 to 7.

10. A storage medium, said storage medium being a computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions for causing a computer to perform: The method as described in any one of claims 1 to 4; or, The method as described in any one of claims 5 to 7.

Images