Method, device, equipment and storage medium for processing classification model prediction results

Abstract

The present application discloses a processing method, device, equipment and storage medium for the prediction results of a classification model, and relates to artificial intelligence, machine learning and big data technology in computer technology. The specific implementation scheme is: by performing ciphertext calculation based on the classification prediction results of the ciphertext, the category label data and statistical information of the ciphertext, the index information of the ciphertext of the classification prediction results is obtained, and the calculation participants do not need to obtain the plaintext of the classification prediction results, category label data and statistical information. Based on the classification prediction results, category label data and statistical information of the ciphertext, the ciphertext calculation can complete the calculation of the index information. During the calculation of the index information, the classification prediction results, category label data and statistical information of the ciphertext will not be decrypted, and these data will not be leaked. Under the premise of ensuring that the classification prediction results, classification label data and statistical information are not leaked, the evaluation index information of the model can be calculated to improve data security.

Description

Technical Field

[0001] The present application relates to artificial intelligence, machine learning, and big data technologies in computer technology, and in particular to a method, device, equipment, and storage medium for processing classification model prediction results. Background Art

[0002] Machine learning is widely used in many fields such as finance and medicine. The reason why machine learning can achieve good results in various fields is not only the leap forward of related technologies and the rapid improvement of hardware computing power, but also the explosive growth of data. However, the centralized collection of user data for model training also poses great privacy and security risks. In order to conduct joint analysis / modeling of multi-institutional data while meeting the requirements of user privacy protection, data security and relevant regulations, federated learning technology based on secure multi-party computing technology came into being.

[0003] In machine learning, it is usually necessary to evaluate the acc (accuracy) and f1 score of the trained model to evaluate the effect of model training. In the process of federated learning, model evaluation is also required, and the model prediction results and test set labels are also data that need to be protected in federated learning.

[0004] In the prior art, in the process of federated learning, the evaluation indicators are usually determined directly through plaintext calculations based on the plaintext of the model prediction results combined with the test set labels, which leaks the prediction results of the model on the test set. If the party calculating the model indicators is not the holder of the test set labels, the test set labels will also be leaked. Summary of the invention

[0005] The present application provides a method, apparatus, device and storage medium for processing prediction results of a classification model.

[0006] According to one aspect of the present application, a method for processing a classification model prediction result is provided, comprising:

[0007] Input the sample data into the target classification model to obtain the classification prediction results of the ciphertext;

[0008] Obtaining category label data corresponding to the sample data and statistical information of the category label data, wherein both the category label data and the statistical information are ciphertext;

[0009] Ciphertext calculation is performed based on the classification prediction result, category label data and statistical information to obtain index information of the ciphertext of the classification prediction result.

[0010] According to another aspect of the present application, a device for processing a classification model prediction result is provided, comprising:

[0011] The classification prediction module is used to input sample data into the target classification model to obtain the classification prediction results of the ciphertext;

[0012] A data acquisition module, used to acquire the category label data corresponding to the sample data, and the statistical information of the category label data, wherein the category label data and the statistical information are both ciphertext;

[0013] The indicator determination module is used to perform ciphertext calculation based on the classification prediction result, category label data and statistical information to obtain indicator information of the ciphertext of the classification prediction result.

[0014] According to another aspect of the present application, there is provided an electronic device, including:

[0015] at least one processor; and

[0016] a memory communicatively connected to the at least one processor; wherein,

[0017] The memory stores instructions that can be executed by the at least one processor. The instructions are executed by the at least one processor so that the at least one processor can perform the method described above.

[0018] According to another aspect of the present application, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute the above-described method.

[0019] According to another aspect of the present application, a computer program product is provided, comprising a computer program, wherein the computer program implements any of the above methods when executed by a processor.

[0020] The technology according to the present application improves the security of data during the calculation of model evaluation indicators.

[0021] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present application, nor is it intended to limit the scope of the present application. Other features of the present application will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present application.

[0023] Figure 1 is a schematic diagram of a system framework based on federated learning according to an embodiment of the present application;

[0024] Figure 2 is a flow chart of a method for processing classification model prediction results provided in the first embodiment of the present application;

[0025] Figure 3is a flow chart of a method for processing classification model prediction results provided in the second embodiment of the present application;

[0026] Figure 4 It is a schematic diagram of the overall process of calculating the indicator information provided in the second embodiment of the present application;

[0027] Figure 5 It is a schematic diagram of a processing device for classification model prediction results provided in the third embodiment of the present application;

[0028] Figure 6 It is a block diagram of an electronic device used to implement the method for processing the classification model prediction results of an embodiment of the present application. DETAILED DESCRIPTION

[0029] The following is a description of exemplary embodiments of the present application in conjunction with the accompanying drawings, including various details of the embodiments of the present application to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present application. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0030] As a general technology, machine learning has been widely used in various fields such as finance and medicine. The reason why machine learning can achieve good results in various fields is not only the leap forward of related technologies and the rapid improvement of hardware computing power, but also the explosive growth of data. However, the centralized collection of user data for model training also poses great privacy and security risks. In order to conduct joint analysis / modeling of multi-institutional data while meeting the requirements of user privacy protection, data security and government regulations, secure federated learning technology based on secure multi-party computing technology has emerged.

[0031] In the process of machine learning, it is usually necessary to evaluate the trained model, such as calculating the accuracy (acc) of the prediction results, f1 score (F1 score) and other indicators to evaluate the effect of training. The same is true in the process of federated learning. However, if the plain text of the model prediction results is restored, combined with the labels of the test set, and the above indicators are directly calculated through the plain text, the model's prediction results or the labels of the test set will be exposed. The model's prediction results and the labels of the test set are also parts that need to be protected in federated learning. How to protect the model's prediction results and the labels of the test set while evaluating the model training effect in the process of federated learning is an urgent problem to be solved.

[0032] The present application provides a method, device, equipment and storage medium for processing prediction results of a classification model, which are applied to artificial intelligence, machine learning, and big data technologies in the field of computer technology to achieve the purpose of calculating the evaluation indicators of the model while ensuring that intermediate prediction results, label data, etc. are not leaked, thereby ensuring data security.

[0033] The method for processing the prediction results of the classification model provided in the embodiment of the present application can be applied to a system framework based on federated learning, such as Figure 1 As shown, the system framework based on federated learning may include a label data owner 101, a computing participant 102, and a result processor 103. The label data owner 101 has sample data and category label data corresponding to the sample data for predicting the classification model. The label data owner 101 can analyze and count the sample data and category label data it owns, and obtain the statistical information of the category label data required for calculating the index information of the model; and provide the category label data and the statistical information of the category label data in the form of ciphertext to the computing participant 102, so as to avoid the leakage of the label data and ensure the security of the label data. The computing participant 102 calculates the classification prediction result of the ciphertext based on the sample data, and obtains the index information of the ciphertext corresponding to the classification prediction result based on the classification prediction result of the ciphertext and the category label data and its statistical information provided by the label data owner 101 through the ciphertext calculation. In this process, the leakage of the prediction result can be avoided, wherein the index information may be precision, recall, F1 score, accuracy (acc), etc. In order to avoid leakage of indicator information, the computing participant 102 sends the encrypted indicator information to the result processor 103, which decrypts the encrypted indicator information to obtain the plaintext indicator information. The result processor 103 has the right to obtain the plaintext indicator information, for example, it can be the demander who requires model prediction.

[0034] The processing method of the classification model prediction result provided in this embodiment can be used to calculate the precision, recall, F1 score, accuracy rate acc and other indicator information of the target classification model. Among them, the target classification model can be a binary classification model such as a neural network for solving binary classification problems, and can specifically be a face recognition model, text classification model, behavior recognition model, etc. for binary classification, which is not specifically limited here.

[0035] Figure 2 1 is a flow chart of a method for processing the prediction results of the classification model provided in the first embodiment of the present application. The execution subject of this embodiment is a computing participant, which is an electronic device. Figure 2 As shown, the specific steps of this method are as follows:

[0036] Step S201: Input sample data into the target classification model to obtain the classification prediction result of the ciphertext.

[0037] When the model needs to be evaluated, the computing participants can input the sample data into the target classification model, perform classification prediction on the sample data through the target classification model, and obtain the classification prediction results of the sample data; the classification prediction results are encrypted to obtain the ciphertext classification prediction results.

[0038] In this embodiment, the system framework based on federated learning may include multiple computing participants. In order to avoid leakage of intermediate data, each computing participant uses multi-party secure computing technology to jointly calculate and obtain the classification prediction results of the ciphertext of the sample data.

[0039] Step S202: Obtain category label data corresponding to the sample data, as well as statistical information of the category label data, where both the category label data and the statistical information are ciphertext.

[0040] In order to evaluate the classification model, it is necessary to calculate the classification prediction results of the participants based on the sample data, as well as the classification label data corresponding to the sample data and its statistical information, to calculate the indicator information of the classification prediction results.

[0041] In this embodiment, based on the system framework of federated learning, the label data owner owns the category label data corresponding to the sample data. In order to ensure the security of the category label data corresponding to the sample data and its statistical information, the label data owner provides the encrypted category label data and statistical information to the computing participants.

[0042] Step S203: perform ciphertext calculation based on the classification prediction result, the category label data and the statistical information to obtain the index information of the ciphertext of the classification prediction result.

[0043] The indicator information includes at least one of the following: precision, recall, F1 score, accuracy rate acc, etc.

[0044] After obtaining the classification prediction results, category label data and statistical information of the ciphertext, the ciphertext calculation is performed according to the type and calculation method of the indicator to be evaluated to obtain the indicator information of the ciphertext of the classification prediction results. During the calculation process, there is no need to decrypt the classification prediction results, category label data and statistical information of the ciphertext, and these data will not be leaked.

[0045] The embodiment of the present application obtains the classification prediction result of the ciphertext by inputting the sample data into the target classification model; obtains the category label data corresponding to the sample data, and the statistical information of the category label data, both of which are ciphertext; performs ciphertext calculation according to the classification prediction result, the category label data and the statistical information, and obtains the index information of the ciphertext of the classification prediction result. The calculation participants do not need to obtain the plaintext of the classification prediction result, the category label data, and the statistical information. The calculation of the index information can be completed by performing ciphertext calculation based on the classification prediction result, the category label data and the statistical information of the ciphertext. During the calculation of the index information, the classification prediction result, the category label data and the statistical information of the ciphertext will not be decrypted, and the data will not be leaked. The evaluation index information of the model can be calculated under the premise of ensuring that the classification prediction result, the classification label data and the statistical information are not leaked, thereby improving data security.

[0046] Figure 3 It is a flow chart of the processing method of the classification model prediction result provided by the second embodiment of the present application. On the basis of the above-mentioned first embodiment, in this embodiment, the ciphertext calculation is performed according to the classification prediction result, the category label data and the statistical information to obtain the index information of the ciphertext of the classification prediction result, which can be implemented in the following manner: according to the preset threshold, the classification prediction result is binarized to obtain the binarized prediction result; the binarized prediction result is stored in the form of a confusion circuit or a multi-party secure computing Boolean circuit to obtain the ciphertext binarization prediction result; the ciphertext calculation is performed according to the binarization prediction result of the ciphertext, the category label data and the statistical information to obtain the index information of the ciphertext.

[0047] like Figure 3 As shown, the specific steps of this method are as follows:

[0048] Step S301: Input sample data into the target classification model to obtain the classification prediction result of the ciphertext.

[0049] When the model needs to be evaluated, the computing participants can input the sample data into the target classification model, perform classification prediction on the sample data through the target classification model, and obtain the classification prediction results of the sample data; the classification prediction results are encrypted to obtain the ciphertext classification prediction results.

[0050] In this embodiment, the system framework based on federated learning may include multiple computing participants. In order to avoid leakage of intermediate data, each computing participant uses multi-party secure computing technology to jointly calculate and obtain the classification prediction results of the ciphertext of the sample data.

[0051] Exemplarily, multi-party secure computing technology can be used to distribute sample data to various computing participants. Each computing participant inputs the sample data into a target classification model based on the sample data allocated to it, and classifies and predicts the sample data through the target classification model to obtain a classification prediction result of the sample data. The classification prediction result is encrypted to obtain a classification prediction result of the ciphertext. The classification prediction results of the ciphertext of multiple computing participants can be summarized as needed to obtain a complete classification prediction result of the ciphertext.

[0052] Step S302: Obtain category label data corresponding to the sample data, as well as statistical information of the category label data, where both the category label data and the statistical information are ciphertext.

[0053] In this embodiment, based on the system framework of federated learning, the label data has the category label data corresponding to the sample data.

[0054] In order to evaluate the classification model, it is necessary to calculate the classification prediction results of the participants based on the sample data, as well as the classification label data corresponding to the sample data and its statistical information, to calculate the indicator information of the classification prediction results.

[0055] In order to ensure the security of the classification label data and its statistical information corresponding to the sample data, the label data owner obtains the statistical information of the classification label data required for calculating the indicator information based on the type of indicator information to be calculated, and stores the category label data and statistical information as ciphertext information in the form of a garbled circuit or a multi-party secure calculation Boolean circuit, and sends the ciphertext classification label data and statistical information to the calculation participants. The garbled circuit mentioned in this embodiment is also called Yao's GC or Yao's garbled circuit.

[0056] In this step, the computing participants receive the category label data and statistical information sent by the label data owner. The category label data and statistical information are stored as ciphertext information in the form of a confusion circuit or a multi-party secure computing Boolean circuit, which can avoid the leakage of the category label data and its statistical information corresponding to the sample data and ensure the security of the category label data and its statistical information corresponding to the sample data.

[0057] Exemplarily, multi-party secure computing technology may be used, whereby the label data owner distributes the classified label data and statistical information of the ciphertext to each computing participant.

[0058] Step S303: binarize the classification prediction result according to a preset threshold value to obtain a binarized prediction result.

[0059] The classification prediction result includes the ciphertext information of the predicted probability value of the sample in the sample data. For a binary classification model, the class label of the sample data is "1" or "0", and the class label "1" is also called the class label is true, and the class label "0" is also called the class label is false.

[0060] In this embodiment, the classification prediction result of the sample data is first binarized to "1" or "0" to obtain a binary prediction result. In the binary prediction result, the binary prediction result corresponding to each sample data is "1" or "0".

[0061] This step can be implemented in the following ways:

[0062] Based on the ciphertext information of the predicted probability value of the sample, compare the predicted probability value of the sample with the preset threshold; if the predicted probability value of the sample is greater than or equal to the preset threshold, the ciphertext information of the predicted probability value of the sample is binarized to 1; if the predicted probability value of the sample is less than the preset threshold, the ciphertext information of the predicted probability value of the sample is binarized to 0.

[0063] The preset threshold is in the range of (0, 1), and can be set and adjusted according to the actual application scenario, and is not specifically limited here.

[0064] Optionally, the ciphertext information of the predicted probability value of the sample is compared with a preset threshold value by calculating the difference between the two and judging the sign bit of the difference to determine whether the difference is positive, negative or zero.

[0065] In addition, the ciphertext information of the predicted probability value of the sample is compared with the preset threshold value, and any method of comparing the sizes of two numerical values ​​based on the ciphertext information of two numerical values ​​in the prior art can be used, which will not be repeated here.

[0066] In this way, by comparing the predicted probability value of the sample with the preset threshold, the binarization processing of the classification prediction result is achieved, providing a data basis for calculating various indicator information of the classification prediction result.

[0067] Step S304: store the binarization prediction result in the form of a confusion circuit or a multi-party secure computing Boolean circuit to obtain a ciphertext binarization prediction result.

[0068] In this embodiment, the binary prediction result is encrypted by storing the binary prediction result in the form of a confusion circuit or a multi-party secure computing Boolean circuit to obtain a ciphertext binary prediction result.

[0069] In addition, the storage method of the binarized prediction results in this step should be consistent with the storage method of the category label data and statistical information in the above steps, so as to facilitate the ciphertext calculation of the binarized prediction results and the category label data and statistical information. If the category label data and statistical information are stored in the form of a multi-party secure computing Boolean circuit, the binarized prediction results can be stored in the form of a multi-party secure computing Boolean circuit in this step. If the category label data and statistical information are stored in the form of a confusion circuit, the binarized prediction results can be stored in the form of a confusion circuit in this step.

[0070] Step S305: perform ciphertext calculation based on the ciphertext binarization prediction result, category label data and statistical information to obtain ciphertext indicator information.

[0071] In this embodiment, after obtaining the classification prediction results, category label data and statistical information of the ciphertext, the ciphertext calculation is performed according to the type and calculation method of the indicator to be evaluated, and the indicator information of the ciphertext of the classification prediction result is obtained. During the calculation process, there is no need to decrypt the classification prediction results, category label data and statistical information of the ciphertext, and these data will not be leaked.

[0072] In this embodiment, for the convenience of description, TP (true_positive) refers to the number of samples whose category labels are true and whose classification prediction results are also true; FP (false_positive) refers to the number of samples whose category labels are false and whose classification prediction results are true; FN (false_negative) refers to the number of samples whose category labels are true and whose classification prediction results are false; TN (true_negative) refers to the number of samples whose category labels are false and whose classification prediction results are false; precision refers to the precision of the classification prediction results; recall refers to the recall rate of the classification prediction results; and acc refers to the accuracy of the classification prediction results.

[0073] Then, the following formula is used to calculate the accuracy of the classification prediction result:

[0074]

[0075] The recall rate of the classification prediction result is calculated using the following formula 2:

[0076]

[0077] The F1 score of the classification prediction result is calculated using the following formula 3:

[0078]

[0079] The accuracy of the classification prediction result acc is calculated using the following formula 4:

[0080]

[0081] In an optional implementation, ciphertext calculations are performed based on the ciphertext binarization prediction results, category label data, and statistical information to obtain the precision and / or recall of the classification prediction results.

[0082] Specifically, a logical AND operation is performed on the ciphertext binarized prediction result and the category label data to obtain binarized data, the binarized data including binarized information corresponding to each sample in the sample data, wherein the binarized information of the sample whose prediction result and category label are both true is 1, and the binarized information of the sample whose prediction result and category label are at least one false is 0; the binarized data is stored in the form of a multi-party secure computing arithmetic circuit, and the binarized information of the ciphertext binarized data is accumulated to obtain first ciphertext information, the first ciphertext information is the ciphertext information of the number of samples whose prediction results and category labels are both true; based on the first ciphertext information, the precision and / or recall of the classification prediction result is determined.

[0083] In this way, the precision and / or recall of the classification prediction results can be calculated without leaking the classification prediction results, label data and their statistical information, thereby realizing the evaluation of the precision and / or recall of the target classification model.

[0084] Exemplarily, a logical AND operation of the ciphertext binarization prediction result and the category label data is implemented by performing an AND gate operation on the ciphertext binarization prediction result and the category label data.

[0085] If both the binary prediction result and the category label of the sample are true, then after performing a logical AND operation on the ciphertext binary prediction result and the category label data of the sample, the binary information of the sample is 1. If at least one of the binary prediction result and the category label of the sample is false, then after performing a logical AND operation on the ciphertext binary prediction result and the category label data of the sample, the binary information of the sample is 0. In this way, by performing a logical AND operation on the ciphertext binary prediction result and the category label data, samples whose binary prediction results and category labels are both true can be screened out, and the number of samples whose binary prediction results and category labels are both true can be further determined.

[0086] By storing the binarized data in the form of a multi-party secure computing arithmetic circuit and accumulating the binarized information of the ciphertext binarized data, the first ciphertext information of the number of samples whose binary prediction results and category labels are both true (i.e. TP) can be obtained. TP is the key data for calculating the precision and recall of classification prediction results.

[0087] Furthermore, the accuracy of the classification prediction result is determined according to the first ciphertext information, which can be specifically implemented in the following manner:

[0088] The ciphertext binarization prediction result is stored in the form of a multi-party secure computing arithmetic circuit, and the binarization information of the ciphertext binarization prediction result is accumulated to obtain the second ciphertext information, which is the ciphertext information of the number of samples whose prediction results are true; based on the first ciphertext information and the second ciphertext information, a ciphertext division calculation is performed to determine the accuracy of the classification prediction result.

[0089] Through this implementation, the accuracy of the classification prediction results can be calculated without leaking the classification prediction results, label data and their statistical information, thereby realizing the evaluation of the accuracy of the target classification model.

[0090] Specifically, by storing the ciphertext binary prediction result in a multi-party secure computing arithmetic circuit and accumulating the binary information of the ciphertext binary prediction result, the second ciphertext information of the number of samples (i.e., TP+FP) whose binary prediction result is true can be determined. Then, according to the above formula 1, by performing ciphertext division calculation on the first ciphertext information (i.e., the ciphertext of TP) and the second ciphertext information (i.e., the ciphertext of TP+FP), the accuracy of the classification prediction result of the ciphertext can be obtained.

[0091] Furthermore, the recall rate of the classification prediction result is determined according to the first ciphertext information, which can be specifically implemented in the following manner:

[0092] The third ciphertext information is obtained from the statistical information. The third ciphertext information is the ciphertext information of the number of samples with true category labels in the sample data. The third ciphertext information is stored in the form of a confusion circuit or a multi-party secure computing Boolean circuit. According to the first ciphertext information and the third ciphertext information, a ciphertext division calculation is performed to determine the recall rate of the classification prediction result.

[0093] Through this implementation, the recall rate of the classification prediction results can be calculated without leaking the classification prediction results, label data and their statistical information, thereby realizing the evaluation of the recall rate of the target classification model.

[0094] Specifically, according to the above formula 2, by performing ciphertext division calculation on the first ciphertext information (ie, the ciphertext of TP) and the third ciphertext information (ie, the ciphertext of TP+FN), the recall rate of the classification prediction result of the ciphertext can be obtained.

[0095] In another optional implementation, ciphertext calculation is performed based on the ciphertext binarization prediction result, category label data and statistical information to obtain the F1 score of the classification prediction result.

[0096] Exemplarily, firstly, ciphertext calculation is performed based on the ciphertext binary prediction result, the category label data and the statistical information to obtain the precision and / or recall rate of the classification prediction result; then, ciphertext calculation is performed based on the precision and recall rate of the classification prediction result to determine the F1 score of the classification prediction result. Through this implementation, the F1 score of the classification prediction result can be calculated without leaking the classification prediction result, label data and its statistical information, thereby realizing the evaluation of the target classification model.

[0097] Among them, the ciphertext calculation is performed based on the ciphertext binarization prediction result, the category label data and the statistical information, and the precision and / or recall rate of the classification prediction result can be calculated using the above implementation method.

[0098] According to the above formula 3, by performing ciphertext multiplication and ciphertext addition calculations on the precision and recall of the classification prediction results, respectively, we can obtain the precision×recall and precision+recall of the ciphertext, and then perform ciphertext division calculation on the precision×recall and precision+recall of the ciphertext, we can get the F1 score of the classification prediction result of the ciphertext.

[0099] In another optional implementation, the ciphertext is calculated based on the ciphertext binary prediction result, the category label data and the statistical information to obtain the accuracy of the classification prediction result. This can be implemented in the following way:

[0100] A logical exclusive OR operation is performed on the ciphertext binary prediction result and the category label data to obtain binary data, the binary data including the binary information corresponding to each sample in the sample data, wherein the binary information of the sample whose prediction result is consistent with the category label is 0, and the binary information of the sample whose prediction result is inconsistent with the category label is 1; the binary data is stored in the form of a multi-party secure computing arithmetic circuit, and the binary information of the ciphertext binary data is accumulated to obtain fourth ciphertext information, the fourth ciphertext information is the ciphertext information of the number of samples whose prediction results are inconsistent with the category labels; the fifth ciphertext information is obtained from the statistical information, the fifth ciphertext information is the ciphertext information of the total number of samples in the sample data, and the fifth ciphertext information is stored in the form of a confusion circuit or a multi-party secure computing Boolean circuit; according to the fourth ciphertext information and the fifth ciphertext information, a ciphertext calculation is performed to determine the accuracy of the classification prediction result.

[0101] Through this implementation, the accuracy acc of the classification prediction result can be calculated without leaking the classification prediction result, label data and its statistical information, thereby realizing the evaluation of the accuracy of the target classification model.

[0102] Specifically, if the binary prediction result and the category label of the sample are both true or both false, that is, the binary prediction result and the category label are consistent, then after performing a logical XOR operation on the ciphertext binary prediction result and the category label data of the sample, the binary information of the sample is 0. If only one of the binary prediction result and the category label of the sample is true, that is, the binary prediction result and the category label are inconsistent, then after performing a logical XOR operation on the ciphertext binary prediction result and the category label data of the sample, the binary information of the sample is 1. In this way, by performing a logical XOR operation on the ciphertext binary prediction result and the category label data, samples whose binary prediction results and category labels are inconsistent can be screened out, and the number of samples whose binary prediction results and category labels are inconsistent can be further determined.

[0103] Binarized data is obtained by performing a logical XOR operation on the ciphertext binary prediction result and the category label data, and stored in a multi-party secure computing arithmetic circuit. By accumulating the binary information therein, the fourth ciphertext information of the number of samples whose binary prediction results and category labels are inconsistent (that is, FP+FN) can be obtained. FP+FN is the key data for calculating the accuracy acc of the classification prediction result.

[0104] Furthermore, the fifth ciphertext information of the total number of samples (or the total number of labels) in the sample data (i.e., FP+FN+TP+TN of the ciphertext) can be obtained from the statistical information of the ciphertext, and the ciphertext TP+TN can be obtained by performing ciphertext subtraction calculation on the fifth ciphertext information and the fourth ciphertext information. Then, according to the above formula 4, the ciphertext TP+TN and the fifth ciphertext information are divided to obtain the accuracy acc of the classification prediction result.

[0105] Step S306: Send the ciphertext indicator information of the classification prediction result to the result processing party for decryption to obtain the plaintext indicator information of the classification prediction result.

[0106] In order to avoid leakage of indicator information, after the computing participants obtain the ciphertext indicator information of the classification prediction result, they can send the ciphertext indicator information to the result processing party, which will decrypt the ciphertext indicator information to obtain the plaintext indicator information.

[0107] The result processing party has the authority to obtain the plain text indicator information, for example, it can be the demander who requires model prediction.

[0108] In addition, in this embodiment, all processes involving multi-party interactions are implemented based on multi-party secure computing technology to avoid data leakage.

[0109] Combine the following Figure 4, taking the calculation of the F1 score of the classification prediction result as an example, the overall process of calculating the indicator information of the classification prediction result is explained:

[0110] For the classification prediction results of sample data, when evaluating the indicator information of the classification prediction results, the data that needs to be obtained are the classification prediction results of the ciphertext (computation participants), the preset threshold (computation participants), and the category label data and statistical information (label data owner). The classification prediction results of the ciphertext are compared with the preset threshold, the classification prediction results are binarized to obtain the binarized prediction results, and the binarized prediction results are stored in the form of multi-party secure computing Boolean circuits to obtain the ciphertext binarized prediction results.

[0111] For the category label data, the category label data is stored in the form of a multi-party secure computing Boolean circuit, and the category label data is encrypted through the multi-party secure computing Boolean circuit to obtain the ciphertext category label data. In addition, the plaintext category label data is statistically analyzed to obtain the number of samples with true category labels, that is, TP+FN; TP+FN is encrypted through a multi-party secure computing arithmetic circuit to obtain the ciphertext TP+FN.

[0112] The binary prediction result of the ciphertext is logically ANDed with the category label data of the ciphertext, and the result of the logical AND operation is circuit-converted and encrypted with a multi-party secure computing arithmetic circuit. The ciphertext is then accumulated to obtain the ciphertext information of the number of samples whose binary prediction results and category labels are both true, i.e., the TP of the ciphertext.

[0113] The binary prediction result of the ciphertext is converted into a circuit and stored in the form of a multi-party secure computing arithmetic circuit. Then, it is accumulated based on the obtained ciphertext to obtain the ciphertext information of the number of samples whose binary prediction result is true, that is, the TP+FP of the ciphertext.

[0114] The ciphertext TP and the ciphertext TP+FP are divided to obtain the precision of the ciphertext. The ciphertext TP and the ciphertext TP+FN are divided to obtain the recall of the ciphertext.

[0115] The precision and recall of the ciphertext are calculated by ciphertext multiplication and ciphertext addition respectively to obtain the precision×recall and precision+recall of the ciphertext, and then the precision×recall and precision+recall of the ciphertext are calculated by ciphertext division to obtain the F1 score of the classification prediction result of the ciphertext.

[0116] The result processor decrypts the F1 score of the ciphertext classification prediction result to obtain the F1 score of the plaintext classification prediction result.

[0117] The embodiment of the present application provides a ciphertext calculation method for indicator information such as precision, recall rate, F1 score and accuracy rate. The calculation of these indicator information can be completed by performing ciphertext calculation based on the classification prediction results, category label data and statistical information of the ciphertext. During the calculation of the indicator information, the classification prediction results, category label data and statistical information of the ciphertext will not be decrypted, and these data will not be leaked. The evaluation indicator information of the model can be calculated under the premise of ensuring that the classification prediction results, classification label data and statistical information are not leaked, thereby improving data security.

[0118] Figure 5 Schematic diagram of a processing device for classification model prediction results provided in the third embodiment of the present application. The processing device for classification model prediction results provided in the embodiment of the present application can execute the processing flow provided in the embodiment of the processing method for classification model prediction results. Figure 5 As shown, the processing device 50 for the classification model prediction result includes: a classification prediction module 501, a data acquisition module 502 and an indicator determination module 503.

[0119] Specifically, the classification prediction module 501 is used to input sample data into a target classification model to obtain a classification prediction result of the ciphertext.

[0120] The data acquisition module 502 is used to acquire the category label data corresponding to the sample data, and the statistical information of the category label data, where both the category label data and the statistical information are ciphertext.

[0121] The indicator determination module 503 is used to perform ciphertext calculation according to the classification prediction result, the category label data and the statistical information to obtain the indicator information of the ciphertext of the classification prediction result.

[0122] The device provided in the embodiment of the present application can be specifically used to execute the method embodiment provided in the above-mentioned first embodiment, and the specific functions will not be repeated here.

[0123] The embodiment of the present application obtains the classification prediction result of the ciphertext by inputting the sample data into the target classification model; obtains the category label data corresponding to the sample data, and the statistical information of the category label data, both of which are ciphertext; performs ciphertext calculation according to the classification prediction result, the category label data and the statistical information, and obtains the index information of the ciphertext of the classification prediction result. The calculation participants do not need to obtain the plaintext of the classification prediction result, the category label data, and the statistical information. The calculation of the index information can be completed by performing ciphertext calculation based on the classification prediction result, the category label data and the statistical information of the ciphertext. During the calculation of the index information, the classification prediction result, the category label data and the statistical information of the ciphertext will not be decrypted, and the data will not be leaked. The evaluation index information of the model can be calculated under the premise of ensuring that the classification prediction result, the classification label data and the statistical information are not leaked, thereby improving data security.

[0124] On the basis of the third embodiment described above, in a fourth embodiment of the present application, the classification prediction module is further used for:

[0125] The sample data is input into the target classification model, and the sample data is classified and predicted by the target classification model to obtain the classification prediction result of the sample data; the classification prediction result is encrypted to obtain the classification prediction result of the ciphertext.

[0126] In an optional implementation manner, the data acquisition module is further used to:

[0127] The category label data and statistical information sent by the label data owner are received, and the category label data and statistical information are stored as ciphertext information in the form of a confusion circuit or a multi-party secure computing Boolean circuit.

[0128] In an optional implementation manner, the indicator determination module is further used to:

[0129] According to the preset threshold, the classification prediction result is binarized to obtain a binarized prediction result; the binarized prediction result is stored in the form of a confusion circuit or a multi-party secure computing Boolean circuit to obtain a ciphertext binarized prediction result; the ciphertext is calculated based on the ciphertext binarized prediction result, category label data and statistical information to obtain the ciphertext indicator information.

[0130] In an optional implementation manner, the indicator determination module is further used to:

[0131] A logical exclusive OR operation is performed on the ciphertext binary prediction result and the category label data to obtain binary data, the binary data including the binary information corresponding to each sample in the sample data, wherein the binary information of the sample whose prediction result is consistent with the category label is 0, and the binary information of the sample whose prediction result is inconsistent with the category label is 1; the binary data is stored in the form of a multi-party secure computing arithmetic circuit, and the binary information of the ciphertext binary data is accumulated to obtain fourth ciphertext information, the fourth ciphertext information is the ciphertext information of the number of samples whose prediction results are inconsistent with the category labels; the fifth ciphertext information is obtained from the statistical information, the fifth ciphertext information is the ciphertext information of the total number of samples in the sample data, and the fifth ciphertext information is stored in the form of a confusion circuit or a multi-party secure computing Boolean circuit; according to the fourth ciphertext information and the fifth ciphertext information, a ciphertext calculation is performed to determine the accuracy of the classification prediction result.

[0132] In an optional implementation manner, the indicator determination module is further used to:

[0133] A logical AND operation is performed on the ciphertext binary prediction result and the category label data to obtain binary data, wherein the binary data includes the binary information corresponding to each sample in the sample data, wherein the binary information of the sample whose prediction result and the category label are both true is 1, and the binary information of the sample whose prediction result and the category label are at least one false is 0; the binary data is stored in the form of a multi-party secure computing arithmetic circuit, and the binary information of the ciphertext binary data is accumulated to obtain first ciphertext information, wherein the first ciphertext information is the ciphertext information of the number of samples whose prediction results and the category labels are both true; and the precision and / or recall rate of the classification prediction result is determined according to the first ciphertext information.

[0134] In an optional implementation manner, the indicator determination module is further used to:

[0135] The ciphertext binarization prediction result is stored in the form of a multi-party secure computing arithmetic circuit, and the binarization information of the ciphertext binarization prediction result is accumulated to obtain the second ciphertext information, which is the ciphertext information of the number of samples whose prediction results are true; based on the first ciphertext information and the second ciphertext information, a ciphertext division calculation is performed to determine the accuracy of the classification prediction result.

[0136] In an optional implementation manner, the indicator determination module is further used to:

[0137] The third ciphertext information is obtained from the statistical information. The third ciphertext information is the ciphertext information of the number of samples with true category labels in the sample data. The third ciphertext information is stored in the form of a confusion circuit or a multi-party secure computing Boolean circuit. According to the first ciphertext information and the third ciphertext information, a ciphertext division calculation is performed to determine the recall rate of the classification prediction result.

[0138] In an optional implementation manner, the indicator determination module is further used to:

[0139] According to the precision and recall of the classification prediction results, ciphertext calculation is performed to determine the F1 score of the classification prediction results.

[0140] In an optional implementation manner, the indicator determination module is further used to:

[0141] The classification prediction result includes the ciphertext information of the predicted probability value of the sample in the sample data; based on the ciphertext information of the predicted probability value of the sample, the predicted probability value of the sample is compared with the preset threshold; if the predicted probability value of the sample is greater than or equal to the preset threshold, the ciphertext information of the predicted probability value of the sample is binarized to 1; if the predicted probability value of the sample is less than the preset threshold, the ciphertext information of the predicted probability value of the sample is binarized to 0.

[0142] In an optional implementation manner, the indicator determination module is further used to:

[0143] The ciphertext is calculated based on the classification prediction results, category label data and statistical information to obtain the index information of the ciphertext of the classification prediction results. The index information of the ciphertext of the classification prediction results is sent to the result processing party for decryption to obtain the plaintext index information of the classification prediction results.

[0144] The device provided in the embodiment of the present application can be specifically used to execute the method embodiment provided in the above-mentioned second embodiment, and the specific functions will not be repeated here.

[0145] The embodiment of the present application provides a ciphertext calculation method for indicator information such as precision, recall rate, F1 score and accuracy rate. The calculation of these indicator information can be completed by performing ciphertext calculation based on the classification prediction results, category label data and statistical information of the ciphertext. During the calculation of the indicator information, the classification prediction results, category label data and statistical information of the ciphertext will not be decrypted, and these data will not be leaked. The evaluation indicator information of the model can be calculated under the premise of ensuring that the classification prediction results, classification label data and statistical information are not leaked, thereby improving data security.

[0146] According to an embodiment of the present application, the present application also provides an electronic device and a readable storage medium.

[0147] According to an embodiment of the present application, the present application also provides a computer program product, the program product includes: a computer program, the computer program is stored in a readable storage medium, at least one processor of an electronic device can read the computer program from the readable storage medium, and at least one processor executes the computer program so that the electronic device executes the solution provided by any of the above embodiments.

[0148] like Figure 6, is a block diagram of an electronic device according to a method for processing a classification model prediction result according to an embodiment of the present application. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples, and are not intended to limit the implementation of the present application described and / or required herein.

[0149] like Figure 6 As shown, the electronic device 600 includes: one or more processors 601, a memory 602, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. The various components are interconnected using different buses and can be installed on a common mainboard or installed in other ways as needed. The processor can process instructions executed in the electronic device, including instructions stored in or on the memory to display graphical information of a GUI on an external input / output device (such as a display device coupled to an interface). In other embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories. Similarly, multiple electronic devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 6 A processor 601 is taken as an example.

[0150] The memory 602 is a non-transitory computer-readable storage medium provided in the present application. The memory stores instructions executable by at least one processor to enable at least one processor to perform the method for processing the classification model prediction results provided in the present application. The non-transitory computer-readable storage medium of the present application stores computer instructions, which are used to enable a computer to perform the method for processing the classification model prediction results provided in the present application.

[0151] The memory 602 is a non-transient computer-readable storage medium that can be used to store non-transient software programs, non-transient computer executable programs and modules, such as the program instructions / modules corresponding to the method for processing the classification model prediction results in the embodiment of the present application (for example, the attached Figure 5 The processor 601 executes various functional applications and data processing of the server by running the non-transient software programs, instructions and modules stored in the memory 602, that is, the method for processing the classification model prediction results in the above method embodiment is implemented.

[0152] The memory 602 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and applications required for at least one function; the data storage area may store data created by the use of the electronic device 600 for processing the classification model prediction results, etc. In addition, the memory 602 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory 602 may optionally include a memory remotely arranged relative to the processor 601, and these remote memories may be connected to the electronic device 600 for processing the classification model prediction results via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0153] The electronic device 600 of the method for processing the classification model prediction result may also include: an input device 603 and an output device 604. The processor 601, the memory 602, the input device 603 and the output device 604 may be connected via a bus or other means. Figure 6 The example of connecting through bus is taken in the following.

[0154] The input device 603 can receive input digital or character information, and generate key signal input related to user settings and function control of the electronic device 600 for processing the classification model prediction results, such as a touch screen, a keypad, a mouse, a track pad, a touch pad, an indicator rod, one or more mouse buttons, a trackball, a joystick and other input devices. The output device 604 may include a display device, an auxiliary lighting device (e.g., an LED) and a tactile feedback device (e.g., a vibration motor), etc. The display device may include, but is not limited to, a liquid crystal display (LCD), a light emitting diode (LED) display and a plasma display. In some embodiments, the display device may be a touch screen.

[0155] Various implementations of the systems and techniques described herein can be realized in digital electronic circuit systems, integrated circuit systems, dedicated ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0156] These computer programs (also referred to as programs, software, software applications, or code) include machine instructions for programmable processors and can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, device, and / or means (e.g., disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.

[0157] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0158] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0159] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship between the client and the server is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services ("Virtual Private Server", or "VPS" for short). The server may also be a server of a distributed system, or a server combined with a blockchain.

[0160] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this application can be executed in parallel, sequentially or in different orders, as long as the expected results of the technical solution disclosed in this application can be achieved, and this document is not limited here.

[0161] The above specific implementations do not constitute a limitation on the protection scope of this application. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of this application should be included in the protection scope of this application.

Claims

1. A method for processing prediction results of a classification model, applied to a computing participant in a system architecture of federated learning, wherein the system architecture of federated learning includes a label data owner and a computing participant; the method comprises: Input the sample data into the target classification model to obtain the classification prediction results of the ciphertext; Obtaining category label data corresponding to the sample data and statistical information of the category label data from the label data owner, wherein both the category label data and the statistical information are ciphertext; According to a preset threshold, the classification prediction result is binarized to obtain a binarized prediction result; The binary prediction result is stored in the form of a confusion circuit or a multi-party secure computing Boolean circuit to obtain a ciphertext binary prediction result; The ciphertext is calculated based on the ciphertext binarization prediction result, the category label data and the statistical information to obtain the index information of the ciphertext of the classification prediction result.

2. The method according to claim 1, wherein: The step of inputting the sample data into the target classification model to obtain the classification prediction result of the ciphertext includes: Inputting the sample data into the target classification model, performing classification prediction on the sample data through the target classification model, and obtaining a classification prediction result of the sample data; The classification prediction result is encrypted to obtain the classification prediction result of the ciphertext.

3. The method according to claim 1, wherein: The obtaining of the category label data corresponding to the sample data from the label data owner, and statistical information of the category label data, wherein both the category label data and the statistical information are ciphertext, includes: The category label data and statistical information sent by the label data owner are received, and the category label data and statistical information are stored as ciphertext information in the form of a confusion circuit or a multi-party secure computing Boolean circuit.

4. The method according to claim 1, wherein: The ciphertext calculation is performed according to the ciphertext binarization prediction result, the category label data and the statistical information to obtain the index information of the ciphertext, including: Performing a logical XOR operation on the ciphertext binary prediction result and the category label data to obtain binary data, wherein the binary data includes binary information corresponding to each sample in the sample data, wherein the binary information of the sample whose prediction result is consistent with the category label is 0, and the binary information of the sample whose prediction result is inconsistent with the category label is 1; The binarized data is stored in a multi-party secure computing arithmetic circuit, and the binarized information of the ciphertext binarized data is accumulated to obtain fourth ciphertext information, where the fourth ciphertext information is ciphertext information of the number of samples whose prediction results are inconsistent with the category labels; Acquire fifth ciphertext information from the statistical information, where the fifth ciphertext information is ciphertext information of the total number of samples in the sample data, and the fifth ciphertext information is stored in the form of a confusion circuit or a multi-party secure computing Boolean circuit; Ciphertext calculation is performed based on the fourth ciphertext information and the fifth ciphertext information to determine the accuracy of the classification prediction result.

5. The method according to claim 1, wherein: The ciphertext calculation is performed according to the ciphertext binarization prediction result, the category label data and the statistical information to obtain the index information of the ciphertext, including: Performing a logical AND operation on the ciphertext binarization prediction result and the category label data to obtain binarized data, wherein the binarized data includes binarized information corresponding to each sample in the sample data, wherein the binarized information of a sample whose prediction result and category label are both true is 1, and the binarized information of a sample whose prediction result and category label are at least one false is 0; The binarized data is stored in a multi-party secure computing arithmetic circuit, and the binarized information of the ciphertext binarized data is accumulated to obtain first ciphertext information, where the first ciphertext information is ciphertext information of the number of samples in which both the prediction results and the category labels are true; The precision and / or recall of the classification prediction result is determined according to the first ciphertext information.

6. The method according to claim 5, wherein: Determining the accuracy of the classification prediction result according to the first ciphertext information includes: The ciphertext binary prediction result is stored in a multi-party secure computing arithmetic circuit, and the binary information of the ciphertext binary prediction result is accumulated to obtain second ciphertext information, where the second ciphertext information is the ciphertext information of the number of samples whose prediction results are true; A ciphertext division calculation is performed according to the first ciphertext information and the second ciphertext information to determine the accuracy of the classification prediction result.

7. The method according to claim 6, wherein: Determining the recall rate of the classification prediction result according to the first ciphertext information further includes: Acquire third ciphertext information from the statistical information, where the third ciphertext information is ciphertext information of the number of samples whose category labels are true in the sample data, and the third ciphertext information is stored in the form of a confusion circuit or a multi-party secure computing Boolean circuit; A ciphertext division calculation is performed according to the first ciphertext information and the third ciphertext information to determine the recall rate of the classification prediction result.

8. The method according to any one of claims 5 to 7, wherein: The ciphertext calculation is performed according to the classification prediction result, the category label data and the statistical information to obtain the index information of the ciphertext of the classification prediction result, and further includes: According to the precision and recall of the classification prediction result, ciphertext calculation is performed to determine the F1 score of the classification prediction result.

9. The method according to any one of claims 5 to 7, wherein: The binarization of the classification prediction result according to the preset threshold value to obtain the binarization prediction result includes: The classification prediction result includes ciphertext information of the prediction probability value of the sample in the sample data; Based on the ciphertext information of the predicted probability value of the sample, comparing the predicted probability value of the sample with the preset threshold; If the predicted probability value of the sample is greater than or equal to the preset threshold, the ciphertext information of the predicted probability value of the sample is binarized to 1; If the predicted probability value of the sample is less than the preset threshold, the ciphertext information of the predicted probability value of the sample is binarized to 0.

10. The method according to any one of claims 1 to 7, wherein: After performing ciphertext calculation according to the classification prediction result, the category label data and the statistical information to obtain the ciphertext indicator information of the classification prediction result, the method further includes: The ciphertext indicator information of the classification prediction result is sent to the result processing party for decryption to obtain the plaintext indicator information of the classification prediction result.

11. A device for processing prediction results of a classification model, applied to a computing participant in a system architecture of federated learning, wherein the system architecture of federated learning includes a label data owner and a computing participant; the device comprises: The classification prediction module is used to input sample data into the target classification model to obtain the classification prediction results of the ciphertext; A data acquisition module, used to acquire the category label data corresponding to the sample data from the label data owner, and the statistical information of the category label data, wherein the category label data and the statistical information are both ciphertext; An indicator determination module is used to binarize the classification prediction result according to a preset threshold value to obtain a binarized prediction result; The binary prediction result is stored in the form of a confusion circuit or a multi-party secure computing Boolean circuit to obtain a ciphertext binary prediction result; The ciphertext is calculated based on the ciphertext binarization prediction result, the category label data and the statistical information to obtain the index information of the ciphertext of the classification prediction result.

12. The device according to claim 11, wherein The classification prediction module is also used for: Inputting the sample data into the target classification model, performing classification prediction on the sample data through the target classification model, and obtaining a classification prediction result of the sample data; The classification prediction result is encrypted to obtain the classification prediction result of the ciphertext.

13. The device according to claim 11, wherein: The data acquisition module is also used for: The category label data and statistical information sent by the label data owner are received, and the category label data and statistical information are stored as ciphertext information in the form of a confusion circuit or a multi-party secure computing Boolean circuit.

14. The device according to claim 11, wherein: The indicator determination module is also used for: Performing a logical XOR operation on the ciphertext binary prediction result and the category label data to obtain binary data, wherein the binary data includes binary information corresponding to each sample in the sample data, wherein the binary information of the sample whose prediction result is consistent with the category label is 0, and the binary information of the sample whose prediction result is inconsistent with the category label is 1; The binarized data is stored in a multi-party secure computing arithmetic circuit, and the binarized information of the ciphertext binarized data is accumulated to obtain fourth ciphertext information, where the fourth ciphertext information is ciphertext information of the number of samples whose prediction results are inconsistent with the category labels; Acquire fifth ciphertext information from the statistical information, where the fifth ciphertext information is ciphertext information of the total number of samples in the sample data, and the fifth ciphertext information is stored in the form of a confusion circuit or a multi-party secure computing Boolean circuit; Ciphertext calculation is performed based on the fourth ciphertext information and the fifth ciphertext information to determine the accuracy of the classification prediction result.

15. The device according to claim 11, wherein The indicator determination module is also used for: Performing a logical AND operation on the ciphertext binarization prediction result and the category label data to obtain binarized data, wherein the binarized data includes binarized information corresponding to each sample in the sample data, wherein the binarized information of a sample whose prediction result and category label are both true is 1, and the binarized information of a sample whose prediction result and category label are at least one false is 0; The binarized data is stored in a multi-party secure computing arithmetic circuit, and the binarized information of the ciphertext binarized data is accumulated to obtain first ciphertext information, where the first ciphertext information is ciphertext information of the number of samples in which both the prediction results and the category labels are true; The precision and / or recall of the classification prediction result is determined according to the first ciphertext information.

16. The device according to claim 15, wherein: The indicator determination module is also used for: The ciphertext binary prediction result is stored in a multi-party secure computing arithmetic circuit, and the binary information of the ciphertext binary prediction result is accumulated to obtain second ciphertext information, where the second ciphertext information is the ciphertext information of the number of samples whose prediction results are true; A ciphertext division calculation is performed according to the first ciphertext information and the second ciphertext information to determine the accuracy of the classification prediction result.

17. The device according to claim 16, wherein: The indicator determination module is also used for: Acquire third ciphertext information from the statistical information, where the third ciphertext information is ciphertext information of the number of samples whose category labels are true in the sample data, and the third ciphertext information is stored in the form of a confusion circuit or a multi-party secure computing Boolean circuit; A ciphertext division calculation is performed according to the first ciphertext information and the third ciphertext information to determine the recall rate of the classification prediction result.

18. The device according to any one of claims 15 to 17, wherein: The indicator determination module is also used for: According to the precision and recall of the classification prediction result, ciphertext calculation is performed to determine the F1 score of the classification prediction result.

19. The device according to any one of claims 15 to 17, wherein: The indicator determination module is also used for: The classification prediction result includes ciphertext information of the prediction probability value of the sample in the sample data; Based on the ciphertext information of the predicted probability value of the sample, comparing the predicted probability value of the sample with the preset threshold; If the predicted probability value of the sample is greater than or equal to the preset threshold, the ciphertext information of the predicted probability value of the sample is binarized to 1; If the predicted probability value of the sample is less than the preset threshold, the ciphertext information of the predicted probability value of the sample is binarized to 0.

20. The device according to any one of claims 11 to 17, wherein: The indicator determination module is also used for: After performing ciphertext calculation based on the classification prediction result, category label data and statistical information to obtain the ciphertext indicator information of the classification prediction result, the ciphertext indicator information of the classification prediction result is sent to the result processing party for decryption to obtain the plaintext indicator information of the classification prediction result.

21. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 10.

22. A non-transitory computer-readable storage medium storing computer instructions, the computer instructions being used to cause the computer to execute the method of any one of claims 1-10.

23. A computer program product comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 10.

Images