Method for evaluating artificial intelligence model performance based on privacy calculation

Through privacy calculation methods, the performance evaluation and statistics of artificial intelligence models are solved, and the problem of how to evaluate and count model performance without disclosing performance parameters is realized, the performance ranking, mean and variance are calculated, and the privacy of model performance data is protected. .

CN118916649AActive Publication Date: 2024-11-08YUKUAI CHUANGLING INTELLIGENT TECH (NANJING) CO LTD
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
CN202411414535.3
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-10-11
Publication Date
2024-11-08
Estimated Expiration
2044-10-11

AI Technical Summary

Technical Problem

Without disclosing the specific performance parameters of each model, how to perform performance evaluation and statistics of the artificial intelligence model, especially calculate the mean and variance of the model to understand the performance distribution.

Method used

A privacy-based computing method is adopted to randomly generate a vendor queue, and asymmetric encryption algorithm is used to generate public and private keys, compare and sort model performance, calculate the mean and variance of the model, and ensure the privacy of performance data.

Benefits of technology

It realizes that without revealing any information, compare and count the performance of artificial intelligence models, protect the privacy of model performance data of each vendor, obtain performance rankings, mean values ​​and variances, and supports the evaluation and improvement of model performance.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118916649A_ABST
    Figure CN118916649A_ABST
Patent Text Reader

Abstract

The invention provides a privacy calculation-based artificial intelligence model performance evaluation method. The method comprises the following steps of S1, randomly generating a manufacturer queue; s2, comparing the performance of the two artificial intelligence models; s3, sorting the performances of the n artificial intelligence models; s4, calculating a mean value of the performance of the n artificial intelligence models; s5, calculating the variance of the performance of the n artificial intelligence models; according to the method for evaluating the performance of the artificial intelligence model based on privacy calculation, performance data of all the artificial intelligence models are compared pairwise through a privacy calculation method, and performance rankings of all the models are obtained through a sorting algorithm; the artificial intelligence model performance is compared and counted by using a privacy calculation method, evaluation and statistics of the artificial intelligence model performance are realized under the condition that manual work only does not disclose own performance data and does not leak any information, and the artificial intelligence model performance data privacy of each manufacturer is protected.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a method for evaluating the performance of artificial intelligence models based on privacy computing. Background Art

[0002] Privacy-preserving computing is an information technology that aims to analyze and compute data without disclosing the original data. Its core purpose is to ensure the security and privacy of data while allowing data users to perform necessary data analysis and processing. As users' demand for data privacy protection grows stronger, privacy-preserving computing technology is becoming increasingly important.

[0003] In recent years, with the emergence of models such as ChatGPT and Sora, artificial intelligence models have increasingly come into the public eye. An artificial intelligence model is a combination of mathematical and statistical methods used to solve specific artificial intelligence tasks. These models are usually trained on a large amount of data in order to discover patterns and regularities in the data, thereby making predictions or performing specific tasks. Artificial intelligence models can learn patterns and regularities from a large amount of data and make decisions or predictions based on the learned knowledge.

[0004] The capabilities of an AI model are typically validated by performing a specific task, such as classification, prediction, or generation. The model's ability to perform this task is called its performance. This performance is measured by testing the same set of data and evaluating how well it performs. Different AI models are tested on the same data, and the results are compared to the correct answers. If the results match the correct answers, the model is considered correct; otherwise, it is considered incorrect. This yields the model's precision or recall on that particular set of test data, which is its performance. Many similar datasets have been compiled in the industry, such as ImageNet and CoCo. Generally, a higher precision rate on a specific dataset is used to measure whether a new AI model has improved performance compared to previous models. The accuracy of an AI model in performing a task on a given dataset is a crucial attribute for evaluating its capabilities. To some extent, this accuracy is a key data metric for the AI ​​model. In the early stages of AI development, research institutions and companies typically release this data proactively to showcase model performance. However, in later, more competitive phases, more companies may treat the true performance of their models as company secrets, prohibiting their public disclosure. Yet, in model performance evaluations or bidding processes, organizers or clients need to know the capabilities of the participating models to provide rankings or evaluation results, and they also need to understand the mean and variance of all model performance to understand the performance distribution. How to conduct performance evaluations between models and calculate metrics such as the mean and variance of AI models without disclosing the specific performance parameters of each model is a pressing problem that needs to be solved. Summary of the Invention

[0005] The purpose of this invention is to address the shortcomings of existing technologies by proposing a method for evaluating the performance of artificial intelligence models based on privacy computing.

[0006] To achieve the above object, the present invention adopts the following technical solutions: A method for evaluating the performance of artificial intelligence models based on privacy-preserving computation includes the following steps: S1: Randomly generate a manufacturer queue; Suppose there are n AI models participating in the evaluation, and each AI model corresponds to a manufacturer. The manufacturers of the n models are randomly arranged into a queue M, M = {M1, M2, ..., Mn}. i M n}; S2: Compare the performance of two artificial intelligence models; Two manufacturers are randomly selected from queue M, denoted as Manufacturer A and Manufacturer B. The two artificial intelligence models held by Manufacturer A and Manufacturer B have been tested on the same data in the evaluation, and the performance data of their own artificial intelligence models have been obtained; let the performance data of Manufacturer A be i and the performance data of Manufacturer B be j. The values ​​i and j are positive integers in the range [0-1000]. The performance data of each manufacturer is confidential and not disclosed to the public. Includes the following sub-steps: S21: Generate public and private keys for the performance data of their respective artificial intelligence models through encryption algorithms. Company A obtains the ciphertext based on Company B's public key, converts the ciphertext into data w, and transmits it to Company B. Both manufacturers jointly determine an asymmetric encryption algorithm, and use the asymmetric encryption algorithm to generate public and private keys for the performance of their respective artificial intelligence models. The private key is kept by the manufacturer itself, while the public key is made public to the other party. Manufacturer A randomly selects data x and y within the range allowed by the formula, and obtains a positive integer z based on x and y. The formula is as follows: ; Manufacturer A encrypts data z using the encryption function Encrypt and Manufacturer B's public key to obtain ciphertext v; v = Encrpt(z). 乙 Where B refers to the public key of vendor B; The ciphertext v is converted into a positive integer. Company A adds the positive integer v to the performance data of the corresponding artificial intelligence model of Company A to obtain w=v+i, and then transmits the value of w to Company B. S22: Manufacturer B receives data w, constructs a sequence pair based on the received data, and decrypts the sequence. Manufacturer B receives data w, but cannot extract any valid information from it. The following steps are taken: Manufacturer B uses the received w to construct a sequence W containing 10,000 elements; W = {w-1, w-2, w-3, ..., wi, ..., w-10000}, where i is a positive integer; Manufacturer B uses the decryption function Decrypt and its private key to decrypt all elements in sequence W, obtaining Decrypt(W). 乙 sequence: Decrypt(W) 乙 = { Decrypt(w-1) 乙 Decrypt(w-2) 乙 Decrypt(w-3) 乙 ,……,Decrypt(wi) 乙 ,……,Decrypt(w-10000) 乙}; And because w = v + i; but ; Since z is obtained by manufacturer A by randomly selecting x and y according to the formula, manufacturer B is unaware of this and therefore cannot identify which element z is in the sequence.

[0007] S23: Take the modulo of the decrypted sequence to obtain the remainder sequence E; Manufacturer B randomly selects a prime number p and applies it to Decrypt(W). 乙 Modulo all elements in the sequence, that is, divide each element in the sequence by p. Since the divisor p is a prime number, the remainders of each element divided by p are all positive integers greater than 0, resulting in a remainder sequence E = {e1, e2, e3, ..., e}. i , ……, e 10000}; S24: Obtain sequence G; In the interval (1, ... Randomly select a positive integer q from the range; Given the model performance data of vendor B as j, perform the following operations on the j-th element and the elements preceding the j-th element in sequence E: If the value of an element plus q is greater than or equal to p, then the element is subtracted from the positive integer q; conversely, if the value of an element plus q is less than p, then the element is either plused or subtracted from the positive integer q. The sequence G is obtained as G = {e1±q, e2±q, e3±q, ……, e j ±q, e j+1 , e j+2 ..., e10000}; If i is less than or equal to j, the element e at the i-th position in sequence E i will become e i ±q. If i is greater than j, the element e at the i-th position in sequence E i remains unchanged; S25: Encrypt all elements in sequence G to obtain sequence H; The second manufacturer encrypts all elements in sequence G through the encryption function Encrypt and the public key of the first manufacturer to obtain sequence H = Encrpt(G) 甲 = { Encrpt (e1±q) 甲 , Encrpt (e2±q) 甲 , Encrpt (e3±q) 甲 [[ID=2」]],……, Encrpt (e j ±q)<000002[6]] 甲 , Encrpt (e j+1 ) 甲 , Encrpt (e j+2 ) 甲 …… Encrpt (e 10000 ) 甲}; Transmit sequence H and prime number p to the first manufacturer; S26: Complete the comparison of the performance of the artificial intelligence models of the first and second manufacturers; The first manufacturer receives sequence H and prime number p, decrypts the elements in sequence H through the decryption function Decrypt and the private key of the first manufacturer to obtain sequence G = { e1±q, e2±q, e3±q, ……, e j ±q, e j+1 , e j+2 ……,e 10000}; Take the modulus of the positive integer z with the prime number p to obtain the remainder s, compare the element at the i-th position in sequence G with the remainder s. If the two are equal, it means that the i-th element has not been added or subtracted by q, that is, the i-th position is after the j-th position, i>j, and the performance data of the first manufacturer's model is greater than that of the second manufacturer's model, and the performance of the first manufacturer is higher than that of the second manufacturer; If the two are not equal, it means that the i-th element has been added or subtracted by q by the second manufacturer, that is, the i-th position is before the j-th position, i<j, and the performance data of the first manufacturer's model is less than that of the second manufacturer's model, and the performance of the first manufacturer is lower than that of the second manufacturer;<0000¡35>Realize the comparison of the performance of the artificial intelligence models of the first and second manufacturers.

[0008] S3: Sort the performance of n artificial intelligence models; Includes the following sub-steps: S31: Compare the performance of the AI ​​models of two adjacent vendors in queue M in turn, find the model with the best performance in queue M, remove this model from queue M and enter queue C; The performance of two adjacent models in queue M is compared step by step using the method for comparing the performance of two artificial intelligence models in step S2. Specifically, the model performance of the first and second vendors in queue M is compared. If the model performance of the first vendor is less than that of the second vendor, their positions remain unchanged. If the model performance of the first vendor is greater than that of the second vendor, their positions in queue M are swapped. Next, compare the model performance of the second and third manufacturers in queue M. If the model performance of the second manufacturer is less than that of the third manufacturer, their positions remain unchanged; if the model performance of the second manufacturer is greater than that of the third manufacturer, their positions are swapped. The model performance of two adjacent vendors in queue M is compared sequentially using the method described above, until the model performance of the (n-1)th vendor and the nth vendor are compared. If the model performance of the (n-1)th vendor is greater than that of the nth vendor, then the model of the (n-1)th vendor is the best performing model in queue M; otherwise, the model of the nth vendor is the best performing model in queue M. The best performing model in queue M is then removed from queue M and placed into an empty queue C. At this time, the length of queue M is decreased by 1 and the length of queue C is increased by 1. S32: Complete the performance ranking of n artificial intelligence models; Repeat step S31 until all elements in queue M have been sequentially removed into queue C, i.e., the length of queue M is 0 and the length of queue C is n; thus, queue C = {C1, C2, C3, ..., Cn} is obtained. n}; At this point, the order of the elements in queue C is the order of the performance of the n manufacturers' artificial intelligence models from high to low, and the performance sorting of the n artificial intelligence models is completed. S4: Calculate the average performance of n artificial intelligence models; Let queue C = {C1, C2, C3, ... C} n Each manufacturer in the process multiplies its corresponding AI model performance data by 10000 and randomly divides it into n-1 parts. These n-1 parts of the data are then randomly sent to the other n-1 manufacturers. Each manufacturer will receive the segmented data from the other n-1 manufacturers. Since the data is segmented, each manufacturer cannot obtain any useful information from these n-1 parts of data. Each vendor adds up the n-1 data sets they receive and publishes the results, resulting in n publicly available data sets. These n publicly available data sets are then added together to obtain the sum of all model performance data. The sum of all model performance data is divided by n, and then divided by 10000 to obtain the average performance of these n AI models.

[0009] S5: Calculate the variance of the performance of n artificial intelligence models; Calculate the queue C = {C1, C2, C3, ..., C}. n The squared difference between the performance data of each manufacturer's AI model and the mean obtained in step S4 is calculated. Each manufacturer randomly divides the corresponding squared value into n-1 unequal parts and randomly transmits these n-1 parts of the data to the other n-1 manufacturers. Thus, each manufacturer will receive the segmented data from the other n-1 manufacturers. Since the data is segmented, each manufacturer cannot obtain any useful information from these n-1 parts of data. Each vendor adds up the n-1 data points they receive and publishes the result, resulting in n publicly available data points. The sum of these n publicly available data points is then obtained. The sum is divided by n, and then by 10000, to obtain the variance of the performance data of these n artificial intelligence models.

[0010] Compared with the prior art, the beneficial effects of the present invention are as follows: The present invention proposes a method for evaluating the performance of artificial intelligence models based on privacy computation. This method compares the performance data of all artificial intelligence models pairwise using privacy computation, and obtains the performance ranking of all models through a ranking algorithm. Then, it obtains the metric of all models through privacy computation and calculates the mean and variance of all models. Using privacy-preserving computation methods to compare and statistically analyze the performance of artificial intelligence models, this method enables the evaluation and statistical analysis of AI model performance without requiring the individual to disclose their own performance data or any other information, thus protecting the privacy of each vendor's AI model performance data. Attached Figure Description

[0011] Figure 1 This is a flowchart illustrating the steps of a method for evaluating the performance of an artificial intelligence model based on privacy computing, as described in this invention. Detailed Implementation

[0012] To provide a further understanding of the purpose, structure, features, and functions of the present invention, detailed descriptions are provided below with reference to specific embodiments.

[0013] like Figure 1 As shown, a method for evaluating the performance of artificial intelligence models based on privacy-preserving computation includes the following steps: S1: Randomly generate a manufacturer queue; Suppose there are n AI models participating in the evaluation, and each AI model corresponds to a manufacturer. The manufacturers of the n models are randomly arranged into a queue M, M = {M1, M2, ..., Mn}. i M n}; S2: Compare the performance of two artificial intelligence models; Two manufacturers are randomly selected from queue M, denoted as Manufacturer A and Manufacturer B. The two artificial intelligence models held by Manufacturer A and Manufacturer B have been tested on the same data in the evaluation, and the performance data of their own artificial intelligence models have been obtained; let the performance data of Manufacturer A be i and the performance data of Manufacturer B be j. The values ​​i and j are positive integers in the range [0-1000]. The performance data of each manufacturer is confidential and not disclosed to the public. Includes the following sub-steps: S21: Generate public and private keys for the performance data of their respective artificial intelligence models through encryption algorithms. Company A obtains the ciphertext based on Company B's public key, converts the ciphertext into data w, and transmits it to Company B. Both manufacturers jointly determine an asymmetric encryption algorithm, and use the asymmetric encryption algorithm to generate public and private keys for the performance of their respective artificial intelligence models. The private key is kept by the manufacturer itself, while the public key is made public to the other party. Manufacturer A randomly selects data x and y within the range allowed by the formula, and obtains a positive integer z based on x and y. The formula is as follows: ; Manufacturer A encrypts data z using the encryption function Encrypt and Manufacturer B's public key to obtain ciphertext v; v = Encrpt(z). 乙 Where B refers to the public key of vendor B; The ciphertext v is converted into a positive integer. Company A adds the positive integer v to the performance data of the corresponding artificial intelligence model of Company A to obtain w=v+i, and then transmits the value of w to Company B. S22: Manufacturer B receives data w, constructs a sequence pair based on the received data, and decrypts the sequence. Manufacturer B receives data w, but cannot extract any valid information from it. The following steps are taken: Manufacturer B uses the received w to construct a sequence W containing 10,000 elements; W = {w-1, w-2, w-3, ..., wi, ..., w-10000}, where i is a positive integer; Manufacturer B uses the decryption function Decrypt and its private key to decrypt all elements in sequence W, obtaining Decrypt(W). 乙 sequence: Decrypt(W) 乙 = { Decrypt(w-1) 乙 Decrypt(w-2) 乙 Decrypt(w-3) 乙 ,……,Decrypt(wi) 乙 ,……,Decrypt(w-10000) 乙}; And because w = v + i; but ; Since z is obtained by manufacturer A by randomly selecting x and y according to the formula, manufacturer B is unaware of this and therefore cannot identify which element z is in the sequence.

[0014] S23: Take the modulo of the decrypted sequence to obtain the remainder sequence E; Manufacturer B randomly selects a prime number p and applies it to Decrypt(W). 乙 Modulo all elements in the sequence, that is, divide each element in the sequence by p. Since the divisor p is a prime number, the remainders of each element divided by p are all positive integers greater than 0, resulting in a remainder sequence E = {e1, e2, e3, ..., e}. i , ……, e 10000}; S24: Obtain sequence G; In the interval (1, ... Randomly select a positive integer q from the range; Given the model performance data of vendor B as j, perform the following operations on the j-th element and the elements preceding the j-th element in sequence E: If the value of an element plus q is greater than or equal to p, then the element is subtracted from the positive integer q; conversely, if the value of an element plus q is less than p, then the element is either plused or subtracted from the positive integer q. The sequence G is obtained as G = {e1±q, e2±q, e3±q, ……, e j ±q, e j+1 , e j+2 ..., e 10000}; If i is less than or equal to j, then the element e at the i-th position in sequence E... i It will become e i ±q, if i is greater than j, then the i-th element e in sequence E i Remain unchanged; S25: Encrypt all elements in sequence G to obtain sequence H; Vendor B encrypts all elements in sequence G using encryption function Encrypt and the public key of Vendor A, obtaining sequence H = Encrpt(G). 甲 = { Encrpt(e1 ± q) 甲 , Encrpt(e2 ± q) 甲 , Encrpt(e3 ± q) 甲 , ……, Encrpt(e j ± q) 甲 , Encrpt(e j+1 ) 甲 , Encrpt(e j+2 ) 甲 …… Encrpt(e 10000 ) 甲};<00002​​​​​​​​​​​​​​​​​​​​​​​​​​​​​Specifically, the model performance of the first and second vendors in queue M is compared. If the model performance of the first vendor is less than that of the second vendor, their positions remain unchanged. If the model performance of the first vendor is greater than that of the second vendor, their positions in queue M are swapped. Next, compare the model performance of the second and third manufacturers in queue M. If the model performance of the second manufacturer is less than that of the third manufacturer, their positions remain unchanged; if the model performance of the second manufacturer is greater than that of the third manufacturer, their positions are swapped. The model performance of two adjacent vendors in queue M is compared sequentially using the method described above, until the model performance of the (n-1)th vendor and the nth vendor are compared. If the model performance of the (n-1)th vendor is greater than that of the nth vendor, then the model of the (n-1)th vendor is the best performing model in queue M; otherwise, the model of the nth vendor is the best performing model in queue M. The best performing model in queue M is then removed from queue M and placed into an empty queue C. At this time, the length of queue M is decreased by 1 and the length of queue C is increased by 1. S32: Complete the performance ranking of n artificial intelligence models; Repeat step S31 until all elements in queue M have been sequentially removed into queue C, i.e., the length of queue M is 0 and the length of queue C is n; thus, queue C = {C1, C2, C3, ..., Cn} is obtained. n}; At this point, the order of the elements in queue C is the order of the performance of the n manufacturers' artificial intelligence models from high to low, and the performance sorting of the n artificial intelligence models is completed. This invention uses an asymmetric encryption algorithm to generate two secret keys: a public key and a private key. The public key is made public, while the private key is kept secret. Data encrypted by the other party using the public key can be decrypted by the private key, ensuring that no information is leaked during the entire transmission process.

[0016] S4: Calculate the average performance of n artificial intelligence models; Let queue C = {C1, C2, C3, ... C} n Each manufacturer in the process multiplies its corresponding AI model performance data by 10000 and randomly divides it into n-1 parts. These n-1 parts of the data are then randomly sent to the other n-1 manufacturers. Each manufacturer will receive the segmented data from the other n-1 manufacturers. Since the data is segmented, each manufacturer cannot obtain any useful information from these n-1 parts of data. Each vendor adds up the n-1 data sets they receive and publishes the results, resulting in n publicly available data sets. These n publicly available data sets are then added together to obtain the sum of all model performance data. The sum of all model performance data is divided by n, and then divided by 10000 to obtain the average performance of these n AI models.

[0017] S5: Calculate the variance of the performance of n artificial intelligence models; Calculate the queue C = {C1, C2, C3, ..., C}. n The squared difference between the performance data of each manufacturer's AI model and the mean obtained in step S4 is calculated. Each manufacturer randomly divides the corresponding squared value into n-1 unequal parts and randomly transmits these n-1 parts of the data to the other n-1 manufacturers. Thus, each manufacturer will receive the segmented data from the other n-1 manufacturers. Since the data is segmented, each manufacturer cannot obtain any useful information from these n-1 parts of data. Each vendor adds up the n-1 data points they receive and publishes the result, resulting in n publicly available data points. The sum of these n publicly available data points is then obtained. The sum is divided by n, and then by 10000, to obtain the variance of the performance data of these n artificial intelligence models.

[0018] After ranking the performance of n artificial intelligence models, this invention calculates the mean and variance of their performance. The mean is used to understand the average performance level across all evaluated models; the variance measures the dispersion or fluctuation range of their performance. A smaller variance indicates more consistent model performance, while a larger variance indicates greater performance differences. These statistical data, such as mean and variance, can serve as performance benchmarks, providing a reference for future model improvements and helping to formulate improvement strategies and optimize the model development and evaluation process.

[0019] The present invention has been described in the above-described embodiments; however, these embodiments are merely examples for implementing the present invention. It must be noted that the disclosed embodiments do not limit the scope of the present invention. Conversely, any modifications and refinements made without departing from the spirit and scope of the present invention are within the scope of patent protection of the present invention.

Claims

1. A method for evaluating the performance of artificial intelligence models based on privacy computing, characterized in that: The following steps are involved: S1: Randomly generate manufacturer queues; Assume that there are n AI models participating in the evaluation, each AI model corresponds to a manufacturer, and the manufacturers of the n models are randomly arranged into a queue M, where M={M1, M2, ..., M i ,......,M n }; S2: Compare the performance of two AI models; Randomly select two manufacturers from the queue M, denoted as Manufacturer A and Manufacturer B. The two artificial intelligence models held by Manufacturer A and Manufacturer B have completed the test of the same data in the evaluation and obtained the performance data of their own artificial intelligence models; the performance data of Manufacturer A is denoted as i, and the performance data of Manufacturer B is denoted as j; The i and j are positive integers in the range [0-1000]. The performance data of each manufacturer is private and not disclosed to the public. The following sub-steps are included: S21: Generate public and private keys of the performance data of each artificial intelligence model through encryption algorithms. Manufacturer A obtains ciphertext based on the public key of Manufacturer B, converts the ciphertext into data w and transmits it to Manufacturer B; S22: Manufacturer B receives data w, constructs a sequence pair based on the received data, and decrypts the sequence; S23: Taking the modulus of the decrypted sequence to obtain a remainder sequence E; Manufacturer B randomly selects a prime number p and Decrypt(W) 乙 All elements in the sequence are modulo, that is, each element in the sequence is divided by p. Since the divisor p is a prime number, the remainder of the element divided by p is a positive integer greater than 0, and the remainder sequence E={e1, e2, e3,……, e i , ……, e 10000 }; S24: obtain sequence G; In the interval (1, ) randomly select a positive integer q; The model performance data of manufacturer B is j. The following operations are performed on the elements before and including the jth position in sequence E: If the value of the element after adding q is greater than or equal to p, then subtract the positive integer q from the element; conversely, if the value of the element after adding q is less than p, then add or subtract the positive integer q from the element; We get the sequence G={e1±q, e2±q, e3±q, …, e j ±q, e j+1 , e j+2 ……, e 10000 }; If i is less than or equal to j, the i-th element e in sequence E i It will become e i ±q, if i is greater than j, then the i-th element e in the sequence E i remain unchanged; S25: Encrypt all elements in the G sequence to obtain the H sequence; Manufacturer B encrypts all elements of the G sequence using the encryption function Encrypt and the public key of Manufacturer A, and obtains the sequence H=Encrpt(G) 甲 = { Encrpt (e1±q) 甲 , Encrpt (e2±q) 甲 , Encrpt (e3±q) 甲 , ……,Encrpt (e j ±q) 甲 , Encrpt (e j+1 ) 甲 , Encrpt (e j+2 ) 甲 …… Encrpt (e 10000 ) 甲 }; Transmit sequence H and prime number p to manufacturer A; S26: Complete the comparison of the performance of the artificial intelligence models of Manufacturer A and Manufacturer B; S3: Rank the performance of n artificial intelligence models; The following sub-steps are included: S31: Compare the performance of the artificial intelligence models of two adjacent manufacturers in queue M in turn, obtain the model with the best performance in queue M, remove this model from queue M and enter queue C; S32: complete the performance ranking of n artificial intelligence models; Step S31 is repeated until all elements in queue M are removed to queue C in sequence, that is, the length of queue M is 0 and the length of queue C is n; Get the queue C={C1, C2, C3, … C n }; At this point, the order of the elements in queue C is the order of the performance of the n vendors’ artificial intelligence models from high to low, and the performance ranking of the n artificial intelligence models is completed; S4: Calculate the mean of the performance of n artificial intelligence models; S5: Calculate the variance of the performance of n AI models.

2. The method for evaluating the performance of artificial intelligence models based on privacy computing as claimed in claim 1, characterized in that: The specific contents of step S21 are as follows: The two parties jointly determine an asymmetric encryption algorithm, and use the asymmetric encryption algorithm to generate public and private keys for the performance of their respective artificial intelligence models, where the private key is kept by the manufacturer itself and the public key is disclosed to the other party; Manufacturer A randomly selects data x and y within the range of values ​​allowed by the formula, and obtains a positive integer z based on x and y. The formula is as follows: ; Manufacturer A encrypts data z according to encryption function Encrypt and manufacturer B's public key to obtain ciphertext v; v=Encrpt(z) 乙 , where B refers to the public key of manufacturer B; Convert the ciphertext v into a positive integer. Manufacturer A adds the positive integer v to the performance data of the artificial intelligence model corresponding to Manufacturer A to obtain w=v+i, and passes the value of w to Manufacturer B.

3. The method for evaluating the performance of artificial intelligence models based on privacy computing as claimed in claim 1, characterized in that: The specific contents of step S22 are as follows: Manufacturer B receives data w, but cannot obtain valid information from it. It takes the following steps: Manufacturer B uses the received w to construct a sequence W containing 10,000 elements; W = {w-1, w-2, w-3, ……, w-i, ……, w-10000}, where i is a positive integer; Manufacturer B decrypts all elements in the W sequence using the decryption function Decrypt and Manufacturer B's private key, and obtains Decrypt(W) 乙 sequence: Decrypt(W) 乙 = { Decrypt(w-1) 乙 ,Decrypt(w-2) 乙 ,Decrypt(w-3) 乙 ,……,Decrypt(w-i) 乙 ,……,Decrypt(w-10000) 乙 }; Also, since w = v + i; but ; Since z is obtained by Party A's manufacturer randomly selecting x and y according to the formula and Party B's manufacturer is unaware of it, Party B's manufacturer cannot identify which element in the sequence z is.

4. The method for evaluating the performance of artificial intelligence models based on privacy computing as claimed in claim 1, characterized in that: The specific content of step S26 is as follows: Manufacturer A receives the sequence H and prime number p, and decrypts the elements in sequence H using the decryption function Decrypt and Manufacturer A's private key to obtain the sequence G={ e1±q, e2±q, e3±q, …, e j ±q, e j+1 , e j+2 ……, e 10000 }; Take the modulus of the positive integer z with the prime number p to obtain the remainder s. Compare the i-th element in sequence G with the remainder s. If the two are equal, it means that the i-th element has not been added or subtracted by q, that is, the i-th position is after the j-th position, i > j, and the performance data of Party A's manufacturer's model is greater than that of Party B's manufacturer's model, so the performance of Party A is higher than that of Party B; If the two are not equal, it means that the i-th element has been added or subtracted by q by Party B's manufacturer, that is, the i-th position is before the j-th position, i < j, and the performance data of Party A's manufacturer's model is less than that of Party B's manufacturer's model, so the performance of Party A is lower than that of Party B; Realize the comparison of the performance of the artificial intelligence models of Party A and Party B.

5. The method for evaluating the performance of artificial intelligence models based on privacy computing as claimed in claim 1, characterized in that: The specific content of step S31 is as follows: Compare the performance of adjacent two models in queue M step by step according to the method of comparing the performance of two artificial intelligence models in step S2; Specifically, compare the model performance of the first manufacturer and the second manufacturer in queue M. If the model performance of the first manufacturer is less than that of the second manufacturer, their positions remain unchanged; If the model performance of the first manufacturer is greater than that of the second manufacturer, exchange their positions in queue M; Then compare the model performance of the second manufacturer and the third manufacturer in queue M. If the model performance of the second manufacturer is less than that of the third manufacturer, their positions remain unchanged; If the model performance of the second manufacturer is greater than that of the third manufacturer, exchange their positions; Compare the model performance of adjacent two manufacturers in queue M in the above method in turn until comparing the model performance of the (n - 1)-th and the n-th manufacturers. If the model performance of the (n - 1)-th manufacturer is greater than that of the n-th manufacturer, the model of the (n - 1)-th manufacturer is the model with the best performance in queue M; otherwise, the model of the n-th manufacturer is the model with the best performance in queue M; Remove the model with the best performance in queue M from queue M and move it into the empty queue C. At this time, the length of queue M is reduced by 1 and the length of queue C is increased by 1.

6. The method for evaluating the performance of artificial intelligence models based on privacy computing as claimed in claim 1, characterized in that: In step S4, queue C={C1, C2, C3, ... C n Each manufacturer in the data multiplies its corresponding AI model performance data by 10,000 and randomly divides it into n-1 parts, and randomly transmits these n-1 parts of split data to other n-1 manufacturers. Then each manufacturer will get the split data from the other n-1 manufacturers, and because they only get the split data, each manufacturer cannot get any valid information from these n-1 parts of data. Each manufacturer adds the n - 1 pieces of data received and makes the result public, then n pieces of public data are obtained. Add the n pieces of public data to get the sum of all model performance data. Divide the sum of the performance data of all models by n, and then divide by 10000 to obtain the average value of the performance of these n artificial intelligence models.

7. The method for evaluating the performance of artificial intelligence models based on privacy computing as claimed in claim 1, characterized in that: In step S5, the queue C={C1, C2, C3, ... C n } and the square value of the difference between the artificial intelligence model performance data of each manufacturer in step S4, each manufacturer randomly divides the corresponding square value into n-1 parts, and randomly transmits the n-1 parts of the split data to the other n-1 manufacturers, then each manufacturer will obtain the split data from the other n-1 manufacturers, and because the obtained data is the split data, each manufacturer cannot obtain any valid information from the n-1 parts of the data; Each manufacturer adds the n - 1 pieces of data received and makes the result public, then n pieces of public data are obtained. Add the n pieces of public data to get the total sum. Divide the total sum by n, and then divide by 10000 to obtain the variance of the performance data of these n artificial intelligence models.

Citation Information

Patent Citations

  • Private data intersection calculation method and device, equipment and storage medium

    CN113343305A

  • Trusted private data comparison method, trusted private data storage device and intelligent terminal

    CN116132012A

  • Student information privacy protection method based on deep learning fusion

    CN116861485A

  • Cipher system by public key encryption method using a plurality of knapsacks, key generating device, encrypting device, decrypting device, data exchange method, and program

    JP2011128281A

  • KR20230036890A