Method for obtaining test samples for artificial intelligence models

Abstract

The application discloses a kind of acquisition methods of test sample applied to artificial intelligence model, comprising: receiving initial sample parameter, the initial sample parameter is the SHAP value of each feature of training sample;According to the SHAP value of the feature of each candidate sample locally and the initial sample parameter, determine the distance between each candidate sample and the training sample corresponding to the initial sample parameter;According to the distance corresponding to each candidate sample, determine the candidate sample closest to the training sample in each candidate sample as test sample according to distance.

Description

Technical Field

[0001] The present invention relates to the field of software technology, and in particular to a method for obtaining test samples applied to an artificial intelligence model. Background Art

[0002] The testing of artificial intelligence technology applications is inseparable from the four elements of data, models, computing power and scenarios.

[0003] Unlike traditional testing methods that focus more on the use of computing power and the testing of the AI ​​model itself, AI testing also needs to consider many issues such as the scenario coverage, feature fit, distribution rationality, channel legitimacy, and accessibility of test data.

[0004] In this exemplary technique, the terminal directly obtains a test sample locally from the training sample of the test model, and then tests the test sample using the test model. This testing method requires testing based on the test sample, which contains some private and other sensitive information, resulting in data insecurity and personal privacy leakage issues. Summary of the Invention

[0005] The main purpose of the present invention is to provide a method for obtaining test samples for artificial intelligence models, aiming to solve the problems of data insecurity and leakage of sensitive information such as personal privacy in the testing method.

[0006] To achieve the above objectives, the present invention provides a method for obtaining test samples applied to an artificial intelligence model, which is applied to a test participant. The method for obtaining test samples applied to an artificial intelligence model comprises the following steps:

[0007] Receiving initial sample parameters, wherein the initial sample parameters are SHAP values ​​of various features of the training sample;

[0008] Determine the distance between each candidate sample and the training sample corresponding to the initial sample parameter according to the SHAP value of the feature of each local candidate sample and the initial sample parameter;

[0009] According to the distance corresponding to each candidate sample, a candidate sample closest to the training sample is determined among the candidate samples as a test sample.

[0010] In one embodiment, after the step of determining, according to the distance corresponding to each candidate sample, a candidate sample closest to the training sample from among the candidate samples and using the candidate sample as the test sample, the method further includes:

[0011] Determining whether the test sample meets a set condition;

[0012] When the test sample does not meet the set conditions, the initial sample parameters are updated according to the SHAP values ​​of the various features of the test sample;

[0013] Return to the step of determining the distance between each candidate sample and the training sample corresponding to the initial sample parameter based on the SHAP value of the local feature of each candidate sample and the initial sample parameter.

[0014] In one embodiment, the step of updating the initial sample parameters according to the SHAP values ​​of the respective features of the test sample includes:

[0015] Determine the SHAP mean corresponding to each feature of the test sample according to the SHAP value of each feature of the test sample and the initial sample data;

[0016] Each of the SHAP means is determined as the updated initial sample parameter.

[0017] In one embodiment, the setting condition includes at least one of the following:

[0018] The total number of the test samples is greater than or equal to a first preset number;

[0019] The number of the remaining candidate samples is less than a second preset number;

[0020] The SHAP values ​​of the M features of the test sample are all less than the threshold value.

[0021] In one embodiment, after the step of determining the candidate sample closest to the training sample from among the candidate samples according to the distance corresponding to each candidate sample and using it as the test sample, the method further includes:

[0022] Determining a data quality indicator based on each of the test samples, and testing each of the test samples using a test model to obtain a test result;

[0023] The data quality indicator and the test result are sent to the test scheduling end.

[0024] In one embodiment, after the step of determining the candidate sample closest to the training sample from among the candidate samples according to the distance corresponding to each candidate sample and using it as the test sample, the method further includes:

[0025] Obtaining a first global data importance value, an allowable difference of the global data importance value, and a maximum difference in feature importance in each of the test samples sent by the test scheduling terminal, and determining a second global data importance value based on the SHAP value of each of the test samples;

[0026] When the difference between the first global data importance value and the second global data importance value and the maximum difference is less than the allowable difference, performing the step of determining the data quality indicator according to each of the test samples;

[0027] When the difference between the first global data importance value and the second global data importance value and the difference between the maximum difference is greater than or equal to the allowed difference, the default result is used as the test result, and the default value is used as the data quality indicator.

[0028] In another aspect, the present invention further provides a method for obtaining test samples applied to an artificial intelligence model, which is applied to a test scheduling end. The method for obtaining test samples applied to an artificial intelligence model includes:

[0029] Receive the test model and initial sample parameters sent by the model provider, wherein the initial sample parameters are the SHAP values ​​of each feature of the training sample;

[0030] Sending the test model and the initial sample parameters to each test participant, wherein the initial sample parameters are used by the test participant to determine the test sample closest to the training sample from each local candidate sample, and the test model is used by the test participant to perform testing based on the test sample;

[0031] Receive the test results and data quality indicators sent by each of the test participating terminals, and determine the global test result based on each of the test results and each of the data quality indicators.

[0032] In one embodiment, the step of sending the test model and the initial sample parameters to each test participant includes:

[0033] Acquiring parameters of each terminal, the parameters including data availability, data type, data quality, and data format in the terminal;

[0034] Determining, according to the parameters, a terminal that meets the test conditions among the terminals as a test participating terminal;

[0035] The test model and the initial sample parameters are sent to each test participant.

[0036] In one embodiment, the step of receiving the test model and initial sample parameters sent by the model provider includes:

[0037] Obtaining a test plan, and sending a request for obtaining a test model to the model provider according to the test plan;

[0038] Receive the test model and the initial sample parameters sent by the model provider.

[0039] In one embodiment, the step of sending the test model and the initial sample parameters to each test participant includes:

[0040] The receiving model provider sends a first global data importance value and an allowable difference between the global data importance values;

[0041] The test model, the initial sample parameters, the first global importance value, and the allowable difference are sent to each test participant.

[0042] On the other hand, the present invention also provides a test participating end, which includes a processor, a memory, and a computer program stored in the memory and runnable on the processor. When the computer program is executed by the processor, it implements the method for obtaining test samples applied to the artificial intelligence model as described above.

[0043] On the other hand, the present invention also provides a test scheduling end, which includes a processor, a memory, and a computer program stored in the memory and runnable on the processor. When the computer program is executed by the processor, it implements the method for obtaining test samples applied to the artificial intelligence model as described above.

[0044] On the other hand, the present invention also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the method for obtaining test samples applied to an artificial intelligence model as described above.

[0045] On the other hand, the present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements the method for obtaining test samples applied to an artificial intelligence model as described above.

[0046] The present invention provides a method for obtaining test samples for an artificial intelligence model. The method receives initial sample parameters, determines the distance between each candidate sample and the training sample corresponding to the initial sample parameters based on the SHAP values ​​of the features of each local candidate sample and the initial sample parameters, and then determines the candidate sample closest to the initial sample parameters among the candidate samples based on the distance corresponding to each candidate sample, as the test sample. In the present invention, the initial sample parameters are the SHAP values ​​of the features of the training sample, that is, the test sample is obtained based on the SHAP values ​​of the training sample, without the need to obtain the test sample from the training sample itself. The test sample does not need to be transmitted through communication during the artificial intelligence model testing process, thus avoiding the leakage of sensitive information such as personal privacy and ensuring data security. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 Schematic diagram of the hardware structure of the test participant terminal / test scheduling terminal involved in the embodiment of the present invention;

[0048] Figure 2 1. A flowchart of a first embodiment of a method for acquiring test samples for an artificial intelligence model according to the present invention;

[0049] Figure 3 2 is a flow chart of a second embodiment of a method for acquiring test samples for an artificial intelligence model according to the present invention;

[0050] Figure 4 1. A flowchart of a third embodiment of a method for acquiring test samples for an artificial intelligence model according to the present invention;

[0051] Figure 5 2 is a flow chart of a fourth embodiment of a method for acquiring test samples for an artificial intelligence model according to the present invention;

[0052] Figure 6 2 is a flow chart of a fifth embodiment of a method for acquiring test samples for an artificial intelligence model according to the present invention;

[0053] Figure 7 A brief interactive diagram of the method for obtaining test samples applied to artificial intelligence models in the present invention.

[0054] The purpose, features and advantages of the present invention will be further described with reference to the accompanying drawings and in conjunction with the embodiments. DETAILED DESCRIPTION

[0055] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0056] The main solution of the embodiment of the present invention is: receiving initial sample parameters, determining the distance between each candidate sample and the training sample corresponding to the initial sample parameters based on the SHAP values ​​of the features of each local candidate sample and the initial sample parameters, and then determining the candidate sample closest to the training sample among the candidate samples based on the distance corresponding to each candidate sample as the test sample.

[0057] In the present invention, the initial sample parameters are the SHAP values ​​of each feature of the training sample, that is, the test sample is obtained based on the SHAP value of the training sample. There is no need to obtain the test sample through the training sample itself. During the artificial intelligence model testing process, there is no need to communicate and transmit the test sample, which avoids the leakage of sensitive information such as personal privacy and ensures data security.

[0058] like Figure 1 As shown, Figure 1 It is a schematic diagram of the hardware structure of the test participating terminal / test scheduling terminal involved in the embodiment of the present invention.

[0059] like Figure 1 As shown, the embodiment of the present invention involves a test participant terminal / test scheduling terminal, which can be a terminal with data processing capabilities such as a computer. The test participant terminal / test scheduling terminal can include: a processor 101, such as a CPU, a communication bus 102, and a memory 103. Among them, the communication bus 102 is used to achieve connection and communication between these components. Those skilled in the art will understand that Figure 1 The structure shown in the figure does not constitute a limitation on the word search device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0060] like Figure 1 As shown, the memory 103 as a computer storage medium may include a computer program.

[0061] exist Figure 1 In the test participant shown, the processor 101 can be used to call the computer program stored in the memory 103 and perform the following operations:

[0062] Receiving initial sample parameters, wherein the initial sample parameters are SHAP values ​​of various features of the training sample;

[0063] Determine the distance between each candidate sample and the training sample corresponding to the initial sample parameter according to the SHAP value of the feature of each local candidate sample and the initial sample parameter;

[0064] According to the distance corresponding to each candidate sample, a candidate sample closest to the training sample is determined among the candidate samples as a test sample.

[0065] In one embodiment, the processor 101 in the test participant may call the computer program stored in the memory 103 and further perform the following operations:

[0066] Determining whether the test sample meets a set condition;

[0067] When the test sample does not meet the set conditions, the initial sample parameters are updated according to the SHAP values ​​of the various features of the test sample;

[0068] Return to the step of determining the distance between each candidate sample and the training sample corresponding to the initial sample parameter based on the SHAP value of the local feature of each candidate sample and the initial sample parameter.

[0069] In one embodiment, the processor 101 in the test participant may call the computer program stored in the memory 103 and further perform the following operations:

[0070] Determining a SHAP mean corresponding to each feature of the test sample according to the SHAP value of each feature of the test sample and the initial sample parameters;

[0071] Each of the SHAP means is determined as the updated initial sample parameter.

[0072] In one embodiment, the processor 101 in the test participant may call the computer program stored in the memory 103 and further perform the following operations:

[0073] The total number of the test samples is greater than or equal to a first preset number;

[0074] The number of the remaining candidate samples is less than a second preset number;

[0075] The SHAP values ​​of the M features of the test sample are all less than the threshold value.

[0076] In one embodiment, the processor 101 in the test participant may call the computer program stored in the memory 103 and further perform the following operations:

[0077] Determining a data quality indicator based on each of the test samples, and testing each of the test samples using a test model to obtain a test result;

[0078] The data quality indicator and the test result are sent to the test scheduling end.

[0079] In one embodiment, the processor 101 in the test participant may call the computer program stored in the memory 103 and further perform the following operations:

[0080] Obtaining a first global data importance value, an allowable difference of the global data importance value, and a maximum difference in feature importance in each of the test samples sent by the test scheduling terminal, and determining a second global data importance value based on a SHAP value of each of the test samples;

[0081] When the difference between the first global data importance value and the second global data importance value and the maximum difference is less than the allowable difference, performing the step of determining the data quality indicator according to each of the test samples;

[0082] When the difference between the first global data importance value and the second global data importance value and the difference between the maximum difference is greater than or equal to the allowed difference, the default result is used as the test result, and the default value is used as the data quality indicator.

[0083] In one embodiment, the processor 101 in the test scheduling terminal may call a computer program stored in the memory 103 and further perform the following operations:

[0084] Receive the test model and initial sample parameters sent by the model provider, wherein the initial sample parameters are the SHAP values ​​of each feature of the training sample;

[0085] The test model and the initial sample parameters are sent to each test participant, wherein the initial sample parameters are used by the test participant to determine the test sample closest to the training sample among various local candidate samples, and the test participant performs testing based on the test sample.

[0086] In one embodiment, the processor 101 in the test scheduling terminal may call a computer program stored in the memory 103 and further perform the following operations:

[0087] Acquiring parameters of each terminal, the parameters including data availability, data type, data quality, and data format in the terminal;

[0088] Determining, according to the parameters, a terminal that meets the test conditions among the terminals as a test participating terminal;

[0089] The test model and the initial sample parameters are sent to each test participant.

[0090] In one embodiment, the processor 101 in the test scheduling terminal may call a computer program stored in the memory 103 and further perform the following operations:

[0091] Obtaining a test plan, and sending a request for obtaining a test model to the model provider according to the test plan;

[0092] Receive the test model and the initial sample parameters sent by the model provider.

[0093] In one embodiment, the processor 101 in the test scheduling terminal may call a computer program stored in the memory 103 and further perform the following operations:

[0094] Receiving test results and data quality indicators sent by each of the test participating terminals;

[0095] A global test result is determined based on each of the test results and each of the data quality indicators.

[0096] In one embodiment, the processor 101 in the test scheduling terminal may call a computer program stored in the memory 103 and further perform the following operations:

[0097] The receiving model provider sends a first global data importance value and an allowable difference between the global data importance values;

[0098] The test model, the initial sample parameters, the first global importance value, and the allowable difference are sent to each test participant.

[0099] According to the above scheme, this embodiment receives initial sample parameters, determines the distance between each candidate sample and the training sample corresponding to the initial sample parameters based on the SHAP values ​​of the features of each local candidate sample and the initial sample parameters, and then determines the candidate sample closest to the training sample from each candidate sample based on the distance corresponding to each candidate sample, as the test sample. In the present invention, the initial sample parameters are the SHAP values ​​of each feature of the training sample, that is, the test sample is obtained based on the SHAP value of the training sample, without the need to obtain the test sample from the training sample itself. During the artificial intelligence model testing process, no communication transmission of the test sample is required, thus avoiding the leakage of sensitive information such as personal privacy and ensuring data security.

[0100] Based on the hardware architecture of the above-mentioned device, an embodiment of the method for obtaining test samples of the artificial intelligence model of the present invention is proposed.

[0101] Reference Figure 2 , Figure 2 This is the first embodiment of the method for obtaining test samples for an artificial intelligence model according to the present invention, which is applied to a test participant. The method for obtaining test samples for an artificial intelligence model includes the following steps:

[0102] Step S10: receiving initial sample parameters, wherein the initial sample parameters are SHAP values ​​of various features of the training sample.

[0103] In this embodiment, the execution subject is the test participant. The test participant can receive the initial sample parameters sent by the test scheduling end. The initial sample parameters are the SHAP values ​​of each feature of the training sample. SHAP refers to: the impact of the feature on the structure predicted based on the sample. The SHAP value of the feature can be understood as the contribution of the feature to the sample. The training sample is the training data in the application scenario of the artificial intelligence model (test model). The features of the training sample are, for example, table data for apartment price prediction, and the construction time, floor, orientation, and area of ​​the apartment data sample. If the training sample is image data, the super pixel formed by the labeled pixel group is also a feature. The initial sample parameters are sent by the model provider to the test scheduler. In addition, the initial sample parameters also include the training sample, that is, the initial sample parameters are the SHAP values ​​of the training sample and each feature of the training sample under the model under test. The model under test refers to the test model.

[0104] Step S20 : ​​determining the distance between each candidate sample and the training sample corresponding to the initial sample parameter based on the SHAP value of the local feature of each candidate sample and the initial sample parameter.

[0105] The test participant locally stores multiple sample data, each of which is defined as a candidate sample. The test participant needs to find the candidate sample that is closest to the training sample among the candidate samples as the test sample.

[0106] Specifically, the test participant can calculate the distance between each candidate sample and the training sample by the SHAP value of the feature of each candidate sample and the initial sample parameter (SHAP value of each feature of the training sample). The distance can be the Euclidean distance. That is, it is necessary to construct a distance space of the SHAP value based on the SHAP value of the features of each local candidate sample and the initial sample parameter. For example, if the candidate sample has three features, the distance space is a three-dimensional space, that is, the distance between the SHAP value of the candidate sample and the SHAP in the initial sample parameter is determined in the three-dimensional space.

[0107] For example, the initial sample data SAMPLE0 = (SHAP00, SHAP10, SHAP20 ... SHAPn0), then the difference between the candidate sample and the training sample is

[0108] Step S30 : determining, according to the distance corresponding to each candidate sample, a candidate sample that is closest to the training sample among the candidate samples as a test sample.

[0109] After determining the distances between each candidate sample and the training sample, the candidate sample closest to the training sample is selected as the test sample based on the distances. This means that the candidate sample closest to the initial sample parameters is selected from the candidate samples based on the distances and updated as the test sample. In other words, the test participant uses a clustering method to select the candidate sample closest to the initial sample parameters as the test sample.

[0110] Specifically, SAMPLE D It is the candidate sample that is closest to the training sample.

[0111] SAMPLE j =(SHAP0 j ,SHAP1 j ,SHAP2 j …SHAPn j )∈Data, Data is the candidate sample set, and D is the data set that meets the test quality requirements, D is also the test sample set, SAMPLE D ∈D.

[0112] In this embodiment, each test participant is a distributed node, meaning that the test participant is located in a distributed environment. In this distributed environment, the test participant does not need to transmit candidate samples. Instead, it only transmits the test results of each distributed node, thus implementing a global statistical test method and meeting the characteristic distribution requirements of the test samples.

[0113] In the technical solution provided in this embodiment, initial sample parameters are received, and based on the SHAP values ​​of the features of each local candidate sample and the initial sample parameters, the distance between each candidate sample and the training sample corresponding to the initial sample parameters is determined. Then, based on the distance corresponding to each candidate sample, the candidate sample closest to the training sample is determined from each candidate sample as the test sample. In the present invention, the initial sample parameters are the SHAP values ​​of each feature of the training sample, that is, the test sample is obtained based on the SHAP values ​​of the training sample, without the need to obtain the test sample from the training sample itself. The test sample does not need to be transmitted through communication during the artificial intelligence model testing process, thus avoiding sensitive information such as personal privacy and ensuring data security.

[0114] Reference Figure 3 , Figure 3 This is a second embodiment of the method for obtaining test samples for an artificial intelligence model according to the present invention. Based on the first embodiment, after step S30, the method further includes:

[0115] Step S40: Determine whether the test sample meets the set conditions.

[0116] In this embodiment, the test participant needs to obtain sufficient test samples. The test participant is provided with set conditions. The set conditions include: the total number of test samples is greater than or equal to a first preset number, the number of remaining candidate samples is less than a second preset number, and / or the SHAP values ​​of the M features of the test samples are all less than a threshold value.

[0117] Specifically, the model provider terminal includes multiple training samples and calculates the SHAP value of each feature of each training sample. The SHAP values ​​of features of the same type are arithmetic averaged to obtain the SHAP mean corresponding to each type of feature. The SHAP means are then sorted from largest to smallest. The model provider terminal selects the top M largest SHAP means and uses the minimum of these SHAP means as the threshold value. The model provider terminal sends the threshold value to the test scheduling terminal, which then sends the threshold value to the test participating terminal.

[0118] The test participant determines whether the test sample meets the set conditions. If the total number of test samples is less than the first preset number, the number of candidate samples remaining on the test participant is greater than or equal to the second preset number, or the SHAP of the M features of the test sample is greater than or equal to the threshold value, the set conditions are not met and the test participant needs to continue to obtain test samples from the remaining candidate samples.

[0119] Step S50 : When the test sample does not meet the set conditions, the initial sample parameters are updated according to the SHAP values ​​of the various features of the test sample.

[0120] If the test sample does not meet the set conditions, the initial sample parameters are updated based on the SHAP values ​​of the test sample, that is, new initial sample parameters are obtained. The test participant then returns to the step of determining the distance between each candidate sample and the training sample corresponding to the initial sample parameters based on the SHAP values ​​of the local candidate sample features and the initial sample parameters, that is, returns to steps S20 and S30 to reacquire the test sample.

[0121] Specifically, the test participant determines the SHAP mean corresponding to each feature of the test sample based on the SHAP value of each feature of the test sample and the initial sample parameters. Each SHAP mean can constitute the updated initial sample parameters.

[0122] For example, if there are two test samples, the initial sample parameters also need to be included in the calculation, that is, the SHAP mean of the test sample feature is the arithmetic mean of the SHAP of the two test sample features and the SHAP of the initial sample parameter (the features of the three SHAP values ​​are of the same type).

[0123] It should be noted that the second preset number is a smaller value, for example, 0. The M features of the test sample used for comparison with the threshold value are the same as the top M features ranked by the model provider. For example, if the top three features ranked by the model provider (M is 3) include apartment price, apartment size, and apartment orientation, then the test sample used for comparison with the threshold value includes apartment price, apartment size, and apartment orientation.

[0124] In the technical solution provided in this embodiment, the test participant determines whether the test sample meets the set conditions. If not, it is necessary to continue to obtain test samples, so as to make the test results more accurate.

[0125] Reference Figure 4 , Figure 4 This is a third embodiment of the method for obtaining test samples for an artificial intelligence model according to the present invention. Based on the first or second embodiment, after step S30, the method further includes:

[0126] Step S60: determining a data quality indicator according to each of the test samples, and testing the test model using each of the test samples to obtain a model test result.

[0127] In this embodiment, the model provider sends the top M largest SHAP means to the test scheduler, and the test scheduler then sends the top M largest SHAP means to each test participant.

[0128] The test participant compares the SHAP values ​​of the M features in the test sample with the mean of the M SHAP values ​​to obtain a quality value corresponding to the test sample. The quality values ​​of multiple test samples constitute a data quality index. The data quality index indicates the extent to which the test sample determined by the test participant meets the quality requirements of the test.

[0129] The test participant also stores the test tasks and test models assigned by the test scheduler. The test model is sent from the model provider to the test participant via the test scheduler. The test participant tests the test model using various test samples based on the test tasks to obtain the model test results.

[0130] Step S70: Send the data quality indicator and the model test result to the test scheduling end.

[0131] The test terminal sends the test results and data quality indicators to the test scheduler. The test scheduler determines the validity of the test results sent by each test participant based on the data quality indicators sent by the test participant, and then statistically analyzes the validity of the test results of each test terminal to form a global test result.

[0132] Furthermore, the model provider will calculate the SFI (SHAP Feature Importance) of the training sample, where SFI is defined as the first global data importance value. For example, is the global data importance of the jth feature, that is, the SHAP mean of the jth feature on the training dataset D provided by the model, and selects the most important M (the largest SHAP means before the sorting) feature values ​​M D ,in,

[0133] is the SHAP value of the j-th eigenvalue of the sample.

[0134] The model provider also sets the permissible difference of the global data importance value (data quality tolerance) ε. The model provider sends the first SFI and ε to each test participant through the test scheduling end.

[0135] After obtaining the test samples, the test participant determines the second global data importance value based on the SHAP value of each test sample:

[0136]

[0137] in, is the data importance of the j-th feature in the test sample.

[0138] The test participant calculates the difference between the first global data importance value and the second global data importance value. The test participant stores δ G , δ G is the maximum difference in feature importance among all test samples in the test participation segment. If ε>|δ G -δ L |, that is, when the difference between the first global data importance value and the second global data importance value and the maximum difference is less than the allowed difference, the step of determining the data quality index according to each test sample is performed, that is, step S60 is performed. When the difference between the first global data importance value and the second global data importance value and the maximum difference is greater than or equal to the allowed difference, the default result is used as the test result, and the default value is used as the data quality index, that is, the test result Eval L =0, data quality index |D L |=0.

[0139] The test scheduler obtains the global test results based on various data quality indicators and individual test results:

[0140]

[0141] Among them, Eval G is the global test result, For the test results, It is an indicator of data quality.

[0142] In the technical solution provided in this embodiment, the test participating end determines the data quality index based on each test sample, and uses the test model to test each test sample to obtain the test results, and then sends the data quality index and test results to the test scheduling end, so that the test scheduling end summarizes the test results of each test participant to form an accurate global test result.

[0143] Reference Figure 5 , Figure 5 The fourth embodiment of the method for obtaining test samples for an artificial intelligence model according to the present invention is applied to a test scheduler. The method for obtaining test samples for an artificial intelligence model includes:

[0144] Step S100: receiving a test model and initial sample data parameters sent by a model provider, wherein the initial sample parameters are SHAP values ​​of various features of the training sample.

[0145] In this embodiment, the execution subject is the test scheduling end. The receiving model end sends the test model and initial sample parameters to the test scheduling end. The initial sample parameters are the SHAP values ​​of each feature of the training sample in the model provider. SHAP refers to: the impact of the feature on the structure based on the sample prediction. The SHAP value of the feature can be understood as the contribution of the feature to the sample. The training sample is the training data in the application scenario of the artificial intelligence model (test model). The features of the training sample are, for example, table data for apartment price prediction, and the construction time, floor, orientation, and area of ​​the apartment data sample. If the training sample is image data, the super pixel formed by the labeled pixel group is also a feature. The test model is a model customized by the model provider based on the test task.

[0146] Alternatively, the test model can be trained based on a test plan provided by the test scheduler. Specifically, the test scheduler obtains the test plan and sends a request to the model provider based on the test plan to obtain the test model. The model provider then receives the test model and initial sample parameters from the model provider.

[0147] Specifically, the test scheduling end sends an acquisition request to the model provider based on the test objectives, test content, and various evaluation requirements of the evaluation plan. The acquisition request includes information such as model packaging requirements, interface packaging requirements, model training data, and explainable test requirements Re.

[0148] Step S110, sending the test model and the initial sample parameters to each test participating terminal, wherein the initial sample data parameters are used by the test participating terminal to determine the test sample closest to the training sample among the local candidate samples, and the test model is used by the test participating terminal to perform testing based on the test sample.

[0149] After receiving the test model and initial sample parameters, the test scheduler sends them to each participating test client. The initial sample parameters are used by each participating test client to determine the test sample closest to the training sample among its local candidate samples, while the test model is used by each participating test client to perform testing based on the test sample. The determination of the test sample by the participating test client and the application of the test sample to the test model are described above and will not be further elaborated here.

[0150] Furthermore, the test scheduling end has certain quality requirements for the test, so it can select from various terminals to determine the test participating terminals. The test scheduling end obtains the parameters of each terminal, and the parameters include data availability, data type, data quality and data format in the terminal. The test scheduling end determines the terminal that meets the test conditions as the test participating terminal according to the parameters, and then sends the test model and initial sample parameters to each test participating terminal. The test conditions include: the terminal's participation method is online, the data is available, all the data owned by the terminal meets the minimum number used in the test and / or the format and type of the data meet the needs of the test. The test scheduler decomposes the test task, and then sends the decomposed test task, initial sample parameters and test model to the test participants.

[0151] In the technical solution provided in this embodiment, the test scheduling end receives the test model and initial sample parameters sent by the model providing end, and then sends the initial sample parameters and the test model to the test participating end, so that the test participating end determines the test sample based on the initial sample parameters. The initial sample parameters are the SHAP values ​​of each feature of the training sample, that is, the test sample is obtained based on the SHAP value of the training sample. There is no need to obtain the test sample through the training sample itself, and there is no need to communicate and transmit the test sample during the artificial intelligence model testing process, which avoids the leakage of sensitive information such as personal privacy and ensures data security.

[0152] Reference Figure 6 , Figure 6 This is a fifth embodiment of the method for obtaining test samples for an artificial intelligence model according to the present invention. Based on the fourth embodiment, after step S110, the method further includes:

[0153] Step S120: receiving the test results and data quality indicators sent by each of the test participating terminals.

[0154] Step S130: determining a global test result based on each of the test results and each of the data quality indicators.

[0155] In this embodiment, the model provider sends the largest M SHAP means before sorting to the test scheduler, and the test scheduler then sends the largest M SHAP means before sorting to each test participant.

[0156] The test participant compares the SHAP values ​​of the M features in the test sample with the mean of the M SHAP values ​​to obtain a quality value corresponding to the test sample. The quality values ​​of multiple test samples constitute a data quality index. The data quality index indicates the extent to which the test sample determined by the test participant meets the quality requirements of the test.

[0157] The test participants also store the test tasks and test models assigned by the test scheduler. The test models are sent from the model provider to the test participants via the test scheduler. Based on the test tasks, the test participants use the test models to test each test sample and obtain test results. The test terminals send the test results and data quality indicators to the test scheduler. The test scheduler determines the validity of each test result based on the data quality indicators sent by each test participant. The scheduler then compiles statistics based on the validity of each test result to form a global test result.

[0158] Furthermore, the test scheduler receives the first global data importance data and the allowed difference of global data importance sent by the model provider, and then sends the test model, initial sample parameters, the first global importance value and the allowed difference to each test participant.

[0159] Specifically, the model provider will calculate the SFI (SHAP Feature Importance) of the training sample, where SFI is defined as the first global data importance value. For example, is the global data importance of the jth feature, that is, the SHAP mean of the jth feature on the training dataset D provided by the model, and selects the most important M (the largest SHAP means before the sorting) feature values ​​M D ,in,

[0160] is the SHAP value of the j-th eigenvalue of the sample.

[0161] The model provider also sets the permissible difference of the global data importance value (data quality tolerance) ε. The model provider sends the first SFI and ε to each test participant through the test scheduling end.

[0162] After obtaining the test samples, the test participant determines the second global data importance value based on the SHAP value of each test sample:

[0163]

[0164] in, is the data importance of the j-th feature in the test sample.

[0165] The test participant calculates the difference between the first global data importance value and the second global data importance value. The test participant stores δ G , δ G is the maximum difference in feature importance among all test samples in the test participation segment. If ε>|δ G -δ L |, that is, when the difference between the first global data importance value and the second global data importance value and the maximum difference is less than the allowed difference, the step of determining the data quality index and determining the model test result according to each test sample is executed, that is, step S60 is executed. When the difference between the first global data importance value and the second global data importance value and the maximum difference is greater than or equal to the allowed difference, the default result is used as the test result, and the default value is used as the data quality index, that is, the test result Eval L =0, data quality index |D L |=0.

[0166] The test scheduler obtains the global test results based on various data quality indicators and individual test results:

[0167]

[0168] Among them, Eval G is the global test result, For the test results, It is an indicator of data quality.

[0169] In the technical solution provided in this embodiment, the test scheduling end forms an accurate global test result based on the test results and data quality indicators sent by each test participating end.

[0170] Reference Figure 7 Based on the above embodiments, the process of the present invention is briefly described.

[0171] The model provider stores data, including training data. Each test participant also stores data, and the data is candidate samples.

[0172] 1. The test scheduling end sends a request to the model provider end, including model packaging requirements, interface packaging requirements, and training data quality index Re;

[0173] 2. The model provider sends the test model DM, initial sample parameters R and other information to the test scheduling end based on the request;

[0174] 3. The test scheduling end sends DM and R to test participant end 1, test participant end 2, ..., test participant end M.

[0175] The present invention also provides a test participant terminal, which includes a processor, a memory, and a computer program stored in the memory and runnable on the processor. When the computer program is executed by the processor, it implements the method for obtaining test samples applied to the artificial intelligence model as described above.

[0176] The present invention also provides a test scheduling end, which includes a processor, a memory, and a computer program stored in the memory and runnable on the processor. When the computer program is executed by the processor, it implements the method for obtaining test samples applied to the artificial intelligence model as described above.

[0177] The present invention also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the method for obtaining test samples applied to an artificial intelligence model as described above.

[0178] The present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements the method for obtaining test samples applied to an artificial intelligence model as described above.

[0179] The serial numbers of the above embodiments of the present invention are for description only and do not represent the advantages or disadvantages of the embodiments.

[0180] It should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or apparatus comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or apparatus comprising the element.

[0181] Through the description of the above embodiments, those skilled in the art can clearly understand that the above embodiment methods can be implemented by means of software plus the necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better embodiment. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes a number of instructions for enabling a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present invention.

[0182] The above are only preferred embodiments of the present invention and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention description and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A method for obtaining test samples for an artificial intelligence model, characterized in that: Applied to a test participant, the method for obtaining test samples applied to an artificial intelligence model includes the following steps: Receive initial sample parameters, where the initial sample parameters are SHAP values ​​of each feature of the training sample, wherein the SHAP value of the feature is the contribution of the feature to the sample; Determining the distance between each candidate sample and the training sample corresponding to the initial sample parameter according to the SHAP value of the feature of each local candidate sample and the initial sample parameter; According to the distance corresponding to each candidate sample, a candidate sample closest to the training sample is determined among the candidate samples to be used as a test sample. After the step of determining, according to the distance corresponding to each candidate sample, a candidate sample closest to the training sample is determined among the candidate samples to be used as a test sample, the method further includes: Determining a data quality indicator based on each of the test samples, and testing each of the test samples using a test model to obtain a test result; Based on the data quality indicators and the test results, the data is sent to the test scheduling end; After the step of determining, according to the distance corresponding to each candidate sample, a candidate sample closest to the training sample from among the candidate samples and using the candidate sample as a test sample, the method further includes: Determining whether the test sample meets a set condition; When the test sample does not meet the set conditions, the initial sample parameters are updated according to the SHAP values ​​of the various features of the test sample; Returning to the step of determining the distance between each candidate sample and the training sample corresponding to the initial sample parameters based on the SHAP values ​​of the features of each local candidate sample and the initial sample parameters; the step of updating the initial sample parameters based on the SHAP values ​​of each feature of the test sample includes: Determining a SHAP mean corresponding to each feature of the test sample according to the SHAP value of each feature of the test sample and the initial sample parameters; Each of the SHAP means is determined as an updated initial sample parameter; the setting conditions include at least one of the following: The total number of the test samples is greater than or equal to a first preset number; The number of the remaining candidate samples is less than a second preset number; The SHAP values ​​of the M features of the test sample are all less than the threshold value.

2. The method for obtaining test samples for an artificial intelligence model according to claim 1, characterized in that: After the step of determining, according to the distance corresponding to each candidate sample, a candidate sample closest to the training sample from among the candidate samples and using the candidate sample as a test sample, the method further includes: Obtaining a first global feature importance value, an allowable difference of the global feature importance value, and a maximum difference in feature importance in each of the test samples sent by the test scheduling terminal, and determining a second global feature importance value based on the SHAP value of each of the test samples; When the difference between the first global feature importance value and the second global feature importance value and the maximum difference is less than the allowable difference, performing the step of determining the data quality indicator according to each of the test samples; When the difference between the first global feature importance value and the second global feature importance value and the difference between the maximum difference is greater than or equal to the allowable difference, the default result is used as the test result, and the default value is used as the data quality indicator.

3. A method for obtaining test samples for an artificial intelligence model, characterized in that: Applied to the test scheduling end, the method for obtaining test samples applied to the artificial intelligence model includes: Receive the test model and initial sample parameters sent by the model provider, where the initial sample parameters are the SHAP values ​​of each feature of the training sample, where the SHAP value of the feature is the contribution of the feature to the sample; The test model and the initial sample parameters are sent to each test participating terminal, wherein the initial sample parameters are used by the participating terminal to determine, from local candidate samples, a candidate sample that is closest to the training sample as a test sample, and the test model is used by the test participating terminal to perform a test based on the test sample. After the participating terminal determines, from local candidate samples, a candidate sample that is closest to the training sample as a test sample, the method further includes: Determining whether the test sample meets a set condition; When the test sample does not meet the set conditions, the initial sample parameters are updated according to the SHAP values ​​of the various features of the test sample; Returning to executing the step of determining, from the local candidate samples, a candidate sample closest to the training sample as a test sample; The step of updating the initial sample parameters according to the SHAP values ​​of the various features of the test sample includes: Determining a SHAP mean corresponding to each feature of the test sample according to the SHAP value of each feature of the test sample and the initial sample parameters; Each of the SHAP means is determined as an updated initial sample parameter; the setting conditions include at least one of the following: The total number of the test samples is greater than or equal to a first preset number; The number of the remaining candidate samples is less than a second preset number; The SHAP values ​​of the M features of the test sample are all less than the threshold value; Receive the test results and data quality indicators sent by each of the test participating terminals, and determine the global test result based on each of the test results and each of the data quality indicators.

4. The method for obtaining test samples for an artificial intelligence model according to claim 3, characterized in that: The step of sending the test model and the initial sample parameters to each test participant includes: Acquiring parameters of each terminal, the parameters including data availability, data type, data quality, and data format in the terminal; Determining, according to the parameters, a terminal that meets the test conditions among the terminals as a test participating terminal; The test model and the initial sample parameters are sent to each test participant.

5. The method for obtaining test samples for an artificial intelligence model according to claim 4, characterized in that: The step of receiving the test model and initial sample parameters sent by the model provider includes: Obtaining a test plan, and sending a request for obtaining a test model to the model provider according to the test plan; Receive the test model and the initial sample parameters sent by the model provider.

6. The method for obtaining test samples for an artificial intelligence model according to claim 5, characterized in that: The step of sending the test model and the initial sample parameters to each test participant includes: receiving a first global feature importance value and an allowable difference between the global feature importance values ​​sent by the model provider; The test model, the initial sample parameters, the first global feature importance value and the allowable difference are sent to each test participant.

Images