A technical risk assessment method and system for artificial intelligence products

By establishing a hierarchical technical risk model of 'risk domain-risk subdomain-metric element', the problem of insufficient risk assessment of artificial intelligence products in the existing technology is solved, and risk assessment is realized throughout the life cycle is provided, and systematized risk is provided. Evaluation system.

CN114091644BActive Publication Date: 2024-12-06INST OF SOFTWARE - CHINESE ACAD OF SCI +1
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202010859193.1
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2020-08-24
Publication Date
2024-12-06
Estimated Expiration
2040-08-24

AI Technical Summary

Technical Problem

There is a lack of comprehensive risk assessment methods for artificial intelligence products in the existing technology, and it is impossible to effectively evaluate the technical risks of artificial intelligence products.

Method used

Establish a hierarchical technical risk model for artificial intelligence products, including data set risks, algorithm model risks and system operation risks. By determining risk subdomains and measurement elements, formulate specific products and to draw technical risk assessment conclusions based on pre-established evaluation criteria.

Benefits of technology

Considering the technical risks of artificial intelligence products from multiple perspectives, a risk assessment method for the whole life cycle has been established, which solves the problem of the lack of the technical risk assessment model of artificial intelligence products, and provides a systematic risk assessment system.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114091644B_ABST
    Figure CN114091644B_ABST
Patent Text Reader

Abstract

The present invention discloses a method and system for assessing the technical risks of artificial intelligence products, and establishes a hierarchical technical risk model of "risk domain-risk subdomain-metric element" for artificial intelligence products, wherein each risk domain has a corresponding risk subdomain, and each risk subdomain contains a corresponding metric element; the risk subdomain and metric element corresponding to the risk domain of the product are determined according to the characteristics of the product to be assessed and the assessment requirements, and a specific technical risk model is formulated; according to the technical risk model of the artificial intelligence product to be assessed, multiple metric element results required for technical risk assessment are collected; and multiple metric element results are evaluated according to the pre-established technical risk assessment criteria, and a technical risk assessment conclusion is obtained. The present invention considers the technical risks of artificial intelligence products from multiple perspectives, establishes an artificial intelligence technical risk assessment method involved in the development and application of artificial intelligence products throughout their life cycle, and solves the problem of the lack of a technical risk assessment model for artificial intelligence products.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of risk assessment technology, and in particular to a method and system for assessing technical risks of artificial intelligence products. Background Art

[0002] In recent years, artificial intelligence (AI) technology has seen its third wave of development. Powerful hardware foundation, massive data and advanced artificial intelligence algorithms have given artificial intelligence technology a strong impetus. Artificial intelligence products have made major breakthroughs in smart healthcare, smart transportation, smart justice, smart education and other fields.

[0003] However, while AI brings many benefits to human life, it also brings many risks. It is particularly important to ensure the smooth and safe implementation of AI technology, actively provide policy guidance and reasonable regulation of AI, and promote AI risk assessment research to achieve the healthy and sustainable development of AI products. However, there is no comprehensive assessment technology in the existing technology to conduct risk assessment on AI products. Summary of the invention

[0004] Therefore, the technical problem to be solved by the present invention is to overcome the deficiency of the lack of technical risk assessment method for artificial intelligence products in the prior art, thereby providing a technical risk assessment method and system for artificial intelligence products.

[0005] In order to achieve the above object, the present invention provides the following technical solutions:

[0006] In a first aspect, an embodiment of the present invention provides a method for assessing technical risks of an artificial intelligence product, comprising the following steps:

[0007] Establish a hierarchical technical risk model of "risk domain - risk subdomain - metric" for AI products, where each risk domain has a corresponding risk subdomain, and each risk subdomain contains a corresponding metric;

[0008] According to the characteristics of the AI ​​product to be evaluated and the evaluation requirements, determine the risk subdomains and metrics corresponding to the risk domain of the AI ​​product to be evaluated, and formulate a technical risk model for the AI ​​product to be evaluated;

[0009] According to the technical risk model of the AI ​​product to be evaluated, collect multiple metric results required for technical risk assessment;

[0010] The results of multiple metrics are evaluated according to the pre-established technical risk assessment criteria to draw technical risk assessment conclusions.

[0011] In one embodiment, the risk domain includes: data set risk, algorithm model risk, and system operation risk;

[0012] The risk subdomains of dataset risk include: unbiasedness, completeness, validity, confidentiality, accuracy, and dataset distribution balance;

[0013] The risk subdomains of algorithmic model risk include: robustness, model coverage, accuracy, transparency, algorithm dependency, explainability, and resistance to deception attacks;

[0014] The risk subdomains of system operation risk include: functional safety, predictability, reproducibility, fault tolerance, traceability, controllability and adaptability.

[0015] In one embodiment, the metrics of data set integrity include: data set model integrity, data set model attribute integrity, and metadata integrity;

[0016] The metrics of data set effectiveness include: data set effectiveness ratio, data element effectiveness;

[0017] Metrics for data set confidentiality include: use of encryption and non-vulnerability;

[0018] The metrics of dataset accuracy include: semantic accuracy, dataset inaccuracy risk, and metadata accuracy;

[0019] The metrics of data set distribution balance include: category balance and misclassification cost balance;

[0020] The metrics of algorithm model robustness include: output accuracy in abnormal samples, local adversarial robustness, and global adversarial robustness;

[0021] The metrics of algorithm model accuracy include: precision and accuracy;

[0022] The metrics of algorithm model transparency include: training process transparency and training result transparency;

[0023] The metrics of algorithm model algorithm dependency include: the correctness of the dependent algorithm, the compliance of the dependent algorithm;

[0024] The metrics of the interpretability of algorithmic models include: ex ante interpretability and ex post interpretability;

[0025] The metrics of the reproducibility of system operation include: the deviation of the reproduction results and the reproduction success rate;

[0026] The measurement elements of the fault tolerance of system operation include: avoidance of failure, resistance to misoperation, and harmfulness of misoperation;

[0027] The measurement elements of system operation traceability include: system execution result traceability and fault tracing and analysis capabilities;

[0028] The controllability metrics of the system operation include: control success rate in dangerous scenarios, control input compliance with distributed consistency;

[0029] The measurement elements of system operation adaptability include: cross-platform adaptability and task adaptability.

[0030] In one embodiment, according to the characteristics of the artificial intelligence product to be evaluated and the evaluation requirements, the risk subdomains and metrics corresponding to the risk domain of the artificial intelligence product to be evaluated are determined, and the process of formulating a technical risk model for the artificial intelligence product to be evaluated includes:

[0031] The risk domains in the AI ​​product technology risk assessment model include data set risk, algorithm model risk, and system operation risk, all of which are mandatory options;

[0032] According to the characteristics of the AI ​​product, determine the importance of each metric in the selected risk subdomain, and divide the metrics in the selected risk subdomain into important metrics and general metrics, where important metrics are mandatory and general metrics are optional;

[0033] In each risk subdomain, select optional items and at least one required item based on the assessment requirements to establish a technical risk assessment model for the specific AI product to be tested.

[0034] In one embodiment, the pre-established technical risk assessment passing criteria include:

[0035] Set the thresholds that need to be reached for each metric element based on the relevant historical similar product metric values ​​or the goals that need to be achieved;

[0036] If all the metrics selected under the risk subdomain meet the threshold requirements, then the risk subdomain meets the assessment requirements;

[0037] If all the selected risk subdomains under the risk domain meet the requirements, then the selected risk domain meets the assessment requirements;

[0038] If all risk domains are met, the technical risk meets the assessment requirements.

[0039] In one embodiment, collecting multiple metric results required for technical risk assessment includes: data collection and data confirmation; wherein,

[0040] During the data collection process, multiple measurement metadata required for technical risk assessment can be collected in categories, including data collected through tools, data collected from introductions and demonstrations, data obtained from interviews, and data obtained from reviews of relevant documents;

[0041] During the data confirmation process, at least one of the measurement metadata collected according to the technical risk assessment requirements is confirmed using preset verification rules, including: the data comes from at least two independent sources, the data is obtained during at least two different collection periods, and the data is confirmed by at least one data source that reflects that the preset rules have been followed.

[0042] In one embodiment, the process of evaluating multiple metric results according to pre-established technical risk assessment criteria to obtain a technical risk assessment conclusion includes:

[0043] By utilizing the collected multiple metric results required for technical risk assessment, the results are judged one by one against the pre-established technical risk assessment criteria, and ultimately the results of the technical risk assessment of the artificial intelligence product to be tested are obtained.

[0044] In a second aspect, an embodiment of the present invention provides a technical risk assessment system for an artificial intelligence product, including:

[0045] The technical risk model building module is used to build a hierarchical technical risk model of "risk domain - risk subdomain - metric element" for artificial intelligence products, where each risk domain has a corresponding risk subdomain, and each risk subdomain contains a corresponding metric element;

[0046] The module for establishing the technical risk model of the product to be evaluated determines the risk subdomains and metrics corresponding to the risk domain of the artificial intelligence product to be evaluated according to the characteristics of the artificial intelligence product to be evaluated and the evaluation requirements, and formulates a technical risk model for the artificial intelligence product to be evaluated;

[0047] A data collection module, used to collect multiple metric results required for technical risk assessment according to the technical risk model of the artificial intelligence product to be assessed;

[0048] The technical risk assessment conclusion acquisition module is used to evaluate multiple metric results according to pre-established technical risk assessment criteria to draw technical risk assessment conclusions.

[0049] In a third aspect, an embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable the computer to execute the technical risk assessment method for an artificial intelligence product of the first aspect of the embodiment of the present invention.

[0050] In a fourth aspect, an embodiment of the present invention provides a computer device, comprising: a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the technical risk assessment method for artificial intelligence products of the first aspect of the embodiment of the present invention by executing the computer instructions.

[0051] The technical solution of the present invention has the following advantages:

[0052] The technical risk assessment method and system for artificial intelligence products provided by the present invention establish a hierarchical technical risk model of "risk domain-risk subdomain-metric element" for artificial intelligence products, each risk domain has a corresponding risk subdomain, and each risk subdomain contains a corresponding metric element; determine the risk subdomain and metric element corresponding to the risk domain of the product according to the characteristics of the product to be evaluated and the evaluation requirements, and formulate a specific technical risk model; according to the technical risk model of the artificial intelligence product to be evaluated, collect multiple metric element results required for technical risk assessment; evaluate multiple metric element results according to the pre-established technical risk assessment criteria, and draw a technical risk assessment conclusion. The present invention considers the technical risks of artificial intelligence products from multiple perspectives, establishes an artificial intelligence technical risk assessment method involved in the development and application of artificial intelligence products throughout their life cycle, and solves the problem of the lack of a technical risk assessment model for artificial intelligence products. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0054] Figure 1 A workflow diagram of a specific example of a method for assessing technical risks of an artificial intelligence product in an embodiment of the present invention;

[0055] Figure 2 It is a framework diagram of a technical risk assessment model in an embodiment of the present invention;

[0056] Figure 3 This is a module composition diagram of a specific example of a technical risk assessment system for an artificial intelligence product in an embodiment of the present invention;

[0057] Figure 4 A composition diagram of a specific example of a computer device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0058] The technical solution of the present invention will be described clearly and completely below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0059] In addition, the technical features involved in the different embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0060] Example 1

[0061] The embodiment of the present invention provides a method for assessing the technical risks of an artificial intelligence product. Figure 1 As shown, the specific steps include:

[0062] Step S10: Establish a hierarchical technical risk model of "risk domain-risk subdomain-metric element" for artificial intelligence products, where each risk domain has a corresponding risk subdomain, and each risk subdomain contains a corresponding metric element.

[0063] In the embodiment of the present invention, Figure 2 As shown in Figure 1, the risk domains include: data set risk, algorithm model risk, and system operation risk.

[0064] The risk subdomains of dataset risk include: unbiasedness, completeness, validity, confidentiality, accuracy, and dataset distribution balance;

[0065] The risk subdomains of algorithmic model risk include: robustness, model coverage, accuracy, transparency, algorithm dependency, explainability, and resistance to deception attacks;

[0066] The risk subdomains of system operation risk include: functional safety, predictability, reproducibility, fault tolerance, traceability, controllability and adaptability.

[0067] The types and corresponding metrics included in each risk subdomain are either qualitative or quantitative, as follows:

[0068] 1. The unbiasedness of a data set means that the mathematical expectation of the sample statistic is equal to the value of the estimated population parameter. When the actual value of the population parameter is equal to its estimated value, the estimate is unbiased; if an infinite number of samples about Y are drawn from the population and an estimate is calculated each time, then the average of these estimates of all random samples (mathematical expectation) is equal to the true value of the unknown parameter, which is the requirement of unbiasedness.

[0069] Suppose A'=g(X1,X2,...,Xn) is a point estimator of the unknown parameter A. If A' satisfies E(A')=A, then A' is called an unbiased estimator of A, otherwise it is a biased estimator.

[0070] Where X1, X2, ..., Xn in A' = g(X1, X2, ..., Xn) are n random samples taken from the sample. Each sample can calculate a point estimate of a certain parameter. After n calculations, n point estimates are obtained, and then the expectation of the n point estimates is calculated.

[0071] 2. Dataset integrity refers to the extent to which a subject dataset associated with an entity in a given environment has values ​​for all expected attributes and associated strength values; it includes: dataset model integrity, dataset model attribute integrity, and metadata integrity, where:

[0072] 1) Dataset model integrity refers to the completeness of the dataset model relative to the entities described in the context model. The formula is as follows:

[0073]

[0074] Among them, A is the number of entities in the dataset model that fully describes the context pattern, and B is the number of entities in the dataset model;

[0075] 2) Dataset model attribute integrity refers to the attribute integrity defined for the dataset model. The formula is as follows:

[0076]

[0077] Among them, B is the number of attributes defined in the dataset model, and A is the number of attributes defined in the dataset model that completely describes the context model.

[0078] 3) Metadata integrity refers to the integrity of metadata attributes, and the formula is as follows:

[0079]

[0080] Among them, A is the number of attributes with complete metadata in the dataset, and B is the number of attributes expected to have metadata in the dataset.

[0081] 3. Dataset validity refers to the degree to which a data set has properties that characterize its ability to be retrieved by authorized users or applications in a specific environment of use, including: data set validity ratio, data element validity, where:

[0082] 1) The dataset validity ratio refers to the probability of a successful request using a data item during the request duration, and the formula is as follows:

[0083]

[0084] Where A is the number of times the data item is valid during the request duration, and B is the number of times the data item is requested during the request duration.

[0085] 2) The validity of elements in a data structure refers to the degree to which the elements in the data structure are valid. The formula is as follows:

[0086]

[0087] Where A is the number of elements in the data structure that are valid for the intended user, and B is the number of elements in the data structure.

[0088] 4. Dataset confidentiality refers to the extent to which the dataset itself can only be accessed and interpreted by authorized users in a given environment; it includes: the use of encryption and non-vulnerability, where:

[0089] 1) Encryption usage refers to the degree to which a data set meets encryption requirements. The formula is as follows:

[0090]

[0091] Among them, A is the number of data correctly and successfully encrypted and decrypted in the data set, and B is the number of data with encryption and decryption requirements in the data set.

[0092] 2) Non-vulnerability refers to the degree to which a confidential data set can only be accessed by authorized users. The formula is as follows:

[0093]

[0094] Among them, A is the number of times that unauthorized users successfully complete access during formal intrusion attempts to obtain the target data set within a specific time period, and B is the number of times unauthorized users attempt to access the target data set within a specific time period.

[0095] 5. Dataset accuracy refers to the degree to which data has the property of correctly representing the true value of the relevant attributes of a concept or event in a given environment, including: semantic accuracy, data set inaccuracy risk, metadata accuracy, among which:

[0096] 1) Semantic accuracy refers to the ratio of the data meaning to the semantics in a specific context, and the formula is as follows:

[0097]

[0098] Among them, A is the number of semantically accurate data values, and B is the number of data values ​​that require semantic accuracy;

[0099] 2) Data set inaccuracy risk refers to the risk that the number of outliers in the value indicates the inaccuracy of the data value in the data set. The formula is as follows:

[0100]

[0101] Among them, A is the number of data values ​​of outliers, and B is the number of data values ​​that need to be considered in the data set;

[0102] 3) Metadata accuracy refers to whether the metadata describes the data with the required accuracy. The formula is as follows:

[0103]

[0104] Among them, A is the number of metadata that provides appropriate requirement information, and B is the number of metadata defined in the data requirement description.

[0105] 6. Data set distribution balance refers to the balance of data distribution in a data set. The measurement elements of data set distribution balance include:

[0106] 1) Category balance, which indicates the degree of balance in the number of different categories. The formula is as follows:

[0107]

[0108] Among them, A i is the number of the i-th category, and n is the total number of categories in the data set.

[0109] 2) Misclassification cost balance, which indicates the balance degree of classification error cost of each category in the data set. The formula is as follows:

[0110]

[0111] Where N is the total number of categories, and Cost[i,j] is the cost of misclassifying a data of category i into category j.

[0112] 7. The robustness of the algorithm model refers to the degree to which the artificial intelligence algorithm can work correctly under invalid input or stressful environment conditions. The measurement elements of the robustness of the algorithm model include: output accuracy in abnormal samples, local adversarial robustness, and global adversarial robustness;

[0113] 1) The output accuracy measurement formula in robust non-normal samples is as follows:

[0114] X=A / B

[0115] Among them, A is the number of samples in the abnormal data samples that can maintain their normal result output, and B is the number of abnormal data samples in the benchmark data.

[0116] 2) The global adversarial robustness measurement formula is as follows:

[0117] r=E(S)-E(δ(S))

[0118] Here, let S be an AI system, E(S) be the correctness of S, and δ(S) be the machine learning system subject to perturbations of any machine learning component (e.g., data, learning procedure, or framework). Robustness is a measure of the difference between E(S) and E(δ(S)):

[0119] 3) The local adversarial robustness formula is as follows:

[0120] Let x be a test input of an AI algorithm model h, and let x' be another test input generated by adversarially perturbing x. For any x', model h has δ-local robustness at input x.

[0121]

[0122] Among them, || || p Represents the p-norm of distance measurement. The commonly used p cases in artificial intelligence algorithm model testing are 0, 2, and ∞.

[0123] 8. The model coverage of the algorithm model reflects to some extent the degree to which the model's learning ability is utilized. For a trained model M, if at least one sample in the test set makes its output exceed the preset threshold, the neuron is considered to be activated. The formula is as follows:

[0124]

[0125] Among them, Na is the number of activated neurons in the model; N is the total number of neurons involved in the calculation in the model.

[0126] 9. Algorithm model accuracy refers to the degree to which the algorithm model has the property of correctly representing the true value of the relevant attributes of a concept or event in a given environment; it includes precision and accuracy, where:

[0127] 1) Precision refers to the ratio of positive samples (true positive examples) predicted by the model to the total number of samples predicted to be positive; the formula is as follows:

[0128]

[0129] Among them, P is the precision rate; TP is the number of samples that are judged as positive but are actually positive; FP is the number of samples that are judged as positive but are actually negative.

[0130] 2) Accuracy is the ratio of the correct results predicted by the model to all classification results. The formula is as follows:

[0131]

[0132] Among them, A is the accuracy rate; TP is the number of samples that are judged to be positive and are actually positive; TN is the number of samples that are judged to be negative and are actually negative; FP is the number of samples that are judged to be positive and are actually negative; FN is the number of samples that are judged to be negative and are actually positive.

[0133] 10. The algorithm model transparency risk subdomain refers to the openness, accessibility, and clear and understandable nature of the algorithm model; the transparency metrics include: training process transparency and training result transparency. Training process transparency: During the training of the algorithm model, the training process can be analyzed by feature extraction so that the model users can understand the training process; result transparency: For the algorithm model, interpretation and analysis based on statistical results can be performed so that the model users can clearly understand the algorithm results.

[0134] 11. The algorithm dependency risk subdomain of the algorithm model refers to the quality requirements, inspection and testing conditions that may exist for the algorithm model when the design and operation of the AI ​​algorithm model rely on other algorithms or programs. The metrics include: the correctness of the dependent algorithm and the compliance of the dependent algorithm.

[0135] 1) The correctness formula of the dependent algorithm is:

[0136] X=A / B

[0137] Among them, A is the total number of defects and vulnerabilities in the dependent algorithms and library files, and B is the total number of other dependent algorithms and library files.

[0138] 2) The compliance formula for the dependent algorithm is:

[0139] X=A / B

[0140] Among them, A is the total number of problems with licenses and intellectual property rights in the algorithms and library files that are relied on, and B is the total number of other algorithms and library files that are relied on.

[0141] 12. The interpretability of algorithm models can be divided into ex ante interpretability and ex post interpretability. Ex ante interpretability: by training a model with a simple structure and good interpretability or by combining interpretability into a sub-interpretation model in a specific model structure, the model itself has the ability to be interpretable. Ex post interpretability: by developing interpretability technology to explain the trained machine learning model. Interpretability refers to the nature and characteristics of the algorithm model that can express the important factors that affect the prediction decision in a way that people can understand; the formula is as follows:

[0142] X=A / B

[0143] Among them, A is the element that users expect AI products to be explainable, and B is the explainable element provided by AI products.

[0144] Explanation accuracy refers to the correctness of the explanation recommended by the AI ​​product. The formula is as follows:

[0145] X=A / B

[0146] Among them, A is the actual correct definition or process of element interpretation, and B is the definition or process of AI product recommendation interpretation.

[0147] 13. The anti-deception attack capability of the algorithm model refers to the ability of the AI ​​algorithm model to correctly identify and process various types of adversarial samples such as forgery and attack without producing abnormal results. The formula is as follows:

[0148] X=A / B

[0149] Among them, A is the number of deception samples that the AI ​​product can accurately and correctly identify and process, and B is the number of adversarial samples detected by the AI ​​product.

[0150] 14. System operational functional safety means that the AI ​​system is free from unacceptable risks, that is, when the system is running, it will not directly or indirectly cause unacceptable risks of personal injury or property loss due to the deterioration of the product or environment; the formula is as follows:

[0151] X=A / B

[0152] Among them, A is the statistical number of times the AI ​​system can withstand risks after a failure, and B is the statistical number of times the AI ​​system fails.

[0153] 15. System operation predictability refers to the ability of developers or users to infer the next action of artificial intelligence in a given environment. The formula is as follows:

[0154] X=A / B

[0155] Among them, A is the number of prediction results provided by AI products and the predictions are accurate, and B is the number of prediction results provided by AI products.

[0156] 16. System operation reproducibility refers to the degree of consistency between the results obtained by the AI ​​system under the same measurement conditions, under the same or similar conditions, when performing multiple consecutive measurements; its measurement elements include: reproduction result deviation and reproduction success rate.

[0157] 1) Reproduction result deviation

[0158]

[0159] Where δ is the average deviation of the replicated results, x is a single measurement value, is the average measurement value, N is the number of measurements;

[0160] 2) Reproduction success rate

[0161] X=A / B

[0162] Where A is the number of times the AI ​​product executes the same process and obtains the same result under the same or similar conditions under the same measurement conditions; B is the number of times the AI ​​product executes the process and obtains the result under the same or similar conditions under the same measurement conditions;

[0163] 17. The fault tolerance of system operation refers to the ability of the system to continue to work normally or enter an emergency working state without eliminating the error or failure when certain errors or failures occur in the artificial intelligence algorithm; it includes: avoidance of failure, resistance to misoperation, harmfulness of misoperation, etc., among which:

[0164] 1) Failure avoidance refers to the ratio of critical and serious failures to be avoided. The formula is as follows:

[0165]

[0166] Where A is the number of test cases without critical and serious failures when executing the test cases required by the indicator test, and B is the total number of test cases required by the indicator test;

[0167] 2) Resistance to misoperation refers to the ratio of effective prevention of misoperation, and the formula is as follows:

[0168]

[0169] Among them, A is the number of test cases in which the model effectively resists the misoperation when executing the test cases of the test points of the misoperation, and B is the total number of test cases of the test points of the misoperation;

[0170] 3) The harmfulness of misoperation refers to the ratio of failure to cause harm in operation, and the formula is as follows:

[0171]

[0172] Among them, A is the number of test cases that cause critical and serious hazards when the test points of incorrect operation are executed and cannot be defended or not defended, and B is the total number of test cases for the test points of incorrect operation.

[0173] 18. Traceability of system operation refers to the extent to which all important matters that need to be considered in the AI ​​system are archived to help fully understand how the results of the system operation occurred; including: traceability of system execution results and fault tracing and analysis capabilities, including:

[0174] 1) Execution result traceability, the formula is as follows:

[0175] X=A / B

[0176] Among them, A is the number of execution results that can provide result logs and other files for the AI ​​system; B is the number of execution results of the AI ​​system.

[0177] 2) Fault tracing and analysis capability, the formula is as follows:

[0178] X=A / B

[0179] Among them, A is the number of AI system failures that can be tracked and analyzed through historical records, logs, etc.; B is the number of AI system failures.

[0180] 19. Controllability of system operation refers to the ability to control the AI ​​system to prevent the AI ​​system from being intentionally or unintentionally abused; including:

[0181] 1) Control success rate in dangerous scenarios, the formula is as follows:

[0182] X=A / B

[0183] Among them, A is the number of successful manual control of the system by the user in dangerous scenarios, and B is the number of tasks performed by the AI ​​system in dangerous scenarios.

[0184] 2) The control input obeys the distributed consistency, and the formula is as follows:

[0185]

[0186] Among them, x i and u i Respectively represent the state and the control input acting on the artificial intelligence product i, i = 1, 2, ..., n, V l ={i1,i2,...,i m} represents an AI product driven by external input, V f =V / V l ,u o,i Represents the external control acting on i.

[0187] 20. System adaptability refers to the degree to which a product or system can effectively and efficiently adapt to different or evolving hardware, software, other operating (or usage) environments, data or interaction methods; including: cross-platform adaptability metrics, task adaptability metrics, among which:

[0188] 1) Cross-platform adaptability metric refers to the ability of AI products to maintain the correct completion of their tasks when the software and hardware environment changes. The formula is as follows:

[0189] X=A / B

[0190] Among them, A is the number of tasks that are completed correctly in the test task, and B is the number of test tasks after the software and hardware environment of the AI ​​system changes.

[0191] 2) Task adaptability metric refers to the ability of the AI ​​system to maintain the correct completion of the task it is performing after the task it is performing changes. The formula is as follows:

[0192] X=A / B

[0193] Among them, A is the number of tasks that are completed correctly in the test task, and B is the number of test tasks executed by the AI ​​system after the task changes.

[0194] It should be noted that the above risk domains, risk subdomains and corresponding metrics are only used as examples for specific implementations, but are not limited to this. In actual applications, they can be adaptively deleted or expanded according to the types of specific artificial intelligence products and specific evaluation requirements to cover more forms of combination types, all of which are covered by the content of the hierarchical technical risk model of "risk domain-risk subdomain-metric element" provided by the present invention.

[0195] Step S20: According to the characteristics of the artificial intelligence product to be evaluated and the evaluation requirements, determine the risk subdomains and metrics corresponding to the risk domain of the artificial intelligence product to be evaluated, and formulate a technical risk model for the artificial intelligence product to be evaluated.

[0196] In the application, the risk domain in the technical risk assessment model of artificial intelligence products includes data set risk, algorithm model risk, and system operation risk, which are all mandatory options; according to the characteristics of artificial intelligence products, determine the importance of each metric in the selected risk subdomain, and divide the metric in the selected risk subdomain into important metric and general metric, where important metric is mandatory and general metric is optional (when there is only one metric in the risk subdomain, the metric is mandatory as an important metric); in each risk subdomain, select optional items and at least one mandatory item according to the assessment requirements, so as to establish a technical risk assessment model for the specific artificial intelligence product to be tested. For example, for military artificial intelligence products, the assessment requirements of their products focus on stability, security, and effectiveness; taking effectiveness as an example, since military artificial intelligence products have high requirements for the effectiveness of output results, the "data set effectiveness ratio" can be set as an important metric, and the "effectiveness of elements in the data structure" can be set as a general metric. It should be noted that the general metric elements in the embodiments of the present invention are optional items determined according to actual evaluation requirements and are not mandatory items, that is, they can be selected or not; important metric elements are mandatory items, and at least one item will be selected during the evaluation process, that is, it can be one, multiple, or all important metric elements.

[0197] Step S30: According to the technical risk model of the artificial intelligence product to be evaluated, multiple metric results required for technical risk assessment are collected.

[0198] In actual applications, the specific metric meta-assessment activities for technical risks collect multiple metric meta-results required for technical risk assessment, including assessment plan formulation, data set collection preparation and risk assessment implementation; among them, the content of assessment plan formulation includes assessment level goals, schedule, team members, assessment requirements and passing criteria, etc.; the content of data set collection preparation includes on-site confirmation, tool validity verification, confirmation of assessment methods for different risk assessment requirements, data set collection strategy, etc.; the content of risk assessment implementation includes on-site interviews, listening to statements, reviewing documents, etc.

[0199] The results of collecting multiple metrics required by technical risk assessment include data collection and data confirmation; in the data collection process, the multiple metric metadata required by technical risk assessment can be collected in categories, including data collected through tools, data collected from introductions and demonstrations, data obtained from interviews, and data obtained from reviews of relevant documents; in the data confirmation process, at least one of the metric metadata collected according to the requirements of technical risk assessment is confirmed using preset verification rules, including: the data comes from at least two independent sources, the data is obtained during at least two different collection periods, and the data is confirmed by at least one data source that can reflect that the preset rules have been followed.

[0200] Step S40: Evaluate multiple metric results according to pre-established technical risk assessment criteria to obtain a technical risk assessment conclusion.

[0201] The technical risk assessment pass criteria pre-established in the embodiment of the present invention include: setting a threshold value to be reached for each metric element based on the metric values ​​of related historical similar products or based on the goals to be achieved; if all the metric elements selected under the risk subdomain meet the threshold requirements, then the risk subdomain meets the assessment requirements; if all the selected risk subdomains under the risk domain meet the requirements, then the selected risk domain meets the assessment requirements; if all the risk domains meet the requirements, then the technical risk meets the assessment requirements.

[0202] Utilizing the collected results of multiple metrics required for technical risk assessment, the results are judged one by one against the pre-established technical risk assessment criteria, and finally the results of the technical risk assessment of the artificial intelligence product to be tested are obtained to determine whether the selected metrics meet the standards and whether the risk subdomains meet the standards. Finally, the evaluation results for AI products are obtained from the three aspects of data set, algorithm model, and system operation, that is, whether the three aspects meet the standards.

[0203] The technical risk assessment method for artificial intelligence products proposed in the embodiment of the present invention builds a technical risk assessment system for artificial intelligence products by establishing a hierarchical technical risk model of "risk domain-risk subdomain-metric element" for artificial intelligence products. It considers the technical risks of artificial intelligence products from multiple aspects, establishes an AI technical risk assessment method involved in the entire life cycle of development and application of artificial intelligence products, and solves the problem of the lack of a technical risk assessment model for artificial intelligence products.

[0204] Example 2

[0205] The embodiment of the present invention provides a technical risk assessment system for artificial intelligence products, such as Figure 3 As shown, including:

[0206] The technical risk model establishment module 10 is used for the technical risk model establishment module, and is used to establish a hierarchical technical risk model of "risk domain-risk subdomain-metric element" for artificial intelligence products, wherein each risk domain has a corresponding risk subdomain, and each risk subdomain contains a corresponding metric element. This module executes the method described in step S10 of embodiment 1, which will not be repeated here.

[0207] The module 20 for establishing the technical risk model of the product to be evaluated determines the risk subdomains and metrics corresponding to the risk domain of the artificial intelligence product to be evaluated according to the characteristics of the artificial intelligence product to be evaluated and the evaluation requirements, and formulates a technical risk model for the artificial intelligence product to be evaluated. This module executes the method described in step S20 of embodiment 1, which will not be repeated here.

[0208] The data collection module 30 is used to collect multiple metric results required for technical risk assessment according to the technical risk model of the artificial intelligence product to be assessed. This module executes the method described in step 30 of embodiment 1, which will not be repeated here.

[0209] The technical risk assessment conclusion acquisition module 40 is used to evaluate multiple metric results according to pre-established technical risk assessment criteria to obtain a technical risk assessment conclusion. This module executes the method described in step 40 of embodiment 1, which will not be described in detail here.

[0210] The technical risk assessment system for artificial intelligence products provided by the embodiments of the present invention constructs a technical risk assessment system for artificial intelligence products by establishing a hierarchical technical risk model of "risk domain-risk subdomain-metric element" for artificial intelligence products. It considers the technical risks of artificial intelligence products from multiple aspects, establishes an AI technical risk assessment method involved in the entire life cycle of development and application of artificial intelligence products, and solves the problem of the lack of a technical risk assessment model for artificial intelligence products.

[0211] Example 3

[0212] An embodiment of the present invention provides a computer device, such as Figure 4 As shown, the device may include a processor 51 and a memory 52, wherein the processor 51 and the memory 52 may be connected via a bus or other means. Figure 4 Take the example of connecting via a bus.

[0213] The processor 51 may be a central processing unit (CPU). The processor 51 may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or a combination of the above chips.

[0214] The memory 52 is a non-transitory computer-readable storage medium that can be used to store non-transitory software programs, non-transitory computer executable programs and modules, such as the corresponding program instructions / modules in the embodiment of the present invention. The processor 51 executes various functional applications and data processing of the processor by running the non-transitory software programs, instructions and modules stored in the memory 52, that is, the technical risk assessment method of the artificial intelligence product in the above method embodiment 1 is implemented.

[0215] The memory 52 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and applications required for at least one function; the data storage area may store data created by the processor 51, etc. In addition, the memory 52 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage devices. In some embodiments, the memory 52 may optionally include a memory remotely arranged relative to the processor 51, and these remote memories may be connected to the processor 51 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, an intranet, a mobile communication network, and combinations thereof.

[0216] One or more modules are stored in the memory 52, and when executed by the processor 51, the technical risk assessment method of the artificial intelligence product in Example 1 is executed.

[0217] The specific details of the above-mentioned computer device can be understood by referring to the corresponding related descriptions and effects in Example 1, and will not be repeated here.

[0218] Those skilled in the art can understand that the implementation of all or part of the processes in the above-mentioned embodiments can be completed by instructing the relevant hardware through a computer program. The program can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the storage medium can be a disk, an optical disk, a read-only memory (ROM), a random access memory (RAM), a flash memory, a hard disk drive (HDD) or a solid-state drive (SSD), etc.; the storage medium can also include a combination of the above-mentioned types of memory.

[0219] Obviously, the above embodiments are merely examples for the purpose of clear explanation, and are not intended to limit the implementation methods. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the implementation methods here. The obvious changes or modifications derived from these are still within the scope of protection of the invention.

Claims

1. A technical risk assessment method for an artificial intelligence product, characterized in that: The steps include: Establish a hierarchical technical risk model of "risk domain-risk subdomain-metric element" for artificial intelligence products, in which each risk domain has a corresponding risk subdomain, and each risk subdomain contains a corresponding metric element, among which: risk domain includes: data set risk, algorithm model risk, system operation risk; among which, The risk subdomains of dataset risk include: unbiasedness, completeness, validity, confidentiality, accuracy, and dataset distribution balance; The risk subdomains of algorithmic model risk include: robustness, model coverage, accuracy, transparency, algorithm dependency, explainability, and resistance to deception attacks; The risk subdomains of system operation risk include: functional safety, predictability, reproducibility, fault tolerance, traceability, controllability and adaptability; According to the characteristics of the AI ​​product to be evaluated and the evaluation requirements, determine the risk subdomains and metrics corresponding to the risk domain of the AI ​​product to be evaluated, and formulate a technical risk model for the AI ​​product to be evaluated; According to the technical risk model of the AI ​​product to be evaluated, collect multiple metric results required for technical risk assessment; Evaluate multiple metric results according to pre-established technical risk assessment criteria to draw technical risk assessment conclusions, where the pre-established technical risk assessment criteria include: The thresholds to be reached for each metric element are set based on the metric values ​​of related historical similar products or the goals to be achieved; if all the metric elements selected under the risk subdomain meet the threshold requirements, then the risk subdomain meets the assessment requirements; if all the selected risk subdomains under the risk domain meet the requirements, then the selected risk domain meets the assessment requirements; if all the risk domains meet the requirements, then the technical risk meets the assessment requirements; The process of evaluating multiple metric results according to the pre-established technical risk assessment criteria to draw a technical risk assessment conclusion includes: By utilizing the collected multiple metric results required for technical risk assessment, the results are judged one by one against the pre-established technical risk assessment criteria, and ultimately the results of the technical risk assessment of the artificial intelligence product to be tested are obtained.

2. The method for assessing technical risks of artificial intelligence products according to claim 1, characterized in that: The metrics of dataset integrity include: dataset model integrity, dataset model attribute integrity, and metadata integrity; The metrics of data set effectiveness include: data set effectiveness ratio, data element effectiveness; Metrics for data set confidentiality include: use of encryption and non-vulnerability; The metrics of dataset accuracy include: semantic accuracy, dataset inaccuracy risk, and metadata accuracy; The metrics of data set distribution balance include: category balance and misclassification cost balance; The metrics of algorithm model robustness include: output accuracy in abnormal samples, local adversarial robustness, and global adversarial robustness; The metrics of algorithm model accuracy include: precision and accuracy; The metrics of algorithm model transparency include: training process transparency and training result transparency; The metrics of algorithm model algorithm dependency include: the correctness of the dependent algorithm, the compliance of the dependent algorithm; The metrics of the interpretability of algorithmic models include: ex ante interpretability and ex post interpretability; The metrics of the reproducibility of system operation include: the deviation of the reproduction results and the reproduction success rate; The measurement elements of the fault tolerance of system operation include: avoidance of failure, resistance to misoperation, and harmfulness of misoperation; The measurement elements of system operation traceability include: system execution result traceability and fault tracing and analysis capabilities; The controllability metrics of the system operation include: control success rate in dangerous scenarios, control input compliance with distributed consistency; The measurement elements of system operation adaptability include: cross-platform adaptability and task adaptability.

3. The method for assessing technical risks of artificial intelligence products according to claim 1, characterized in that: The process of determining the risk subdomains and metrics corresponding to the risk domain of the AI ​​product to be evaluated based on the characteristics of the AI ​​product to be evaluated and the evaluation requirements, and formulating a technical risk model for the AI ​​product to be evaluated, including: The risk domains in the AI ​​product technology risk assessment model include data set risk, algorithm model risk, and system operation risk, all of which are mandatory options; According to the characteristics of the AI ​​product, determine the importance of each metric in the selected risk subdomain, and divide the metrics in the selected risk subdomain into important metrics and general metrics, where important metrics are mandatory and general metrics are optional; In each risk subdomain, select optional items and at least one mandatory item based on the assessment requirements to establish a technical risk assessment model for the specific AI product to be tested.

4. The method for assessing technical risks of artificial intelligence products according to claim 1, characterized in that: Collecting multiple metrics required for technical risk assessment includes: data collection and data confirmation; among them, During the data collection process, multiple measurement metadata required for technical risk assessment can be collected in categories, including data collected through tools, data collected from introductions and demonstrations, data obtained from interviews, and data obtained from reviews of relevant documents; During the data confirmation process, at least one of the metric metadata collected according to the technical risk assessment requirements is confirmed using preset verification rules, including: the data comes from at least two independent sources, the data is obtained during at least two different collection periods, and the data is confirmed by at least one data source that reflects that the preset rules have been followed.

5. A technical risk assessment system for artificial intelligence products, characterized in that: include: The technical risk model building module is used to build a hierarchical technical risk model of "risk domain-risk subdomain-metric element" for artificial intelligence products, where each risk domain has a corresponding risk subdomain, and each risk subdomain contains a corresponding metric element; The module for establishing the technical risk model of the product to be evaluated determines the risk subdomains and metrics corresponding to the risk domain of the artificial intelligence product to be evaluated according to the characteristics of the artificial intelligence product to be evaluated and the evaluation requirements, and formulates a technical risk model for the artificial intelligence product to be evaluated; A data collection module, used to collect multiple metric results required for technical risk assessment according to the technical risk model of the artificial intelligence product to be assessed; The technical risk assessment conclusion acquisition module is used to evaluate multiple metric results according to the pre-established technical risk assessment passing criteria to obtain the technical risk assessment conclusion. The pre-established technical risk assessment passing criteria include: The thresholds that need to be reached are set for each metric element based on the metric values ​​of related historical similar products or the goals that need to be achieved; if all the metric elements selected under the risk subdomain meet the threshold requirements, then the risk subdomain meets the assessment requirements; if all the selected risk subdomains under the risk domain meet the requirements, then the selected risk domain meets the assessment requirements; if all risk domains meet the requirements, then the technical risk meets the assessment requirements.

6. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable the computer to execute the technical risk assessment method for an artificial intelligence product as described in any one of claims 1 to 4.

7. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the technical risk assessment method for an artificial intelligence product as described in any one of claims 1 to 4 by executing the computer instructions.