Fault mode recognition and analysis method for intelligent autonomous system
By building a framework and guidelines for fail mode analysis of intelligent autonomous systems and combining multi-dimensional analysis to identify and control the failure mode of intelligent autonomous systems, the problem of difficult failure modes in complex environments is solved and the safety and reliability of the system is improved.
Patent Information
- Application Number
- CN202411087576.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-09
- Publication Date
- 2024-12-03
AI Technical Summary
The failure mode of intelligent autonomous systems is difficult to predict and identify in complex environments, resulting in unstable system operation and may cause safety accidents. The existing technology lacks systematic analysis methods to deeply analyze the root causes and influencing factors of the failure.
A method for identifying and analysis of fault modes of intelligent autonomous systems is proposed. By building a fault mode analysis framework and criteria, combining intelligent algorithms, hardware and software composition and environmental scenarios, a compliance checklist is generated to obtain security risk levels, and multi-dimensional analysis and specific criteria are adopted. Guide fault pattern identification and control measures.
It provides a set of structured analysis steps to help designers understand the causes of failures based on the internal mechanism of the system, ensure the accuracy and comprehensiveness of failure mode analysis, improve the safety and reliability of the system, and reduce the impact of failures.
Smart Images

Figure CN119066570A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of intelligent system fault analysis, and in particular to a fault pattern recognition and analysis method for an intelligent autonomous system. Background Art
[0002] With major breakthroughs in key technologies such as machine learning, deep learning, computer vision, and natural language processing, artificial intelligence has penetrated into every aspect of every field. Failure mode analysis of intelligent autonomous systems is a necessary task to ensure that intelligent autonomous systems can operate safely, stably, and efficiently under the background of the rapid development of intelligent and automated technologies. Intelligent autonomous systems use complex algorithms and sensors to achieve perception, decision-making, and execution of complex environments, thereby completing various complex tasks.
[0003] As intelligent autonomous systems are increasingly used in various fields, they inevitably face the risk of failure and failure. System failures not only affect the normal operation of the system, but may even lead to serious safety accidents. Therefore, it is of great significance to analyze the failure modes of intelligent autonomous systems, identify potential failures, and prevent failures from occurring, in order to ensure the safety and reliability of the system. In order to ensure the reliability and stability of intelligent autonomous systems, it is necessary to deeply analyze their possible failure modes so as to identify, prevent, and solve problems as early as possible.
[0004] Failure mode analysis is a systematic approach that identifies, classifies and evaluates potential failures in each component of a system to determine their possible impact and consequences. This approach can help engineers foresee possible failures during the design phase and take appropriate preventive measures. At the same time, during the system operation phase, failure mode analysis can also help quickly locate and resolve failures and reduce the impact of failures on the system.
[0005] In the failure mode analysis of intelligent autonomous systems, special attention should be paid to the autonomy and intelligence of the system. Since intelligent autonomous systems can make decisions and execute autonomously according to environmental changes, their failure modes may also be more complex and difficult to predict. This requires analysts to have not only in-depth technical knowledge, but also the ability to understand and predict system behavior. Summary of the invention
[0006] In view of this, the present invention proposes a fault mode recognition and analysis method for an intelligent autonomous system to solve the problems existing in the above-mentioned prior art.
[0007] To achieve the above object, the present invention proposes a fault mode recognition and analysis method for an intelligent autonomous system, comprising:
[0008] Construct the intelligent autonomous system failure mode analysis framework and intelligent autonomous system failure mode analysis criteria respectively;
[0009] The intelligent autonomous system failure mode analysis framework performs fault identification, recording and evaluation of the intelligent autonomous system based on the intelligent autonomous system failure mode analysis criteria, and generates a compliance checklist;
[0010] A security risk level of the intelligent autonomous system is obtained based on the compliance checklist.
[0011] Optionally, the process of constructing the intelligent autonomous system failure mode analysis framework includes:
[0012] Analyze the functional and technical characteristics of intelligent autonomous systems;
[0013] Obtaining failure influencing factors based on the functional characteristics and the technical characteristics;
[0014] Based on the fault influencing factors, the fault mode analysis framework of the intelligent autonomous system is constructed according to the core intelligent algorithm, software and hardware composition, operating environment and scenarios of the intelligent autonomous system.
[0015] Construct the intelligent autonomous system failure mode analysis framework and intelligent autonomous system failure mode analysis criteria respectively;
[0016] The intelligent autonomous system failure mode analysis framework performs fault identification, recording and evaluation of the intelligent autonomous system based on the intelligent autonomous system failure mode analysis criteria, and generates a compliance checklist;
[0017] A security risk level of the intelligent autonomous system is obtained based on the compliance checklist.
[0018] Furthermore, the construction process of the intelligent autonomous system failure mode analysis framework includes:
[0019] Analyze the functional and technical characteristics of intelligent autonomous systems;
[0020] Obtaining failure influencing factors based on the functional characteristics and the technical characteristics;
[0021] Based on the fault influencing factors, the fault mode analysis framework of the intelligent autonomous system is constructed according to the core intelligent algorithm, software and hardware composition, operating environment and scenarios of the intelligent autonomous system.
[0022] Furthermore, the fault influencing factors include software and hardware fault factors, intelligent algorithm fault factors, training sample fault factors, and OODA task chain fault factors;
[0023] The software and hardware failure factors include: open source component reliability risk, algorithm framework reliability risk, application software security risk, human-computer interaction security risk, hardware environment reliability risk, software and hardware coupling reliability risk, and system behavior reliability risk;
[0024] The intelligent algorithm failure factors include: model accuracy and robustness, model complexity, model hyperparameters, algorithm performance, algorithm bias or discrimination, and algorithm nonlinear characteristics;
[0025] The training sample failure factors include: data set reliability, training sample diversity, training sample annotation quality, training sample balance, training sample preprocessing quality, data annotation security compliance, and automated decision security compliance;
[0026] The OODA task chain failure factors include: the impact of data characteristics on the perception layer, the impact of behavior changes on the judgment layer, the impact of scenario complexity on the judgment layer, and the impact of output instability on the action layer.
[0027] Furthermore, the process of constructing the intelligent autonomous system failure mode analysis criterion includes:
[0028] Obtaining failure modes for input data and training processes of intelligent autonomous systems;
[0029] The intelligent autonomous system failure mode analysis criterion is constructed based on the failure mode.
[0030] Furthermore, the failure modes include intelligent algorithm failure modes, software and hardware composition failure modes, and environment and task failure modes;
[0031] The intelligent algorithm failure modes include data defect failure modes and algorithm failure modes; the data defect failure modes include data imbalance, data inaccuracy, and data incompleteness; the algorithm failure modes include algorithm fitting failure, algorithm implementation failure, algorithm robustness failure, algorithm performance failure, algorithm hardware dependency failure, and algorithm deployment failure; adversarial training sample failure;
[0032] The software and hardware fault modes include software and hardware failure fault modes and software and hardware coupling fault modes; the software and hardware failure fault modes include software item failure faults and hardware item failure faults; the software and hardware coupling fault modes include data coupling faults, control coupling faults and dynamic real-time constraint faults;
[0033] The environment and task failure modes include target recognition failure mode, route planning failure mode, and decision reasoning failure mode.
[0034] Furthermore, the intelligent autonomous system failure mode analysis framework is based on model input, failure criterion input, failure mode input and scenario input, and analyzes the failure mode of the intelligent autonomous system from the aspects of intelligent algorithm, software and hardware composition, environment and task, and on the basis of the intelligent autonomous system failure mode analysis criterion, to obtain the failure cause, failure mode, failure impact and control measures, and generate the compliance checklist.
[0035] Furthermore, the compliance checklist includes a compliance checklist for intelligent algorithms, a compliance checklist for software and hardware configurations, and a compliance checklist for environments and tasks.
[0036] Furthermore, the process of obtaining the security risk level of the intelligent autonomous system based on the compliance checklist includes:
[0037] Obtain failure analysis criteria based on the compliance checklist, and perform coverage analysis and inspection on the demand design elements of the intelligent algorithm;
[0038] Identify the cause of input failure, evaluate the impact of the failure on the system output in a graded manner, generate a severity classification table for the impact of intelligent autonomous system failures, and obtain the impact consequence level;
[0039] Obtaining the fault propagation path and propagation mode, and implementing fault control measures based on the impact consequence level, the fault propagation path and propagation mode;
[0040] Conducting effectiveness analysis on the fault cause and the fault control measures;
[0041] Analyze the control or mitigation capabilities of the intelligent autonomous system when algorithm failures occur, generate a control capability rating table for the intelligent autonomous system, and obtain the control capability rating of the intelligent autonomous system;
[0042] Based on the impact consequence level and the control capability level, a multi-dimensional evaluation matrix is used to construct an intelligent algorithm security risk level index matrix, and the security risk level is obtained based on the intelligent algorithm security risk level index matrix.
[0043] Furthermore, the method for identifying the cause of the fault includes: root cause analysis, fault tree analysis, cause-and-effect diagram, expert review and group discussion, data analysis and statistical methods, simulation and testing, and fault reproduction.
[0044] Furthermore, the autonomous system fault impact severity grading table divides the severity level of intelligent algorithm faults into 5 levels. If there are multiple system function faults that jointly cause a higher level of system hazard, the severity level of the fault is determined according to the highest level of system hazard;
[0045] The intelligent autonomous system control capability level table divides the control capability level of the intelligent autonomous system into four levels. If the same intelligent algorithm corresponds to multiple control levels, the highest level is taken as the control level of the intelligent algorithm.
[0046] Compared with the prior art, the present invention has the following beneficial effects:
[0047] The present invention deeply analyzes the functional and technical characteristics of the intelligent autonomous system, helps designers understand the root causes of failures from the internal mechanism of the system, and also emphasizes the influencing factors of intelligent autonomous system failures, providing users with a set of structured analysis steps. Starting from determining the analysis object, to collecting relevant data and information, and then using different analysis methods and techniques to analyze the failure modes one by one, and finally forming a detailed analysis report. The analysis framework also emphasizes the comprehensive consideration of multiple dimensions. When performing failure mode analysis, users need to consider multiple aspects such as intelligent algorithms, software and hardware composition, environment and tasks. In addition, the analysis framework also includes a series of specific analysis criteria. These criteria cover the classification of failure modes and provide clear guidance for users. By following these criteria, the failure modes of intelligent autonomous systems can be analyzed more objectively and accurately.
[0048] The present invention provides designers with a more comprehensive analysis framework that can provide valuable information and guidance to help users better cope with various failure situations that intelligent autonomous systems may encounter and ensure that the system's safety, reliability and performance reach a high level. BRIEF DESCRIPTION OF THE DRAWINGS
[0049] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present invention. Moreover, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:
[0050] Figure 1 A schematic diagram of a fault mode recognition and analysis method for an intelligent autonomous system of the present invention;
[0051] Figure 2 It is the OODA structure model of the OODA intelligent autonomous system in the embodiment of the present invention;
[0052] Figure 3 It is a failure mode and impact analysis framework of the intelligent autonomous system in the embodiment of the present invention;
[0053] Figure 4 Schematic diagram of the intelligent autonomous system failure mode and its impact analysis process in an embodiment of the present invention. DETAILED DESCRIPTION
[0054] Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided in order to enable a more thorough understanding of the present disclosure and to be able to fully convey the scope of the present disclosure to those skilled in the art. It should be noted that, in the absence of conflict, the embodiments of the present invention and the features described in the embodiments can be combined with each other. The present invention will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.
[0055] This embodiment proposes a fault mode recognition and analysis method for an intelligent autonomous system. Figure 1 As shown, including:
[0056] Construct the intelligent autonomous system failure mode analysis framework and intelligent autonomous system failure mode analysis criteria respectively;
[0057] The intelligent autonomous system failure mode analysis framework performs fault identification, recording and evaluation of the intelligent autonomous system based on the intelligent autonomous system failure mode analysis criteria, and generates a compliance checklist;
[0058] A security risk level of the intelligent autonomous system is obtained based on the compliance checklist.
[0059] The failure mode analysis method of intelligent autonomous systems consists of two key parts: analysis framework and analysis criteria. The construction of the analysis method usually includes the following two elements:
[0060] Failure Mode Analysis Framework for Intelligent Autonomous Systems: The Failure Mode Analysis Framework for Intelligent Autonomous Systems is a structured methodology that specifies the steps, processes, and tools for conducting a failure mode analysis. It provides an organizational structure for the entire analysis, enabling personnel to systematically identify, record, and evaluate various failure modes. The design of the Failure Mode Analysis Framework for Intelligent Autonomous Systems often depends on the specific application domain and problem type. In the Failure Mode Analysis Framework for Intelligent Autonomous Systems, inputs, technical processes, outputs, etc. are included to help relevant personnel determine how to conduct a failure mode analysis.
[0061] Failure mode analysis criteria for intelligent autonomous systems: Failure mode analysis criteria for intelligent autonomous systems are a set of rules, standards, or evaluation methods used to guide intelligent autonomous system developers to identify various failure modes in intelligent autonomous systems from a black box perspective, and provide knowledge support for the criterion-driven intelligent autonomous system failure analysis process. These criteria are usually based on experience, domain knowledge, and previous failure cases. Failure mode analysis criteria for intelligent autonomous systems provide a way to judge different failure modes to determine which failure modes have occurred in the system.
[0062] Analysis of functional technical characteristics and failure influencing factors of intelligent autonomous systems:
[0063] Autonomous intelligent unmanned systems (AIS) rely on big data and artificial intelligence (and other scientific and technological advances) to create unmanned systems with integrated mission and motion planning, as well as decision-making and reasoning capabilities. Autonomous intelligent unmanned systems can complete general tasks with no or limited human involvement, which is made possible by the integration of digital design (artificial intelligence, big data, control, etc.), robotics, and constructed environments. Typical autonomous intelligent unmanned systems include self-driving cars, intelligent manufacturing robots, and robots for companionship or comfort.
[0064] The intelligent autonomous system involves four functional characteristics during operation, namely "perception", "judgment", "decision-making" and "behavior", forming an OODA functional feedback loop, which mainly includes the perception layer, judgment layer, decision-making layer and behavior layer. Figure 2 shown.
[0065] Perception layer: including the hardware and software components for realizing target perception (for example, radar sensors, target perception and acquisition software, etc.), intelligent learning algorithms for target perception (for example, neural network algorithms), training samples for target perception intelligent learning, target perception intelligent learning capabilities (for example, target perception accuracy, recall rate and other indicators), etc.;
[0066] Judgment layer: including intelligent learning algorithms for target judgment fusion (e.g., neural network algorithms), training samples for target judgment fusion intelligent learning, target judgment fusion intelligent learning capabilities (e.g., accuracy and recall of target judgment fusion), etc.;
[0067] Decision-making layer: including intelligent learning algorithms for comprehensive decision-making (e.g., neural network algorithms), training samples for comprehensive decision-making intelligent learning, comprehensive decision-making intelligent learning capabilities (e.g., comprehensive decision-making accuracy, recall rate and other indicators), and comprehensive decision-making intelligent prediction capabilities (based on collected data and intelligent learning algorithms, predicting future system behaviors);
[0068] Behavioral layer: includes the hardware and software components that implement system behavior (for example, the device mechanism that executes control instructions, the intelligent decision-making software that outputs control instructions, etc.).
[0069] Depend on Figure 2 It can be seen that based on the OODA loop and IDEF modeling method, the intelligent autonomous system is divided into four parts when performing tasks: perception layer, judgment layer, decision layer, and action layer. The functional information interaction and dynamic transfer relationship between these four parts are analyzed as follows:
[0070] The perception layer of the intelligent system is mainly responsible for information collection. When performing tasks, the perception layer will extract task environment information, location information of the intelligent system, status data, etc. from the constantly changing external environment, that is, other information.
[0071] The perception layer passes the collected environmental information to the judgment layer. The judgment layer of the intelligent system is mainly responsible for analyzing the tasks performed by the intelligent system. When performing tasks, the judgment layer of the intelligent system will fuse the information passed by the perception layer, and perform target detection, environmental information analysis and prediction through information fusion.
[0072] The judgment layer passes the analysis results to the decision layer of the intelligent system. The decision layer of the intelligent system is mainly responsible for task decision-making. When executing a task, the decision layer of the intelligent system will plan the action path of the intelligent system, make decisions on the actions to be taken by the intelligent system (for example, whether to strike the target), and handle errors.
[0073] The decision layer of the intelligent system will feed back some information to the perception layer and pass the action plan to the action layer. The action layer of the intelligent system is mainly responsible for the execution of the tasks of the intelligent system. Specifically, it includes trajectory tracking, motion control, communication control, etc. of the intelligent system, and finally feeds back the updated information to the perception layer.
[0074] Technical characteristics of intelligent autonomous systems:
[0075] Technical characteristics include autonomy, collaboration, learning, and adaptability.
[0076] Autonomy: refers to the independence and initiative of intelligent autonomous systems in tasks, which is manifested in the ability to complete related tasks under human guidance or independently.
[0077] Synergy: refers to the ability of intelligent autonomous systems to achieve self-organization, task allocation and activity coordination through interaction.
[0078] Learning: refers to the ability of an intelligent autonomous system to summarize knowledge and conduct deductive reasoning based on historical data and external resources, and to have the ability to upgrade and improve itself.
[0079] Adaptability: refers to the ability of intelligent autonomous systems to autonomously adjust their own behaviors and strategies according to environmental changes and task requirements.
[0080] Failure factors affecting intelligent autonomous systems:
[0081] According to the reliability connotation of intelligent autonomous systems and the OODA functional characteristic loop theory, in a step-by-step manner of "fault perception-abnormal judgment-decision error-execution failure", the internal and external factors that lead to intelligent autonomous system failures are analyzed and summarized in terms of the hardware and software composition, intelligent algorithms, training data, and mission scenarios of intelligent autonomous systems. Mainly including:
[0082] (1) Data has complex characteristics such as multi-dimensional space, multi-source heterogeneity, and adversarial game, which affect the real-time fusion and detection of the "perception layer" of the intelligent autonomous system, thus leading to perception failure.
[0083] The situational awareness system, which is supported by various sensor devices such as radar and infrared, provides a large amount of data with characteristics such as multi-dimensional space, multi-source heterogeneity, and confrontational game. As a result, the single-source data is often incomplete and invalid data that is interfered with. Therefore, how to achieve intelligent fusion of detection data from multiple sensor platforms, detect and eliminate interference / abnormal data, and provide accurate and effective data information for various tasks is the premise and basis for conducting fault analysis of the "perception layer" of intelligent autonomous systems.
[0084] (2) Behavior is difficult to predict and changes rapidly, which affects the accuracy of the "judgment layer" and leads to abnormal judgment.
[0085] Based on real-time perception information, the attributes and location of the target source are identified to achieve rapid analysis and judgment of the target attributes, levels and behaviors. Further mining of massive information is to extract, understand and predict behaviors, clarify the current location of the target source, and thus evaluate environmental information to provide a reliable basis for decision-making. However, behaviors in the environment are difficult to predict and change rapidly, which affects the "judgment layer" of the intelligent autonomous system to accurately diagnose and judge the perception data.
[0086] (3) Complex scenarios have complex characteristics such as "highly complex environment, highly real-time dynamics, uncertain boundaries, incomplete information, and highly confrontational games", which affect the accuracy and effectiveness of the "decision-making layer" and thus lead to wrong decisions.
[0087] Intelligent autonomous systems usually work in complex scenarios such as "highly complex environment, high dynamic real-time, uncertain boundaries, incomplete information, and strong game confrontation". Complex scenarios will produce unexpected abnormal interference incentives for the intelligent algorithm training and learning process of the decision-making layer, which will lead to output errors or unknown decision results.
[0088] (4) The “perception-judgment-decision-making” of intelligent autonomous systems leads to uncertain and unstable outputs, which affects the robustness and controllability of the “behavior layer”:
[0089] Intelligent autonomous systems operate in complex environments. The intelligent characteristics of "perception-diagnosis-decision-making" such as autonomous learning and evolution lead to uncertainty and instability in the system output, and place high demands on the response and control capabilities of the actuators, which in turn affects the robustness and controllability of the "behavioral layer" of the intelligent autonomous system, such as data output and mechanism manipulation.
[0090] Therefore, for the four-layer structure model of the intelligent autonomous system, namely the perception layer, judgment layer, decision layer and action layer, the uncertainty of the intelligent algorithm, and the OODA structure model of the intelligent autonomous system / algorithm, combined with information such as tasks and environment, the failure influencing factors are analyzed from the perspectives of the hardware and software composition of the intelligent autonomous system, intelligent algorithms, training data, and mission scenarios as follows:
[0091] (1) Failure factors in software and hardware components:
[0092] The composition of intelligent autonomous systems is usually composed of upper-level application software and algorithm frameworks, especially the algorithm framework, which affects reliability. In terms of software composition, reliability influencing factors include:
[0093] Open source component reliability risks:
[0094] The current rapid development of artificial intelligence technology and industry is largely due to the open source of necessary experimental and production tools such as mainstream artificial intelligence software, frameworks, and dependency libraries. More and more entrepreneurs can rely on open source results to conduct artificial intelligence research. The open source community has played a key role in the development of artificial intelligence in terms of function optimization and framework design, but often ignores the security risks of its results. Open source components may contain undiscovered or unfixed security vulnerabilities, which may be maliciously exploited, thus affecting the reliability of the entire system. Open source components may also be subject to license restrictions. Violating these restrictions may lead to legal issues and indirectly affect the reliability of the system.
[0095] Algorithm framework reliability risks:
[0096] Machine learning-related software frameworks, tools, and dependency libraries such as TensorFlow and Caffe have security vulnerabilities that may be used in cyber attacks, posing new threats and challenges to artificial intelligence applications.
[0097] Application software security risks:
[0098] Artificial intelligence infrastructure consists of software and hardware, and also faces traditional software security risks. It is necessary to pay attention to issues such as service interface security, software and hardware security, and service availability.
[0099] Human-computer interaction safety risks:
[0100] In many cases, the performance of intelligent autonomous systems is greatly affected by human interactions. Different people may have different response characteristics, and changes in human responses may affect the safety of the system.
[0101] Hardware environment reliability risks:
[0102] Hardware devices may have faults or performance issues, such as sensor failure, actuator failure, etc. These problems will directly affect the reliability and performance of the system. The stability of the hardware environment may also be affected by external factors, such as power fluctuations, electromagnetic interference, etc.
[0103] Software and hardware coupling reliability risks:
[0104] There may be problems with the interface or communication between software and hardware, such as communication delays, data loss, etc., which will affect the overall reliability of the system. The matching degree between software and hardware may also affect the stability and reliability of the system.
[0105] System behavior reliability risks:
[0106] In many cases, intelligent autonomous systems are designed to operate in complex environments with a large number of potential states that cannot be exhaustively examined or tested. The system may face situations that were never considered during the design process. For intelligent autonomous systems that learn after deployment, the behavior of the system may be mainly determined by the situations learned under unsupervised conditions. In this case, it may be difficult to predict the behavior of the system.
[0107] (2) Failure factors affecting intelligent algorithms
[0108] The algorithm model is the core of the artificial intelligence system, and the potential safety hazards in the algorithm model may bring fatal safety consequences to the artificial intelligence system. When performance refers to the accuracy of the algorithm, especially when the algorithm to be discussed is based on a classifier, the performance at this time refers to the "generalization ability" of the algorithm. When performance refers to the accuracy of the algorithm, the factors affecting reliability are as follows:
[0109] It is difficult to balance model accuracy and robustness:
[0110] While pursuing high model accuracy, its robustness (i.e., the model's tolerance to noise and outliers) may be sacrificed. This requires a trade-off during model design and training to find the best balance between accuracy and robustness.
[0111] Model complexity has a great impact on model accuracy:
[0112] A model that is too complex may lead to overfitting, that is, it performs well on the training data but poorly on the test data. On the contrary, a model that is too simple may not fully capture the inherent laws of the data, leading to underfitting. Therefore, a trade-off needs to be made between model complexity and accuracy.
[0113] Model hyperparameters have a great impact on model accuracy:
[0114] Hyperparameters are parameters that need to be manually set during model training, such as learning rate, number of iterations, etc. The selection of these hyperparameters has an important impact on the training effect of the model. If the hyperparameters are not set properly, it may lead to poor model training effect or failure to converge.
[0115] Algorithm performance reliability challenges:
[0116] The performance of the algorithm may be challenged due to factors such as data noise and outliers. In addition, different data sets and scenarios may also lead to differences in algorithm performance. Therefore, the algorithm needs to be fully tested and verified to ensure its stability and reliability in various situations.
[0117] Algorithms may harbor bias or discrimination, leading to skewed or inappropriate results:
[0118] During the algorithm design and training process, if there is bias or discrimination in the data, the algorithm may learn these biases and reflect them in its output results. This may lead to biased results or improper processing, which will have a negative impact on society. In order to avoid this, corresponding measures need to be taken during the algorithm design and training process to reduce or eliminate bias and discrimination.
[0119] The “black box” nature of AI algorithm decision-making raises issues of interpretability and transparency of results:
[0120] The "black box" feature of AI algorithm decision-making means that its decision-making process is opaque to humans, that is, we cannot directly observe and understand how the algorithm makes the final decision based on the input data. This is mainly due to the complexity and nonlinear characteristics of complex models such as deep learning, which makes its internal working principles and calculation processes difficult to be intuitively understood and explained. This makes it impossible to fully explain and verify the output results of the algorithm, thereby reducing the reliability and credibility of the algorithm.
[0121] (3) Fault influencing factors in training samples
[0122] Data is the basic resource of artificial intelligence. Machine learning requires a large amount of data with many types and high quality for training. The data types are divided into massive raw data, training data sets, test data sets, field data, etc. In terms of training samples, the factors affecting reliability are as follows:
[0123] The size of the training dataset has a significant impact on the reliability of autonomous AI systems
[0124] The size of the training dataset has an important impact on the reliability of AI autonomous systems. Generally, larger datasets can cover more situations and scenarios, helping the model learn richer features, thereby improving the generalization ability and accuracy of the model. However, as the size of the dataset increases, it may also bring about problems such as increased consumption of computing resources and longer training time.
[0125] The diversity of training samples will also affect the reliability of the model:
[0126] Diversity is the key to ensure that the model can handle a variety of complex situations. The training samples should cover as many situations and categories as possible in actual application scenarios to improve the robustness and generalization ability of the model.
[0127] The quality of the annotations of the training samples has a great impact on the reliability of the model:
[0128] The quality of annotation directly determines the accuracy and reliability of model learning. If there are errors or deviations in the annotation data, the model may learn wrong information, resulting in poor performance in practical applications. Therefore, high-quality annotation data is the key to ensuring model performance.
[0129] The balance of training samples will also affect the reliability of the model:
[0130] The balance of training samples refers to whether the number of samples of different categories is balanced. If the number of samples of certain categories is too large or too small, it may cause the model's recognition ability of these categories to decline. For example, in a binary classification task, if the number of positive samples is much larger than the number of negative samples, the model may overfit the positive samples, resulting in a decline in the recognition ability of negative samples. Therefore, in order to ensure the overall performance of the model, it is necessary to ensure that the number of different categories in the training samples is relatively balanced. This can be achieved by oversampling the minority category or undersampling the majority category. By adjusting the number of samples of different categories, the model can treat each category more fairly and improve its overall recognition ability.
[0131] The preprocessing of training samples will also affect the reliability of the model:
[0132] Preprocessing is a very critical step in a machine learning project, which includes data cleaning, feature selection, normalization and other links. The quality of preprocessing directly affects the effect and accuracy of subsequent model training. If the preprocessing is not done properly, it may cause the model to learn wrong information or fail to extract features effectively. For example, if there are missing values or outliers in the data without proper processing, then these values may interfere with the model training process and cause the model performance to deteriorate. Similarly, if the feature selection is improper or the normalization method is not appropriate, it may also affect the performance of the model. Therefore, it is necessary to carefully consider and select appropriate preprocessing methods and techniques to ensure the reliability and performance of the model.
[0133] Data labeling security risks and compliance issues:
[0134] The data labeling process may involve the processing of sensitive information, such as personal privacy, business secrets, etc. If this data is improperly used or leaked, it may cause serious losses to individuals or companies. Therefore, it is necessary to ensure that the data labeling process complies with the requirements of relevant laws and regulations, and take corresponding security measures to protect sensitive information. For example, a strict data access permission control mechanism can be established to ensure that only authorized personnel can access sensitive data. At the same time, encryption technology can also be used to protect the security of data during transmission and storage. By strengthening security management and compliance control in the data labeling process, the security and privacy of sensitive information can be effectively protected.
[0135] Automated decision-making privacy compliance issues:
[0136] Automated decision-making systems may involve the processing and use of personal privacy data. When building and using these systems, it is necessary to ensure compliance with relevant laws and regulations on privacy protection. This includes the need to follow the principles of privacy protection in all aspects of data collection, storage, processing and use. For example, in the EU, it is necessary to follow the provisions of the General Data Protection Regulation (GDPR) to ensure the legality and compliance of personal data. This includes obtaining the user's explicit consent, providing transparent data processing information, and protecting the user's data access and deletion rights. By complying with these privacy protection laws and regulations and requirements, the legality and compliance of automated decision-making systems can be ensured, and the personal privacy rights and interests of users can be protected.
[0137] (4) Factors affecting OODA task chain failures:
[0138] The intelligent autonomous system involves four functional features, namely "perception", "judgment", "decision-making" and "behavior", during operation, forming an OODA functional feedback loop. Aiming at the four-layer structure model of the intelligent autonomous system, namely the perception layer, judgment layer, decision-making layer and action layer, the uncertainty of the intelligent algorithm, and the OODA structure model of the intelligent autonomous system / algorithm, combined with information such as tasks and environment, from the perspectives of "perception layer, judgment layer and decision-making layer", the possible security risks of the OODA structure model of the intelligent autonomous system are analyzed as follows:
[0139] 1) Data has complex characteristics such as multi-dimensional space, multi-source heterogeneity, and confrontational games, which affect the real-time fusion and detection of the "perception layer". The main influencing factors are (including but not limited to):
[0140] Massive data, irregular data, erroneous data, multi-source data, source device status, unbalanced data, adversarial samples, and strong real-time data.
[0141] 2) Behaviors are difficult to predict and change rapidly, which affects the accurate judgment of the "judgment layer". The main influencing factors are (including but not limited to):
[0142] Complex target attributes, intelligent algorithm anomalies (over-learning, over-sensitivity, etc.), system security vulnerabilities (access rights, reserved backdoors, etc.), algorithm model parameters, complex environments, and real-time environmental analysis.
[0143] 3) Complex scenarios have complex characteristics such as "highly complex environment, dynamic and real-time, uncertain boundaries, incomplete information, and strong game confrontation", which affect the accuracy and effectiveness of the "decision-making layer". The main influencing factors are (including but not limited to):
[0144] Intelligent algorithm anomalies (over-learning, over-sensitivity, etc.), system security vulnerabilities (access rights, reserved backdoors, etc.), complex environments, complex task profiles, the effectiveness of training data, and the complexity of test data.
[0145] 4) Intelligent “perception-diagnosis-decision-making” leads to uncertain and unstable output, which affects the robustness and controllability of the “action layer”. The influencing factors mainly include but are not limited to:
[0146] Uncertain and unpredictable output, abnormal output, unstable output changes, and the status of the destination device.
[0147] Failure Mode Analysis Framework for Intelligent Autonomous Systems:
[0148] like Figure 3As shown, this embodiment takes the "thought-body-behavior" of human beings as the idea, is oriented to the application of intelligent autonomous systems, and refers to standards such as ISO 26262, ISO-PAS_21448, CESA 1036, GJB / g 1391, etc., and establishes a fault mode analysis framework based on "intelligent algorithm-hardware composition-environmental scenario" according to the core intelligent algorithm (such as neural network algorithm) of the intelligent autonomous system, software and hardware composition (such as sensors, intelligent autonomous control software, actuators and other software and hardware equipment) and operating environment (such as mountains, plains, forests, rainy days and other complex environments) and scenarios (such as no-fly zones, discontinuous areas and other complex scenarios).
[0149] The intelligent autonomous system is analogous to the behavior of "people", that is, the intelligent algorithm is analogous to the human mind and brain, the hardware and software composition is analogous to the human body, and the environmental scenario is analogous to the external environment and various scenarios in which people are located. Compared with the existing failure mode analysis technology, the failure mode analysis framework based on "intelligent algorithm-hardware composition-environmental scenario" can more fully cover the design elements, abnormal incentives and operation scenarios of the intelligent autonomous system, and guide technical personnel to target the functional architecture of the intelligent autonomous system, and effectively identify various types of complex failure modes such as illegal data, unknown environments, adversarial samples, and algorithm anomalies. It has the advantages and characteristics of solid theoretical foundation, profound and comprehensive, and strong pertinence (combining the characteristics of intelligent autonomous systems with specific tasks and work scenarios for failure mode analysis).
[0150] The framework includes multiple failure mode dimensions for intelligent autonomous systems. The input of the framework includes four aspects: model input, failure criterion input, failure mode input and scenario input.
[0151] Model input: data interaction, state transfer, function execution and function combination
[0152] Data interaction, state transition, function execution, and function combination are inputs in the failure mode analysis process of intelligent autonomous systems.
[0153] Data interaction is the starting point of failure mode analysis, which involves the information exchange between the system and the environment. Through data interaction, the system can obtain information about the external environment and provide basic data for subsequent failure mode analysis.
[0154] State transitions describe how a system changes from one state to another. In failure mode analysis, focusing on the behavior of the system in each state helps identify potential failure modes. When a system transitions from one state to another, there may be changes in functionality or fluctuations in performance, which need to be carefully analyzed.
[0155] Function execution is a key step for a system to achieve a specific task. In failure mode analysis, an in-depth analysis of function execution can reveal possible defects or risks in the function implementation process. These defects may lead to system failure or performance degradation, so the function execution process needs to be strictly controlled.
[0156] Functional combination focuses on the interaction and influence between multiple functions. In an intelligent autonomous system, each function depends on and influences each other, and the failure of any one function may affect the normal operation of other functions. Therefore, in failure mode analysis, it is necessary to comprehensively consider the functional combination to identify possible failure modes and formulate corresponding countermeasures.
[0157] Fault criterion input: fault analysis criteria for intelligent algorithms, fault analysis criteria for software and hardware composition, and analysis criteria for the environment and tasks.
[0158] The fault analysis criteria for intelligent algorithms support system designers to identify and analyze typical failure modes such as training data set failure modes, algorithm training failure modes, algorithm reasoning failure modes, algorithm deployment failure modes, and complex scenario failure modes based on the input data and training process of intelligent autonomous systems.
[0159] The fault analysis criteria for software and hardware components support system designers to identify and analyze system failure modes caused by failure of software and hardware themselves and system failure modes caused by coupling between software and hardware, targeting the software and hardware components of intelligent autonomous systems and their coupling relationships.
[0160] Environment- and task-oriented analysis criteria support system designers to identify and analyze the failure modes of intelligent autonomous systems under specific environments and tasks based on the input data and training process of intelligent autonomous systems.
[0161] Failure Mode Input
[0162] Intelligent autonomous systems need to draw on knowledge and experience in related fields to guide the failure mode analysis process. Relevant knowledge can come from multiple sources such as expert opinions, historical data, technical documents, research papers, etc., providing a deep understanding of system behavior, performance, and potential failure modes, especially existing failure mode libraries. For example, some failure modes in intelligent autonomous systems can refer to the failure modes of traditional software.
[0163] Scene Input
[0164] Scenario input specifically involves clear analysis objects for the environment and tasks, which provide the specific usage and task requirements of the system in the actual operating environment. In different scenarios, the failure modes of intelligent autonomous systems are different. It is very necessary to make a specific analysis of the failure modes of intelligent autonomous systems for specific scenarios.
[0165] Based on model input, fault criterion input, fault mode input, and scenario input, the analysis object is clarified, and the failure mode of the intelligent autonomous system is analyzed from three aspects: intelligent algorithm, software and hardware composition, and environment and task.
[0166] First, the fault mode analysis for intelligent algorithms is carried out. Intelligent algorithms are the core of intelligent autonomous systems. Combined with the algorithm-oriented fault analysis criteria, the fault modes related to the autonomous decision-making of intelligent algorithms are identified. Then, combined with the hardware and software components such as sensors and actuators related to the intelligent autonomous system, the fault modes of the intelligent autonomous system are analyzed from the perspective of hardware and software composition, and the system failure modes caused by hardware failures, software coding errors, and hardware and software coupling conflicts are identified; finally, combined with the operating environment and task scenarios of the intelligent autonomous system, the relevant fault modes are analyzed and identified from the perspectives of environment, task, and use.
[0167] After a detailed analysis of the failure mode, based on the intelligent autonomous system failure mode analysis criteria, combined with complex scenarios, system hardware and software composition, and intelligent algorithm logic, the failure mode and impact analysis is carried out. The analysis process of failure cause analysis, failure mode analysis, failure impact analysis, and control measure acquisition is clarified.
[0168] Criteria for Failure Mode Analysis of Intelligent Autonomous Systems
[0169] This embodiment is based on the typical failure modes of intelligent autonomous systems, combined with the complex scenarios and structural models of intelligent autonomous systems, and formulates the failure mode analysis criteria for intelligent autonomous systems from the perspectives of failure modes such as intelligent algorithms, hardware and software composition, environment and tasks, training data set failure modes, algorithm training failure modes, algorithm reasoning failure modes, algorithm deployment failure modes, intelligent algorithm failure modes for complex scenarios, system failure modes caused by hardware and software failures, system failure modes caused by hardware and software coupling, failure modes caused by complex environments, and failure modes caused by specific tasks. It guides the developers of intelligent autonomous systems to identify various failure modes in intelligent autonomous systems from a black box perspective, and provides knowledge support for the failure analysis process of intelligent autonomous systems driven by criteria.
[0170] Failure Mode Analysis Criteria for Intelligent Algorithms
[0171] It supports system designers to identify and analyze typical failure modes such as training data set failure modes, algorithm training failure modes, algorithm reasoning failure modes, algorithm deployment failure modes, and complex scenario failure modes based on the input data and training process of intelligent autonomous systems.
[0172] (1) Model failure modes caused by data defects
[0173] It supports system designers to identify and analyze "data balance failure mode, data accuracy failure mode, data integrity failure mode" and other failure modes for the training data set of intelligent autonomous systems, which are failure modes that cause the algorithm trained based on the data set to output errors or poor performance due to problems with the data set.
[0174] Model failure modes caused by data imbalance: For the training data set of the intelligent autonomous system, confirm whether the intelligent autonomous system can effectively handle and detect data balance failure modes such as "class imbalanced data set";
[0175] Model failure modes caused by inaccurate data: For the training data set of the intelligent autonomous system, confirm whether the intelligent autonomous system can effectively handle and detect data accuracy failure modes such as "training data errors";
[0176] Model failure modes caused by incomplete data: For the training data set of the intelligent autonomous system, confirm whether the intelligent autonomous system can effectively handle and detect data integrity failure modes such as "small target features";
[0177] (2) Algorithm failure mode
[0178] Support system designers to identify and analyze the failure modes of the intelligent autonomous system during the training process of the intelligent autonomous system, such as "algorithm fitting failure mode, algorithm implementation failure mode, algorithm learning rate setting failure mode, algorithm dependency failure mode", etc., which are caused by various reasons during the algorithm training stage and lead to abnormal output of the intelligent autonomous system.
[0179] Algorithm fitting failure mode: During the training process of the intelligent autonomous system, confirm whether the intelligent autonomous system can effectively handle and detect algorithm fitting failure modes such as "algorithm model overfitting, algorithm model underfitting";
[0180] Algorithm implementation failure modes: During the training process of the intelligent autonomous system, confirm whether the intelligent autonomous system can effectively handle and detect algorithm implementation failure modes such as "algorithm structure change";
[0181] Algorithm-dependent failure modes: During the training process of the intelligent autonomous system, confirm whether the intelligent autonomous system can effectively handle and detect algorithm-dependent failure modes such as "method library incompatibility";
[0182] Algorithm robustness failure mode: During the use of the intelligent autonomous system, confirm whether the intelligent autonomous system can effectively handle and detect algorithm robustness failure modes such as "the impact of various occlusion situations on visual tasks";
[0183] Algorithm performance failure mode: During the use of the intelligent autonomous system, confirm whether the intelligent autonomous system can effectively handle and detect algorithm performance failure modes such as "the algorithm accuracy drops too much";
[0184] Algorithm hardware-dependent failure modes: During the deployment process of the intelligent autonomous system, confirm whether the intelligent autonomous system can effectively handle and detect algorithm hardware-dependent failure modes such as "changes in hardware attributes leading to reduced model accuracy";
[0185] Algorithm deployment failure modes: During the deployment process of the intelligent autonomous system, confirm whether the intelligent autonomous system can effectively handle and detect algorithm deployment failure modes such as "algorithm framework migration leading to model incompatibility";
[0186] Adversarial training sample failure modes: For the training samples used by the intelligent autonomous system, confirm whether the intelligent autonomous system can effectively handle and detect adversarial training sample failure modes such as "adversarial samples, imbalanced samples, contaminated samples, and unlabeled samples".
[0187] Failure mode analysis criteria for hardware and software components:
[0188] Support system designers to identify and analyze system failure modes caused by failure of software and hardware themselves and system failure modes caused by coupling between software and hardware, targeting the software and hardware components of intelligent autonomous systems and their coupling relationships.
[0189] (1) System failure mode analysis criteria caused by hardware and software failures
[0190] Support system designers to identify and analyze system failure modes caused by failure of software and hardware themselves, such as "failure modes caused by failure of software items and failure modes caused by failure of hardware items" for the software and hardware components of intelligent autonomous systems and their coupling relationships:
[0191] Failure modes caused by software failure: Analyze the status of the software components of the intelligent autonomous system to confirm whether the intelligent autonomous system can effectively handle and detect software failure modes such as "abnormal data definition and assignment, data calculation and processing errors, overfitting of the objective function, underfitting of the objective function, difficulty in model convergence, invalid logical processing of the optimization target quantity, abnormal function use, abnormal file processing, stack overflow vulnerability, buffer overflow, program memory and pointer usage errors, array out of bounds, interruption and multi-threading processing";
[0192] Failure modes caused by hardware failure: Analyze the status of the hardware components of the intelligent autonomous system to confirm whether the intelligent autonomous system can effectively handle and detect hardware failure modes such as "sensors, actuators, computing platforms, communication networks, etc. are faulty or in abnormal states"
[0193] (2) System failure mode analysis criteria caused by coupling between software and hardware
[0194] Support system designers to identify and analyze system failure modes caused by failure of software and hardware themselves, such as "data coupling conflict failure mode, control coupling conflict failure mode, hardware architecture abnormal failure mode", etc., based on the software and hardware components of intelligent autonomous systems and their coupling relationships:
[0195] Data coupling failure mode: Analyze the software and hardware coupling relationship of the intelligent autonomous system to confirm whether the intelligent autonomous system can effectively handle and detect failure modes such as "data access conflicts or errors";
[0196] Control coupling failure mode: Analyze the software and hardware coupling relationship of the intelligent autonomous system to confirm whether the intelligent autonomous system can effectively handle and detect failure modes such as "actuator in abnormal state such as occupied or faulty";
[0197] Dynamic real-time constraint failure mode: for the real-time constraint relationship such as the response time of the intelligent autonomous system to the input data, confirm whether the intelligent autonomous system can effectively handle and detect the dynamic real-time constraint modes such as "response time delay, training timeout convergence";
[0198] The failure mode analysis criteria for hardware and software components are shown in Table 1.
[0199] Table 1
[0200]
[0201]
[0202]
[0203]
[0204] Environment- and mission-oriented failure mode analysis criteria
[0205] Support system designers to identify and analyze the failure modes of intelligent autonomous systems under specific environments and tasks based on the input data and training process of intelligent autonomous systems.
[0206] Support system designers to identify and analyze the failure modes of intelligent autonomous systems under specific tasks such as "specific target identification, specific route planning":
[0207] Target recognition failure mode: Analyze the input data and training process of the intelligent autonomous system to confirm whether the intelligent autonomous system can effectively detect and process specific targets such as "unrecognizable scene signs, targets of unknown categories, disappeared targets, and targets of unknown friend or foe attributes."
[0208] Route planning failure modes: Analyze the input data and training process of the intelligent autonomous system to confirm whether the intelligent autonomous system can effectively detect and process specific routes such as "unrecognizable routes, surrounded by multiple obstacles, in discontinuous boundary areas, and changes in area boundaries."
[0209] Decision-making and reasoning failure modes: Analyze the decision-making process of the intelligent autonomous system to confirm whether the intelligent autonomous system can effectively detect and handle specific task failures such as "decision logic errors, reasoning logic errors", etc.
[0210] The failure mode analysis criteria for environment and task are shown in Table 2
[0211] Table 2
[0212]
[0213]
[0214] Failure Mode and Impact Analysis Process of Intelligent Autonomous Systems
[0215] Combined with the intelligent autonomous system failure mode analysis criteria, corresponding inspection items are formulated from different angles to form an operational compliance checklist, covering the main design elements of the intelligent autonomous system. Several checklists are prepared for different categories of failure mode criteria such as "intelligent algorithm-oriented, hardware and software composition-oriented, environment and task-oriented".
[0216] Based on the above-mentioned different types of checklists, support the intelligent autonomous system designers to check various design elements and identify the hidden dangers of non-compliance between the intelligent autonomous system and the intelligent autonomous system failure mode analysis criteria. Based on the compliance checklist, the inspection process of the intelligent autonomous system is as follows:
[0217] According to various failure analysis criteria, conduct coverage analysis and inspection on the required design elements such as intelligent algorithm training samples, input data, learning process, output data, etc.;
[0218] If the intelligent algorithm requirement elements fully implement the criterion, fill in "YES" in the "Compliance Check" column. And in the "Description" column, describe the relevant content implemented by the criterion;
[0219] If the criterion is not applicable to the current intelligent algorithm requirement elements, fill in "N / A" in the "Compliance Check" column. In the "Description" column, describe the reasons why the criterion is not applicable and the related design content that is not applicable;
[0220] If the intelligent algorithm requirement element does not implement this criterion, fill in "NO" in the "Compliance Check" column. In the "Description" column, describe the relevant requirement design chapter content that does not implement this criterion. Further analysis is conducted to clarify the possible problems that may arise when the requirements do not meet the analysis criteria.
[0221] Based on the intelligent algorithm failure analysis criteria, combined with complex scenarios and algorithm logic, conduct failure mode and impact analysis. That is, establish an intelligent algorithm failure mode and impact analysis framework, clarify the analysis process and results of failure cause analysis, failure mode analysis, failure impact analysis, control measures / safety requirements acquisition, etc. Support developers to conduct systematic traversal analysis around various complex scenarios to ensure the standardization and adequacy of the analysis process. Figure 4 :
[0222] The process of analyzing the failure modes and their effects of intelligent autonomous systems can be divided into the following steps:
[0223] Identify the cause of input failure: Identify and analyze the cause of the failure. These causes may come from the input end of the system, such as sensor data errors, external command abnormalities, etc. Failures may also come from errors that occur during the system processing, typical examples are algorithm errors or software vulnerabilities. Failures can also come from the output end. For example, in the output link of the control system, if the actuator (such as a motor or valve) fails, the system output instructions cannot be correctly executed.
[0224] Analyze the impact of the failure: Once the cause of the failure is determined, it is necessary to evaluate the impact of the failure on the system output. Output failures may manifest as system performance degradation, data errors, or complete failure. By analyzing the output failure, you can preliminarily determine the extent of the failure's impact on the overall function of the system. When analyzing the impact of the failure, you also need to pay attention to the transmission process of the failure within the system. If the failure is not identified and handled in a timely manner, it may be transmitted to other components or devices through the interaction and communication mechanisms within the system, thereby expanding the scope of the failure. Therefore, it is necessary to analyze the path and method of the failure's propagation within the system.
[0225] Implement fault control measures: Based on the fault analysis results and the fault transmission process within the system, take appropriate control measures to prevent the fault from further expanding or affecting other systems. This may include isolating the fault source, starting the backup system, correcting erroneous data, etc. By implementing these control measures, the impact of the fault on the normal operation of the system can be effectively reduced.
[0226] Summary and improvement: After the fault is handled, it is necessary to summarize the entire process and analyze the cause of the fault and the effectiveness of the handling measures in order to provide experience and lessons for future fault prevention and handling.
[0227] For each failure mode, analyze and determine its root cause. This requires a deep understanding of the input, processing, etc. There are several ways to analyze the cause of the failure:
[0228] Root Cause Analysis (RCA): Root Cause Analysis is a systematic problem-solving method that aims to find the root cause of the problem, not just the superficial phenomenon or direct cause. Through in-depth analysis and exploration, RCA can help us find the real source of the problem and develop effective solutions and preventive measures.
[0229] Fault Tree Analysis (FTA): Although FTA is mainly used to identify and analyze failure modes, it can also be used to evaluate the causes of failures. By constructing a logical tree diagram, FTA shows all possible causes that lead to a top event (i.e., the most undesirable failure).
[0230] Cause and Effect Diagram (Ishikawa Diagram): A cause and effect diagram, also known as a fishbone diagram or Ishikawa diagram, is used to visually display the potential causes of a problem. A cause and effect diagram can be used to identify a failure mode as a "problem" and then list and categorize its possible causes.
[0231] Expert review and group discussion: By organizing experts or team members in related fields to discuss and review, their knowledge and experience can be used to identify and analyze the cause of the failure. This method can brainstorm and discover causes that may have been overlooked.
[0232] Data analysis and statistical methods: If sufficient historical data or test data is collected, statistical methods can be used to analyze the cause of the failure. For example, regression analysis, variance analysis, hypothesis testing, etc. can be used to identify factors related to the occurrence of failures.
[0233] Simulation and testing: By simulating the failure scenario or conducting actual tests, the causes of the failure can be observed and analyzed. This method can simulate various conditions in the real environment to discover possible causes.
[0234] Fault reproduction: In some cases, you can try to reproduce the conditions and environment in which the fault occurred in order to observe and analyze the cause of the fault. This requires carefully recording all relevant parameters and conditions when the fault occurred and attempting to reproduce these conditions in a controlled environment.
[0235] The impact analysis of each failure mode is carried out, and the severity of the impact of the intelligent algorithm failure mode is described from the perspectives of operational safety and task completion, and the severity of the failure mode is quantitatively characterized in a graded form. The impact assessment of the failure from the perspective of "operational safety" should focus on the impact of the intelligent algorithm failure on system operational safety, equipment, personnel and other factors; while the impact assessment of the failure from the perspective of "task completion" should focus on the impact of the intelligent algorithm failure on the performance quality, real-time and cost of task completion.
[0236] With reference to GJB / Z 102A and other standards, the intelligent algorithm fault impact severity classification table is shown in Table 3:
[0237] Table 3
[0238]
[0239]
[0240] It should be noted that:
[0241] According to the above table, the severity level of intelligent algorithm failure is divided into 5 levels, namely catastrophic, serious, severe, mild and no impact, represented by letters a to e respectively. The higher the letter, the higher the severity level.
[0242] According to the results of the intelligent algorithm failure mode analysis, the severity level of the intelligent algorithm failure is determined by comparing the contents of the above table. As long as the consequences of the failure meet the description of any aspect in the above table (i.e., operational safety impact or task completion impact), the severity of the failure is determined to be the corresponding level.
[0243] If an intelligent algorithm failure will cause multiple system function failures, and multiple system function failures will jointly cause a higher level of system hazard, the severity level of the intelligent algorithm failure should be determined based on the highest level of system hazard.
[0244] Then, analyze the control or mitigation capabilities of the intelligent autonomous system when various algorithm failures occur, and formulate the evaluation basis and requirements of the "control capability level" from the perspectives of "complete control, mitigation control, and inability to intervene". The implementation process is as follows:
[0245] The control level of intelligent autonomous system over intelligent algorithm failure, the control level of intelligent algorithm over system function, and algorithm redundancy backup should be considered, and the control level of intelligent autonomous system over algorithm failure should be evaluated accordingly. With reference to standards such as MIL-STD-882E, the proposed intelligent autonomous system control capability level table is shown in Table 4:
[0246] Table 4
[0247]
[0248]
[0249] The contents that need to be explained include:
[0250] This guideline divides the control capability of intelligent autonomous systems into four levels, represented by Latin letters I-IV
[0251] Describe the ability of intelligent algorithms to control dangers from three aspects: the degree of control of intelligent algorithms over system / subsystem dangers, the degree of control of intelligent algorithms over system / subsystem functions, and redundancy backup measures;
[0252] Compare the control capability of the intelligent algorithm with the control level description above to determine the control capability level. If the same intelligent algorithm may correspond to multiple control levels, the highest level should be taken as the control level of the intelligent algorithm;
[0253] Among them, control capability level I means that the intelligent algorithm can fully control the system / subsystem hazards; control capability levels II and III mean that the intelligent algorithm can mitigate and control the system / subsystem hazards; control capability level IV means that the intelligent algorithm cannot intervene in the system / subsystem hazards;
[0254] Finally, the "impact consequence level and control capability level" of the integrated intelligent autonomous system are used to achieve a comprehensive assessment of the security risk level of the intelligent algorithm with the help of a multi-dimensional evaluation matrix. An index matrix of the security risk level of the intelligent algorithm is established, as shown in Table 5. The severity of the impact of the failure of the intelligent algorithm is used as a row, and the control capability level of the intelligent algorithm is used as a column. A discrete number is assigned to each cell to represent the index value of the security risk level of the intelligent algorithm. The smaller the index value, the higher the security risk level of the intelligent algorithm.
[0255] Table 5
[0256]
[0257] The index values in the intelligent algorithm security risk level index matrix can be segmented according to the actual research situation of the intelligent autonomous system, so as to divide the intelligent algorithm security risks into several levels, and determine the content of the security testing and evaluation work that should be carried out for the intelligent algorithm.
[0258] Table 7 lists the security risk levels of intelligent algorithms determined by segmentation based on index values. The security risk levels of intelligent algorithms are divided into 5 levels, represented by letters A to E. The higher the letter, the higher the security risk level of the intelligent algorithm, and the more complete the security test and evaluation work that should be completed.
[0259] Table 7
[0260]
[0261] This embodiment not only provides designers with a theoretical basis for fully understanding and identifying various fault conditions that intelligent autonomous systems may encounter, but also provides a practical analysis method to help them foresee and prevent potential problems during the design and development phase.
[0262] As the forefront of current technological development, the complexity and autonomy of intelligent autonomous systems make the analysis of failure modes particularly complex and critical. This embodiment explains the relevant concepts of failure modes of intelligent autonomous systems, hoping that designers can establish a clear understanding of failure modes. At the same time, this embodiment also deeply analyzes the functional and technical characteristics of intelligent autonomous systems to help designers understand the root causes of failures based on the internal mechanisms of the system. This embodiment also emphasizes the influencing factors of intelligent autonomous system failures. Whether it is environmental factors, human factors or design defects of the system itself, they may affect the normal operation of intelligent autonomous systems.
[0263] In addition to the theoretical in-depth discussion of the failure mode of the intelligent autonomous system, this embodiment further provides a set of systematic analysis frameworks and criteria to help designers conduct a comprehensive and in-depth analysis of the failure mode from multiple dimensions. The purpose of this framework and criteria is to enable designers to have rules to follow when conducting failure mode analysis, avoid missing key information, and ensure the accuracy and completeness of the analysis. The analysis framework provides designers with a set of structured analysis steps. From determining the analysis object, to collecting relevant data and information, to using different analysis methods and techniques to analyze the failure mode one by one, and finally forming a detailed analysis report. The analysis framework also emphasizes the comprehensive consideration of multiple dimensions. When conducting failure mode analysis, designers need to consider multiple aspects such as intelligent algorithms, software and hardware composition, environment and tasks. In addition, the analysis framework also includes a series of specific analysis criteria. These criteria cover the classification of failure modes and provide clear guidance for designers. By following these criteria, designers can analyze the failure mode of intelligent autonomous systems more objectively and accurately.
[0264] In the process of failure mode and its impact analysis, this embodiment explains how to analyze the failure causes and impacts of intelligent autonomous systems, and how to formulate corresponding countermeasures. Through this process, designers can have a deeper understanding of the impact of failures on system performance and safety, and thus formulate more effective failure prevention and response strategies.
[0265] It is worth mentioning that with the continuous advancement of technology and applications, the failure modes of intelligent autonomous systems may also continue to change and increase. Therefore, this embodiment emphasizes the importance of continuous failure mode analysis and improvement. Only by constantly following the development of technology and continuously improving analysis methods and tools can the continuous and reliable operation of intelligent autonomous systems be ensured.
[0266] In general, this paper provides a more comprehensive analysis framework for designers. Whether for beginners or experienced practitioners, this paper can provide valuable information and guidance to help them better deal with various failure situations that intelligent autonomous systems may encounter and ensure that the system's safety, reliability and performance reach a high level. It is hoped that more designers, researchers and practitioners can study and apply this guide in depth to jointly promote the healthy development of intelligent autonomous systems.
[0267] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the relevant field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents. Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.
Claims
1. A fault pattern recognition and analysis method for an intelligent autonomous system, characterized in that: include: Construct the intelligent autonomous system failure mode analysis framework and intelligent autonomous system failure mode analysis criteria respectively; The intelligent autonomous system failure mode analysis framework performs fault identification, recording and evaluation of the intelligent autonomous system based on the intelligent autonomous system failure mode analysis criteria, and generates a compliance checklist; A security risk level of the intelligent autonomous system is obtained based on the compliance checklist.
2. The fault pattern recognition and analysis method of the intelligent autonomous system according to claim 1 is characterized in that: The construction process of the intelligent autonomous system failure mode analysis framework includes: Analyze the functional and technical characteristics of intelligent autonomous systems; Obtaining failure influencing factors based on the functional characteristics and the technical characteristics; Based on the fault influencing factors, the fault mode analysis framework of the intelligent autonomous system is constructed according to the core intelligent algorithm, software and hardware composition, operating environment and scenarios of the intelligent autonomous system.
3. The fault pattern recognition and analysis method of the intelligent autonomous system according to claim 2 is characterized in that: The failure influencing factors include software and hardware failure factors, intelligent algorithm failure factors, training sample failure factors, and OODA task chain failure factors; The software and hardware failure factors include: open source component reliability risk, algorithm framework reliability risk, application software security risk, human-computer interaction security risk, hardware environment reliability risk, software and hardware coupling reliability risk, and system behavior reliability risk; The intelligent algorithm failure factors include: model accuracy and robustness, model complexity, model hyperparameters, algorithm performance, algorithm bias or discrimination, and algorithm nonlinear characteristics; The training sample failure factors include: data set reliability, training sample diversity, training sample annotation quality, training sample balance, training sample preprocessing quality, data annotation security compliance, and automated decision security compliance; The OODA task chain failure factors include: the impact of data characteristics on the perception layer, the impact of behavior changes on the judgment layer, the impact of scenario complexity on the judgment layer, and the impact of output instability on the action layer.
4. The fault pattern recognition and analysis method of the intelligent autonomous system according to claim 1, characterized in that: The construction process of the intelligent autonomous system failure mode analysis criterion includes: Obtaining failure modes for input data and training processes of intelligent autonomous systems; The intelligent autonomous system failure mode analysis criterion is constructed based on the failure mode.
5. The fault pattern recognition and analysis method of the intelligent autonomous system according to claim 4 is characterized in that: The failure modes include intelligent algorithm failure mode, software and hardware failure mode, and environment and task failure mode; The intelligent algorithm failure modes include data defect failure modes and algorithm failure modes; the data defect failure modes include data imbalance, data inaccuracy, and data incompleteness; The algorithm failure modes include algorithm fitting failure, algorithm implementation failure, algorithm robustness failure, algorithm performance failure, algorithm hardware dependency failure, algorithm deployment failure; adversarial training sample failure; The software and hardware fault modes include software and hardware failure fault modes and software and hardware coupling fault modes; the software and hardware failure fault modes include software item failure fault and hardware item failure fault; The software and hardware coupling failure modes include data coupling failure, control coupling failure, and dynamic real-time constraint failure; The environment and task failure modes include target recognition failure mode, route planning failure mode, and decision reasoning failure mode.
6. The fault pattern recognition and analysis method of the intelligent autonomous system according to claim 1, characterized in that: The intelligent autonomous system failure mode analysis framework is based on model input, failure criterion input, failure mode input and scenario input. It analyzes the failure mode of the intelligent autonomous system from the aspects of intelligent algorithm, software and hardware composition, environment and task, and on the basis of the intelligent autonomous system failure mode analysis criterion, obtains the failure cause, failure mode, failure impact and control measures, and generates the compliance checklist.
7. The fault pattern recognition and analysis method of an intelligent autonomous system according to claim 1, characterized in that: The compliance checklist includes a compliance checklist for intelligent algorithms, a compliance checklist for software and hardware composition, and a compliance checklist for the environment and tasks.
8. The fault pattern recognition and analysis method of an intelligent autonomous system according to claim 1, characterized in that: The process of obtaining the security risk level of the intelligent autonomous system based on the compliance checklist includes: Obtain failure analysis criteria based on the compliance checklist, and perform coverage analysis and inspection on the demand design elements of the intelligent algorithm; Identify the cause of input failure, evaluate the impact of the failure on the system output in a graded manner, generate a severity classification table for the impact of intelligent autonomous system failures, and obtain the impact consequence level; Obtaining the fault propagation path and propagation mode, and implementing fault control measures based on the impact consequence level, the fault propagation path and propagation mode; Conducting effectiveness analysis on the fault cause and the fault control measures; Analyze the control or mitigation capabilities of the intelligent autonomous system when algorithm failures occur, generate a control capability rating table for the intelligent autonomous system, and obtain the control capability rating of the intelligent autonomous system; Based on the impact consequence level and the control capability level, a multi-dimensional evaluation matrix is used to construct an intelligent algorithm security risk level index matrix, and the security risk level is obtained based on the intelligent algorithm security risk level index matrix.
9. The fault pattern recognition and analysis method of the intelligent autonomous system according to claim 8, characterized in that: The method for identifying the cause of the fault includes: root cause analysis, fault tree analysis, cause-effect diagram, expert review and group discussion, data analysis and statistical methods, simulation and testing, and fault reproduction.
10. The fault pattern recognition and analysis method of the intelligent autonomous system according to claim 8, characterized in that: The table for grading the severity of the autonomous system failures divides the severity of the intelligent algorithm failures into five levels. If there are multiple system function failures that jointly cause a higher level of system hazard, the severity level of the failure is determined according to the highest level of system hazard. The intelligent autonomous system control capability level table divides the control capability level of the intelligent autonomous system into four levels. If the same intelligent algorithm corresponds to multiple control levels, the highest level is taken as the control level of the intelligent algorithm.
Citation Information
Cited By
Intelligent scene linkage control platform and equipment
CN120029124A
