A method and apparatus for resilient testing of unmanned systems to address their differences

Abstract

This application provides a resilient testing method and apparatus for unmanned systems, relating to the field of artificial intelligence technology. Step 1: Input the initial test cases of the proxy prediction model corresponding to the actual system under test into the criticality prediction model to determine the criticality score of the initial test cases; Step 2: Determine key test cases from the initial test cases based on the criticality score and criticality threshold, and predict test deviations based on the actual test cases and key test cases of the actual system under test; Step 3: Correct the criticality prediction model based on the test deviations while collecting actual test cases online to generate an updated criticality prediction model; Step 4: Repeat steps 1-3 using the updated criticality prediction model according to the preset maximum number of tests to determine the final key test cases. This improves the resilience of intelligent testing technology in testing different models of unmanned systems and enables rapid optimization and adaptation of the test case library.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, for example to a resilient testing method and apparatus for unmanned systems to address their differences. Background Technology

[0002] Unmanned system testing refers to testing and verifying the functions and performance of unmanned systems (such as unmanned vehicles, drones, and unmanned ships) to determine whether they have the ability to autonomously complete tasks without human control. Unlike traditional systems, intelligent unmanned systems are characterized by a wide range of applications, diverse needs, complex tasks, and unclear behavioral mechanisms. Testing unmanned systems faces challenges such as unknown system structure, difficulty in obtaining operational status, and high-dimensional test case space. Using traditional testing techniques leads to low testing efficiency, high testing costs, and difficulty in reaching capability boundaries, severely hindering the iterative optimization and application process of unmanned systems. Intelligent testing of unmanned systems involves introducing artificial intelligence technology to generate test case libraries, enabling efficient searching of representative key test case libraries within an ultra-high-dimensional test case space. This accelerates the understanding of the capability boundaries of unmanned systems and ultimately addresses the challenge that traditional testing methods cannot meet the testing needs of intelligent unmanned systems. In essence, the core idea of ​​intelligent testing is "AI (Artificial Intelligence) testing AI." On one hand, the tested object (i.e., the intelligent unmanned system) is a system with a high level of intelligence; on the other hand, the testing methods also possess a high level of intelligence. The "AI-to-AI" testing technology has initially demonstrated its advantages and potential in the testing and verification of intelligent unmanned systems. However, to enhance the practical application value of intelligent testing, further in-depth research is needed on related fundamental theories and key technologies. As a crucial link in the development and practical application of unmanned systems, the testing of intelligent unmanned systems faces challenges such as unknown system structure, difficulty in obtaining operational status, and high dimensionality of test case space. Traditional testing techniques lead to low testing efficiency, high testing costs, and insufficient assessment of key performance indicators, severely hindering the iterative optimization and application process of unmanned systems. Intelligent testing technology based on the core concept of "AI-to-AI" has begun to show its advantages, demonstrating great potential in accelerating the understanding of the capability boundaries of unmanned systems and reducing testing costs.

[0003] In related technologies, intelligent testing heavily relies on the scale and quality of training data, making the generation of test case libraries using surrogate prediction models of unmanned systems a crucial means of implementing intelligent testing. However, due to the differences between surrogate prediction models and the actual systems under test, offline-generated test case libraries suffer from poor universality and are difficult to apply effectively to the testing and verification of unmanned systems with different mechanisms and performance characteristics, severely limiting the rapid promotion and application of intelligent testing technology. Therefore, it is urgent to improve the flexibility of intelligent testing technology in testing different types of unmanned systems and to achieve rapid optimization and adaptation of test case libraries. Summary of the Invention

[0004] This application aims to provide a method and apparatus for resilient testing of the differences in unmanned systems.

[0005] According to one aspect of this application, a resilient testing method for unmanned systems is proposed, comprising: Step 1: Inputting initial test cases of the proxy prediction model corresponding to the actual system under test into a preset criticality prediction model to determine the criticality score of the initial test cases; Step 2: Based on the criticality score and a preset criticality threshold, determining critical test cases from the initial test cases, and predicting test deviations based on the actual test cases and critical test cases of the actual system under test; Step 3: With the actual test cases collected online, correcting the criticality prediction model based on the test deviations to generate an updated criticality prediction model; Step 4: Repeating steps 1-3 using the updated criticality prediction model according to a preset maximum number of tests, and determining the final critical test cases.

[0006] According to one aspect of this application, a resilient testing device for the differences in unmanned systems is proposed, comprising:

[0007] The score determination module is used to input the initial test cases of the proxy prediction model corresponding to the actual system under test into the preset criticality prediction model in order to determine the criticality score of the initial test cases. The test deviation determination module is used to identify critical test cases from the initial test cases based on the criticality score and the preset criticality threshold, and to predict test deviations based on the actual test cases and critical test cases of the system under test. The model correction module is used to correct the criticality prediction model based on test deviations when real test cases are collected online, so as to generate an updated criticality prediction model. The iterative test correction module is used to repeatedly call the score determination module, test deviation determination module, and model correction module according to the preset maximum number of tests, and determine the final critical test cases.

[0008] According to one aspect of this application, an electronic device is provided, comprising: a processor; and a memory storing a computer program that, when executed by the processor, causes the processor to perform the method described above.

[0009] According to one aspect of this application, a non-transitory computer-readable medium is proposed, on which readable instructions are stored, which, when executed by a processor, cause the processor to perform the method described above.

[0010] It should be understood that the above general description and the following detailed description are merely exemplary and do not limit this application.

[0011] Beneficial effects: The embodiments provided in this application introduce an intelligent elastic testing mechanism. By constructing a dynamic evaluation system combining a proxy prediction model and a criticality prediction model, the differences between the unmanned system proxy model and the actual system are effectively quantified, providing a measurable technical path to solve the problem of poor test case universality. Based on an adaptive correction framework using active learning, test deviations can be detected and corrected in real time during the online collection of real test cases, significantly improving the rapid optimization and adaptation capabilities of the test case library under resource-constrained conditions, and realizing a leap from static preset to dynamic elasticity in testing strategies. Addressing the curse of dimensionality caused by the high-dimensional parameter space of unmanned systems, deep learning technology is innovatively integrated. A criticality prediction model is used to intelligently filter and prioritize massive test cases, breaking through the limitations of traditional methods in handling high-dimensional data in complex scenarios, and significantly improving testing efficiency and coverage. A closed-loop optimization system of "AI testing AI" is constructed. Through multiple iterations of model correction and deviation convergence mechanisms, the differences between the proxy model and the actual system are continuously reduced, resulting in a more accurate test case library. Attached Figure Description

[0012] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings, without exceeding the scope of protection claimed by this application.

[0013] Figure 1 A schematic diagram of the framework for intelligent resilience testing provided in the embodiments of this application; Figure 2 A flowchart of a resilience testing method for unmanned systems based on the differences provided in this application embodiment; Figure 3 A schematic diagram illustrating the structural composition of a multilayer perceptron in the proxy prediction model provided in this application embodiment; Figure 4 A block diagram of a resilience testing device for the differences in unmanned systems provided in this application embodiment; Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0014] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the embodiments set forth herein; rather, they are provided so that this application will be thorough and complete, and will fully convey the concept of the exemplary embodiments to those skilled in the art. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted.

[0015] Furthermore, the described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. Numerous specific details are provided in the following description to give a thorough understanding of embodiments of this application. However, those skilled in the art will recognize that the technical solutions of this application can be practiced without one or more of the specific details, or other methods, components, apparatuses, steps, etc., can be employed. In other instances, well-known methods, apparatuses, implementations, or operations are not shown or described in detail to avoid obscuring various aspects of this application.

[0016] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0017] The flowcharts shown in the accompanying drawings are merely illustrative and do not necessarily include all content and operations / steps, nor do they necessarily have to be performed in the described order. For example, some operations / steps can be broken down, while others can be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.

[0018] It should be understood that although the terms first, second, third, etc., may be used herein to describe various components, these components should not be limited by these terms. These terms are used to distinguish one component from another. Therefore, the first component discussed below may be referred to as the second component without departing from the teachings of this application. As used herein, the term "and / or" includes all combinations of any one and more of the associated listed items.

[0019] The intelligent testing method proposed in this application is a resilient testing approach, aiming to enhance the resilience of intelligent testing technology in testing different types of unmanned systems and address the problem of poor versatility in intelligent testing technology for unmanned systems. Specifically, AI-based testing technologies heavily rely on the scale and quality of training data, making the generation of test case libraries using unmanned system proxy prediction models a crucial means of implementing "AI testing AI." However, due to differences between proxy prediction models and the actual systems under test, test case libraries generated using proxy prediction models suffer from poor versatility, making them difficult to apply effectively to the testing and verification of unmanned systems with different mechanisms and performance characteristics, severely limiting the rapid promotion and application of intelligent testing technology. Based on a test case library built offline using proxy prediction models, intelligent resilient testing technology aims to optimize and adapt the test case library online according to the characteristics of the real tested objects, significantly enhancing the practical application value of intelligent testing technology.

[0020] Based on this, this application provides a method and apparatus for resilient testing of the differences in unmanned systems.

[0021] Figure 1 This is a schematic diagram of the intelligent resilience testing framework provided in this application embodiment. In this application, a surrogate model (surrogate prediction model) can be used to train a test case criticality prediction model (hereinafter referred to as the criticality prediction model). Appropriate test cases, i.e., critical test cases, are determined using a sampling strategy and the criticality score output by the criticality prediction model. These critical test cases are then executed on the actual system under test, and the resulting test data is used to predict the criticality prediction deviation of the test cases. Furthermore, the criticality prediction model is corrected online, achieving online optimization.

[0022] For specific implementation details, please refer to the following examples.

[0023] Figure 2 A flowchart illustrating a resilience testing method for unmanned systems tailored to the differences provided in this application. Figure 2 As shown, the method includes steps S20, S21, S22 and S23.

[0024] In step S20, the initial test cases of the proxy prediction model corresponding to the actual system under test are input into the preset criticality prediction model to determine the criticality score of the initial test cases.

[0025] In this application, the surrogate prediction model is generally a different model that has a similar or identical working environment to the actual system under test. It can be a "stand-in" or "twin," or it can be another algorithm model (with greater deviation) in the same working environment. Regardless of the type, it faces the impact of deviation from the actual object under test.

[0026] Initial test cases can be data pairs from multiple experimental tests conducted on the proxy prediction model, including test parameters and success or failure labels for the final test task. The parameters and labels together form a single initial test case. A criticality prediction model can be pre-built, and the initial test cases can be input into this model to directly output the criticality score for each initial test case.

[0027] In some implementations, the surrogate prediction model can be obtained through a data-driven approach using deep learning. The surrogate prediction model can be understood as a keyness prediction function that maps a high-dimensional parameter space S to the interval [0,1]. However, due to the complexity of the application environment, the lack of understanding of the internal mechanisms of the tested object, and the difficulty in predicting the execution performance of the tested object under a certain test case, obtaining the keyness prediction function is challenging. Specifically, this application uses a deep learning-based approach to train a surrogate prediction model based on an artificial neural network.

[0028] An artificial neural network is constructed using gradient backpropagation training. Multi-Layer Perceptrons (MLPs) are a fundamental algorithm in artificial neural networks, capable of fitting functions through multiple fully connected layers and nonlinear activation functions. Using backpropagation, MLPs can learn information from a sufficient amount of input and output data and fit a mapping function from input to output. The surrogate prediction model used in this application consists of multiple MLPs, using ReLU as the activation function for intermediate layers and a Sigmoid function to normalize the output layer output to [0,1]. The structure of the MLP in the surrogate prediction model can be described as follows: Figure 3 As shown.

[0029] The training loss uses binary cross-entropy loss, and gradient descent is performed using Adam (Adaptive Moment Estimation Optimizer). Due to the sparsity of key test cases, the proportion of failed test cases is extremely small. The resulting class imbalance problem can severely affect the training of the multilayer perceptron, causing the multilayer perceptron to be more inclined to predict test cases as negative samples and fail to learn the mapping relationship between input and output. This patent adopts a class-balanced sampling strategy. Specifically, during the training of the multilayer perceptron, multiple training samples are combined into a training batch to uniformly calculate the loss function and perform gradient propagation. When constructing the training batch, the sampling probabilities of positive and negative samples are adjusted so that the ratio of positive to negative samples in each batch is close to 1:1. Specifically, assuming the ratio of the number of positive to negative samples is a:b, the sampling probability ratio of each positive to negative sample is b:a.

[0030] In step S21, critical test cases are determined from the initial test cases based on the criticality score and the preset criticality threshold, and test deviations are predicted based on the actual test cases and critical test cases of the system under test.

[0031] In this application, the selection method for key test cases, which is affected by the comparison between key scores and key thresholds, can be preset. Based on the selection method, the current key score, and the preset key threshold, key test cases are selected from the initial test cases. The key threshold can be set to 0.5, but in some implementations, based on the idea of ​​binary classification in deep learning, the threshold can be set arbitrarily. A higher threshold indicates stricter requirements, which may filter out too much data, but the prediction accuracy is generally higher.

[0032] This application can also pre-set the prediction method for test deviations. First, it obtains the actual test cases of the system under test, i.e., the actual system under test runs according to the test parameters corresponding to the key test cases and generates corresponding labels. In some implementations, test deviations can be characterized by other features, such as recall and precision. This application can, according to the prediction method, substitute the actual test cases and key test cases to obtain parameters used to predict test deviations, and the correspondence between these parameters and test deviations can be pre-set.

[0033] In step S22, with real test cases collected online, the criticality prediction model is corrected based on test deviations to generate an updated criticality prediction model.

[0034] In this application, real test cases may be collected online or offline. For online collection, test biases can be used to select and correct parameters or functions to refine the key score prediction model, resulting in an updated model.

[0035] In some implementations, the offline state is not considered for correction in this method. That is, the offline state can be regarded as the criticality prediction model that will not be adjusted according to real-time test feedback throughout the entire testing process.

[0036] In step S23, steps S20-S22 are repeated using the updated criticality prediction model according to the preset maximum number of tests, and the final critical test cases are determined.

[0037] This application allows you to pre-set the maximum number of tests according to actual requirements. Initial test cases are input into a new criticality prediction model, which outputs the corresponding criticality scores. Then, the test deviation is predicted again, and this process continues until the maximum number of tests has been reached, at which point the final critical test cases are obtained.

[0038] This application introduces an intelligent elastic testing mechanism. By constructing a dynamic evaluation system combining a proxy prediction model and a criticality prediction model, it effectively quantifies the differences between the unmanned system proxy model and the actual system, providing a measurable technical path to solve the problem of poor test case universality. Based on an active learning-based adaptive correction framework, it can detect and correct test deviations in real time during the online collection of real test cases, significantly improving the rapid optimization and adaptation capabilities of the test case library under resource-constrained conditions, and realizing a leap from static preset to dynamic elasticity in testing strategies. Addressing the curse of dimensionality caused by the high-dimensional parameter space of unmanned systems, it innovatively integrates deep learning technology, using a criticality prediction model to intelligently filter and prioritize massive test cases, breaking through the limitations of traditional methods in handling high-dimensional data in complex scenarios, and significantly improving testing efficiency and coverage. A closed-loop optimization system of "AI testing AI" is constructed. Through multiple iterations of model correction and deviation convergence mechanisms, it continuously reduces the differences between the proxy model and the actual system, thereby obtaining a more accurate test case library.

[0039] According to some embodiments, in the process of determining critical test cases, specifically, if the criticality score is greater than the criticality threshold, the initial test case corresponding to the criticality score can be determined as the critical test case; and test deviations can be predicted based on the actual test cases and critical test cases of the system under test.

[0040] In this application, the criticality score and the criticality threshold can be compared. If the criticality score is greater than the criticality threshold, the initial test case corresponding to the criticality score can be used as the critical test case. The critical test case can be used to characterize the test case corresponding to the task whose execution result is a failure label.

[0041] If the criticality score is less than or equal to the criticality threshold, the corresponding initial test case is removed, thus filtering the test cases. Then, test deviations can be predicted based on actual test cases and critical test cases.

[0042] This application, by setting a criticality threshold, automatically filters highly critical test cases from a massive initial test case pool, effectively solving the decision-making challenge of "what to test" under a vast parameter space and limited testing resources. This allows testing activities to precisely focus on the key scenarios most likely to expose system differences, significantly improving testing efficiency. By comparing the selected key test cases with the actual test results of the real system, the application can directly and quantitatively predict and measure the test deviation between the proxy model and the actual system under test. This provides a key technical means for objectively evaluating the fidelity of the proxy model and diagnosing the specific manifestations of the differences between the model and the actual system.

[0043] According to some embodiments, if an offline real dataset exists, in the process of predicting test deviation, real test cases can be extracted from the offline real dataset; using a confusion matrix, a binary classification index for the testing process is determined based on the predicted labels in the key test cases and the real labels in the real test cases; and based on the binary classification index, test deviation is predicted.

[0044] Although model correction in offline mode is not considered in this application, the test deviation of a key use case in offline mode can be calculated as the basis for the final appropriate test case library.

[0045] With offline real-world datasets available, the accuracy of binary classification can be measured by the consistency and discrepancies between the predicted and true labels in a binary classification problem. A confusion matrix is ​​typically used to analyze the prediction results. Based on whether the predicted and true labels match in the real dataset, four scenarios can be identified: A true prediction is characterized by both the true and predicted labels being positive (i.e., the execution result of the task in both the real test case and the critical test case is that the failure event is triggered). This is represented by TP, and its true value is positive. A true prediction is characterized by both the true and predicted labels being negative (i.e., the execution result of the task in both the real and critical test cases triggers a success event). This is denoted by TN, and its true value is negative. A prediction with a positive true label but a negative predicted label is an incorrect prediction, denoted by FN. A prediction with a negative true label but a positive predicted label is considered an incorrect prediction and is represented by FP.

[0046] Using the confusion matrix and the predicted labels mentioned above, binary classification metrics can be calculated, including precision, recall (PR), and average precision (AP), etc. The specific calculation methods are as follows: , .

[0047] Precision is defined as the proportion of samples that are actually positive out of all samples predicted as positive. It measures the accuracy of the model in predicting positive classes. Recall, on the other hand, describes how many samples that are actually positive are correctly identified as positive by the model, reflecting the model's sensitivity to positive samples. In complex problems, precision and recall are generally not optimally achieved simultaneously. For example, increasing the classification threshold may improve precision but decrease recall. Mean precision, however, is a more comprehensive performance evaluation metric. Mean precision measures the overall performance of the model at different thresholds by calculating the area under the precision-recall (PR) curve. Theoretically, mean precision is defined as:

[0048] in It represents the functional relationship between precision and recall in the PR curve. In actual testing, since the curve is discrete, average precision is usually calculated through interpolation. Average precision can comprehensively reflect the model's overall ability for classification tasks, and is particularly suitable for comparing the performance of different models.

[0049] The above metrics, obtained by evaluating the proxy prediction model on real datasets, demonstrate the binary classification capability of the proxy prediction model. This binary classification capability also reflects the magnitude of the prediction bias when the proxy model is used to predict the criticality of test cases.

[0050] This application achieves efficient and standardized utilization of the historical or benchmark behavior of the actual system under test by explicitly extracting real test cases from offline real datasets. This provides a crucial data foundation and technical path for initiating and implementing intelligent resilience testing processes even in scenarios where large-scale real-time testing is not feasible. By introducing the confusion matrix, a classic machine learning evaluation tool, the prediction results of the proxy model on "key test cases" are systematically compared with the "real labels" of the actual system, enabling a structured and refined measurement of the differences between the two. This marks a shift in the evaluation of "test bias" in unmanned systems from qualitative or single-indicator descriptions to quantitative and interpretable analysis based on multi-dimensional binary classification indicators. The binary classification indicators calculated based on the confusion matrix can predict and characterize test bias from different aspects (such as model accuracy, completeness, and overall performance). Compared to a single error value, this multi-dimensional bias description can more accurately pinpoint the differences between the proxy model and the actual system.

[0051] According to some embodiments, for cases where real datasets are obtained online, i.e., when offline real datasets of the actual system under test are not obtained, the prediction of test deviation can be specifically based on a proxy prediction model. Key test cases are used to test the actual system under test to generate corresponding real test cases. Based on a preset keyness prediction function based on a neural network, the keyness scores of key test cases, and the real labels of real test cases, the cross-entropy loss is determined. Based on the cross-entropy loss, the test deviation is predicted.

[0052] In this application, the test parameters of key test cases can be directly used to test the actual system under test, and real test cases can be collected online.

[0053] The criticality prediction bias (or simply prediction bias) estimation function can be represented by the cross-entropy loss, i.e.: , in, Used to characterize the i-th key test case. The criticality score for the predicted i-th test case. Used to characterize keyness prediction functions based on neural networks It is the label representing the success or failure of the i-th real test case, and can be 0 or 1. This is the estimated test bias.

[0054] This application proposes a bootstrapping solution for online dataset acquisition. By directly utilizing key test cases generated by a proxy prediction model to conduct online testing on the actual system, the required set of real test cases can be dynamically generated "from scratch." This breaks through the dependence of intelligent resilience testing on pre-prepared complete data, greatly expanding the applicability and deployment flexibility of the framework. This method seamlessly integrates test execution (online acquisition of real labels) with deviation prediction (calculation of cross-entropy loss), forming a complete automated data acquisition and evaluation loop. Without manual intervention in data analysis, the system can autonomously complete the entire process from triggering tests to quantifying deviations, significantly improving the autonomy and intelligence of the testing process. By pre-setting a keyness prediction function based on a neural network and calculating the cross-entropy loss between its predicted score (keyness score) for key test cases and the real labels, a continuous and differentiable quantitative indicator with a solid information theory foundation is provided for test deviations. This indicator not only reflects the correctness of the prediction but also sensitively captures the probability distribution difference between the predicted confidence level and the actual result, providing a more refined insight into deviations than traditional classification indicators.

[0055] According to some embodiments, when collecting real test cases online, the specific process of correcting the keyness prediction model is to use a preset training strategy, with cross-entropy loss as the loss function, to perform a preset number of fine-tuning rounds of training on the training dataset to generate an updated proxy prediction model. The preset training strategy is the same class balancing strategy used in the training method of the proxy prediction model before correction.

[0056] In this application, the bias estimation function The form of the loss is the same as that of cross-entropy loss. Therefore, the gradient descent method can be used to fine-tune the parameters of the multilayer perceptron of the surrogate prediction model to improve the accuracy and recall of the prediction model.

[0057] In some implementations, during online testing, each critical test case obtains the actual execution result of the tested object, resulting in a "test case parameter-label" data pair, which forms a training dataset. Under sufficient resource conditions, the intelligent elastic testing algorithm pauses testing after collecting a preset number of data points in each round. The criticality prediction model is then fine-tuned on the training dataset to become an updated proxy prediction model. The specific fine-tuning method is as follows: Set the number of fine-tuning epochs, using cross-entropy loss as the loss function, and train on the training dataset to match the number of epochs corresponding to the fine-tuning epochs. To prevent excessive fine-tuning on small datasets from causing the neural network to forget previously learned information, the number of fine-tuning epochs should be small, such as 2. Similarly, to prevent class imbalance from leading to poor training results, the same class balance strategy used during fine-tuning as when training the proxy prediction model is employed.

[0058] This application achieves an automated closed-loop "test-evaluation-correction" process by directly using the predicted "cross-entropy loss" as the loss function. This transforms the optimization of the keyness prediction model from an offline, static task into a dynamic link in the online testing process that can self-adjust based on real-time feedback (test bias), significantly improving the intelligence and autonomous evolution capability of the testing system. By limiting the number of fine-tuning rounds used for training, the risk of drastic performance fluctuations or catastrophic forgetting due to overfitting a small amount of new data during online correction is avoided, ensuring the stability and controllability of model updates. Using the same class balancing strategy as the original surrogate prediction model as the pre-correction training strategy ensures that the model correction process is methodologically consistent with the initial model training. This not only effectively alleviates the class imbalance problem that may arise from online data flow but also guarantees the inherent consistency of the old and new models in data understanding and decision-making logic, avoiding the introduction of new and uncontrollable system biases due to inconsistent correction strategies.

[0059] According to some embodiments, when the criticality score is greater than the criticality threshold, in the process of determining critical test cases, sampling can be performed from the natural distribution according to the sampling probability in the preset sampling strategy in order to determine critical test cases from the initial test cases.

[0060] In this application, in order to balance exploration and development, a relatively small random sampling probability is set. The sampling probability can range from 0.05 to 0.2. For each test, there will be... The probability is directly derived from the natural distribution. Medium-sampled test cases. A natural distribution, or random distribution, is defined, for example, as a uniform distribution between the lowest and highest values ​​of the parameter. Sampling involves generating random numbers, eliminating the need for a huge set of test cases. The criticality prediction model then determines whether a case is critical. After filtering, these cases are used for testing. The set of test cases with criticality scores greater than a criticality threshold is defined as follows: .

[0061] in, As the keyness threshold, Used to characterize a set of pre-set or collected test cases. Used to characterize a test case. This is used to characterize a set of test cases whose criticality scores are greater than a criticality threshold.

[0062] The sampling distribution of the test cases in this application can then be expressed as: , in, For the characteristic function, when hour ,otherwise . Representational functions act on The expected value after that can be viewed as the selection of a set from the natural distribution. The probability of.

[0063] This application upgrades the process of determining key test cases from a simple hard threshold screening to an intelligent decision-making process that combines determinism (criticality score) and randomness (probabilistic sampling) by introducing a random sampling mechanism based on a preset sampling strategy and sampling probability. This avoids the rigidity or overfitting that may result from relying solely on score ranking, and enhances the diversity and robustness of test exploration. Instead of mechanically selecting the highest-scoring test cases, this application samples from high-scoring regions (criticality scores greater than the threshold) based on probability. Essentially, this introduces the exploration-exploitation tradeoff into the test case generation process: it favors selecting test cases that the model considers critical (high uncertainty or high impact) (i.e., the exploitation tradeoff in the "exploration-exploitation tradeoff") while retaining the possibility of exploring different scenarios within that region. This helps to discover critical defects with similar scores but different characteristics, improving the test's ability to uncover unknown risks.

[0064] According to some embodiments, when the criticality score is greater than the criticality threshold, the critical test cases are determined from the initial test cases. Specifically, sampling can be performed from the natural distribution according to the sampling probability to determine the preliminary test cases; random numbers are drawn from the uniform distribution, and if the random number is less than the sampling probability, the preliminary test cases are determined as critical test cases; if the random number is greater than or equal to the sampling probability, updated preliminary test cases are continuously sampled from the natural distribution, and the criticality score of the updated preliminary test cases is compared with the criticality threshold.

[0065] In this application, we first consider the natural distribution. Test cases s are obtained by sampling from a uniform distribution. Random numbers are drawn from the input (using a random number generation function, such as Python's random library). If the random number is less than... If the condition is met, then use that test case directly as the critical test case; otherwise, continue sampling test cases from it. ,Will The criticality score is compared with the criticality threshold until the test case is reached. Use after To test the actual system under test.

[0066] Criticality represents the probability that the test system will trigger a failure event under the current test case. The smaller the criticality prediction deviation, the more likely the test case is to fail, and the higher the test failure rate obtained by using the sampling distribution.

[0067] The sampling testing method in this application, on the one hand, combines the criticality scores of test cases output by the criticality prediction model to filter test cases, thereby increasing the probability of triggering failure events; on the other hand, it introduces random sampling probability to prevent incomplete exploration of the test space caused by sampling only in the natural distribution. By introducing random sampling probability, theoretically, 100% coverage of the entire test space can be achieved with a sufficient number of tests.

[0068] According to some embodiments, when the criticality score is less than or equal to the criticality threshold, this application can also resample from the natural distribution according to the remaining sampling probability to obtain new initial test cases, and compare the criticality score of the new initial test cases with the criticality threshold, wherein the remaining sampling probability is the difference between 1 and the sampling probability.

[0069] In this application, if the key score is less than or equal to the key score threshold, then it is calculated according to 1- The sampling probability is resampled to obtain new initial test cases, and then the criticality score is updated and recalculated.

[0070] This application establishes a structured probability allocation strategy covering the entire parameter space by explicitly dividing the total sampling probability into two parts (the sampling probability is used to select high-criticality use cases, and the remaining sampling probability is used to resample low-criticality regions). This ensures that testing resources are not completely limited to the high-criticality regions identified by the current model, but rather strategically reserves a portion of resources for continuous exploration of the entire parameter space, achieving a dynamic balance and synergy between "focused validation" and "extensive exploration".

[0071] This application involves three typical scenarios: 1. Collaborative search scenario The environment contains a swarm system consisting of one blue agent, three red agents, and two fixed-position obstacles.

[0072] In this environment, the task of the intelligent swarm system is to search for and attack the blue team's intelligent agent through cooperation among the agents. If any agent in the red team's swarm system successfully pursues the blue team's intelligent agent within T time steps, it means the swarm system has completed its task in the current round. Furthermore, the blue team's evasion strategy is a pre-set heuristic rule; the unmanned system being tested in this scenario is the red team's swarm system.

[0073] In practice, the origin of the coordinate system is set at the center of the environment, with the horizontal and vertical axes being the x and y axes, respectively. The range of the environment's x and y coordinates is set as follows: The environment can be configured with 12 parameters, including the 2D coordinates of 3 red agents, 1 blue agent, and 2 obstacles.

[0074] 2. Cooperative navigation scenario The environment includes three red team agents and three target locations to be reached. This scenario is used to test the red team's ability to collaboratively navigate and capture target locations. In this scenario, the unmanned system under test is a swarm system composed of three agents. They need to reach the three destinations without collision or omission within a given time T. If a collision occurs or the three destinations are not reached within the given time, the mission is considered a failure.

[0075] Scene 2 map is the same as Scene 1, with the environment center set as the coordinate origin, and the horizontal and vertical ranges set to... The environment can be configured with 12 parameters, including the 2D coordinates of 3 red team agents and 3 destinations.

[0076] 3. Bipedal robot scenario The environment includes a humanoid bipedal robot and a track composed of various terrains, used to test the bipedal robot's motion control capabilities.

[0077] In this scenario, a bipedal robot walks forward in a straight line, encountering various terrain obstacles along the way. Terrain types include tree stumps, traps, undulating flat ground, and slopes. The unmanned system under test is a bipedal robot whose task is to continuously walk forward and maintain its walking state without falling within a given time step T. In this scenario, the system is considered to have failed its task if it falls.

[0078] In the experiments conducted in this phase, the types of terrain encountered along the bipedal robot's walking route were fixed, and test cases were generated by configuring the length of each terrain type to change the terrain.

[0079] In some implementations, considering the fundamental differences in test cases for the same type of unmanned system under different tasks, this application treats unmanned systems of the same type but equipped with different intelligent algorithms and performing different types of tasks as different test objects. For example, unmanned combat vehicle swarm systems perform completely different tasks in cooperative encirclement and capture scenarios and cooperative navigation scenarios, and the decision-making algorithms used are also fundamentally different, resulting in completely different test cases. Therefore, they are treated as two different test objects.

[0080] In this application, the three scenario test cases are represented by a high-dimensional vector. Specifically, in scenarios 1 and 2, to describe the dynamic information during the operation of the environment, the positions of the scenario entities in the first 5 time steps are concatenated into a high-dimensional vector to represent the entire test case. Since there are 12 variable parameters in scenarios 1 and 2, each test case is represented by a 12×5=60-dimensional high-dimensional vector. In scenario 3, due to the limitation of the total time steps T, terrain that is too far back cannot be encountered by the unmanned system in each test. Therefore, only the length of the first 10 terrain features is considered as a parameter, and the state of the scenario 3 test case is represented by a 10-dimensional high-dimensional vector.

[0081] Due to the high-dimensional parameter space and its continuity, it is impossible to traverse the test space and it is difficult to capture the relationship between high-dimensional parameters and test results. Therefore, this application trains a 4-layer... To obtain the surrogate prediction model, the feature dimension of the input data in the input layer is 'd', a variable representing a specific numerical value that depends on the specific task. For example, if the input is a vector containing 10 features, then 'd' is 10. The surrogate model has three hidden layers, each with 256 neurons, and the final output of the output layer has a dimension of 1. The input of the multilayer perceptron represents the state dimension of each test case, which is 60, 60, and 10 in the three scenarios. For each of the three scenarios, a large-scale training dataset containing the test results of the surrogate model was collected and divided into training, validation, and test sets in a 3:1:1 ratio. As described above, the multilayer perceptron was trained using cross-entropy loss and the Adam optimizer.

[0082] The underlying implementation algorithms of the unmanned systems tested in all three experimental scenarios were deep learning algorithms. The selection method for the proxy model in the experiments is explained below: For scenarios 1 and 2, unmanned systems with the same underlying algorithm but different model parameters are used as surrogate models; for scenario 3, unmanned systems with different underlying algorithms are used as surrogate models. Sufficient data is collected through these surrogate models for training the surrogate prediction model.

[0083] In the specific implementation process, under sufficient resources, a single proxy model was used to achieve intelligent elasticity testing through continuous online testing and adaptive fine-tuning of the prediction model. The fine-tuning interval was 5000, and each fine-tuning utilized existing data for two training rounds. The termination condition for intelligent elasticity testing was set to reach a sufficiently large number of tests, which were 200,000, 200,000, and 100,000 for the three scenarios. Relying solely on the number of tests and the task failure rate of the tested object is insufficient to fully reflect the algorithm's performance. To objectively evaluate the performance of the proxy prediction model during intelligent elasticity testing and to objectively describe the change in test case key prediction bias during the intelligent elasticity testing process, 200,000 sets of test data from the actual testing of the tested object were collected in advance for each scenario to construct a "test dataset." Based on the offline estimation method for key prediction bias, the precision, recall, and average precision of the proxy prediction model in the test dataset will be discussed.

[0084] The following describes an apparatus embodiment of this application, which can be used to perform the method embodiment of this application. For details not disclosed in the apparatus embodiment of this application, please refer to the method embodiment of this application.

[0085] Figure 4 A block diagram of a resilience testing device for unmanned systems, provided as an embodiment of this application. Figure 4 As shown, 400 includes a score determination module 401, a test deviation determination module 402, a model correction module 403, and a cyclic test correction module 404.

[0086] The score determination module 401 is used to input the initial test cases of the proxy prediction model corresponding to the actual system under test into the preset criticality prediction model in order to determine the criticality score of the initial test cases. The test deviation determination module 402 is used to determine key test cases from the initial test cases based on the criticality score and the preset criticality threshold, and to predict test deviations based on the actual test cases and key test cases of the actual system under test. The model correction module 403 is used to correct the criticality prediction model based on test deviations when real test cases are collected online, so as to generate an updated criticality prediction model. The loop test correction module 404 is used to repeatedly call the score determination module 401, the test deviation determination module 402 and the model correction module 403 according to the preset maximum number of tests, and determine the final critical test cases.

[0087] Optionally, the test deviation determination module 402 is specifically used for: If the criticality score is greater than the criticality threshold, the initial test case corresponding to the criticality score will be identified as a critical test case. Predict test deviations based on actual test cases and key test cases of the system under test.

[0088] Optionally, the test deviation determination module 402, when predicting test deviations based on the actual test cases and key test cases of the system under test, is specifically used for: If the actual offline real dataset of the system under test is obtained, real test cases are extracted from the offline real dataset. Using a confusion matrix, a binary classification metric for the testing process is determined based on the predicted labels in key test cases and the real labels in real test cases. Based on binary classification indicators, predict test bias.

[0089] Optionally, the test deviation determination module 402, when predicting test deviations based on the actual test cases and key test cases of the system under test, is specifically used for: Without obtaining the actual offline real dataset of the system under test Based on the proxy prediction model, key test cases are used to test the actual system under test in order to generate corresponding real test cases. Based on a pre-defined neural network-based criticality prediction function, criticality scores of key test cases, and real labels of real test cases, the cross-entropy loss is determined. Predict test bias based on cross-entropy loss.

[0090] Optionally, the model correction module 403 is specifically used for: With real test cases collected online, a preset training strategy is used, with cross-entropy loss as the loss function, to train the model in the training dataset for a preset number of fine-tuning rounds to generate an updated agent prediction model. The preset training strategy is the same class balancing strategy used in the training method of the agent prediction model before the correction.

[0091] Optionally, the test deviation determination module 402, when the criticality score is greater than the criticality threshold, determines the initial test case corresponding to the criticality score as a critical test case, specifically for: If the criticality score is greater than the criticality threshold, sampling is performed from the natural distribution according to the sampling probability in the preset sampling strategy to determine the critical test cases from the initial test cases.

[0092] Optionally, when the criticality score is greater than the criticality threshold, the test deviation determination module 402 samples from the natural distribution according to the sampling probability in the preset sampling strategy to determine the critical test cases from the initial test cases, specifically for: If the criticality score is greater than the criticality threshold, samples are taken from the natural distribution according to the sampling probability to determine the preliminary test cases from the initial test cases; Random numbers are drawn from a uniform distribution, and if the random number is less than the sampling probability, the prepared test cases are identified as critical test cases. If the random number is greater than or equal to the sampling probability, continuously sample updated preliminary test cases from the natural distribution, and compare the criticality score and criticality threshold of the updated preliminary test cases with the sampling probability.

[0093] Optionally, the resilient testing apparatus 400 for unmanned system variability also includes a resampling module 405, used for: If the criticality score is less than or equal to the criticality threshold, new initial test cases are obtained by resampling from the natural distribution according to the remaining sampling probability. The criticality score of the new initial test cases is then compared with the criticality threshold, where the remaining sampling probability is the difference between 1 and the sampling probability.

[0094] The device performs functions similar to those described above; other functions are described in the preceding descriptions and will not be repeated here.

[0095] Figure 5 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application, such as... Figure 5 As shown, the electronic device 500 of this embodiment may include a memory 501 and a processor 502.

[0096] The memory 501 stores a computer program, which, when executed by the processor 502, causes the processor 502 to perform the method described in the above embodiments.

[0097] The processor 502 and the memory 501 are connected, for example, via a bus.

[0098] Optionally, the electronic device 500 may also include a transceiver. It should be noted that in practical applications, the transceiver is not limited to one, and the structure of the electronic device 500 does not constitute a limitation on the embodiments of this application.

[0099] Processor 502 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. Processor 502 may also be a combination that implements computational functions, such as including one or more microprocessor combinations, a combination of a DSP and a microprocessor, etc.

[0100] A bus can include a pathway for transmitting information between the aforementioned components. The bus can be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, only one thick line is used in the diagram, but this does not imply that there is only one bus or one type of bus.

[0101] The memory 501 may be a ROM (Read Only Memory) or other type of static storage device capable of storing static information and instructions, RAM (Random Access Memory) or other type of dynamic storage device capable of storing information and instructions, or an EEPROM (Electrically Erasable Programmable Read Only Memory), CD-ROM (Compact Disc Read Only Memory) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto.

[0102] The memory 501 is used to store application code that executes the solution of this application, and its execution is controlled by the processor 502. The processor 502 is used to execute the application code stored in the memory 501 to implement the content shown in the foregoing method embodiments.

[0103] Electronic devices include, but are not limited to: mobile terminals such as mobile phones, laptops, digital radio receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), and in-vehicle terminals (such as in-vehicle navigation terminals), as well as fixed terminals such as digital TVs and desktop computers. Servers can also be included. Figure 5 The electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0104] The electronic device in this embodiment can be used to execute the method of any of the above embodiments, and its implementation principle and technical effect are similar, so they will not be described again here.

[0105] This application also provides a non-transitory computer-readable storage medium storing computer-readable instructions thereon, which, when executed by a processor, cause the processor to perform the method as described in the above embodiments.

[0106] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a non-transitory computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0107] The embodiments of this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are only for the purpose of helping to understand the method and core ideas of this application. Furthermore, any changes or modifications made by those skilled in the art based on the ideas of this application, and on the specific implementation methods and application scope of this application, are all within the scope of protection of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A resilience testing method for the differences in unmanned systems, characterized in that, include: Step 1: Input the initial test cases of the proxy prediction model corresponding to the actual system under test into the preset criticality prediction model to determine the criticality score of the initial test cases; Step 2: Based on the criticality score and the preset criticality threshold, determine the critical test cases from the initial test cases, and predict the test deviation based on the actual test cases of the system under test and the critical test cases; Step 3: With the real test cases collected online, the criticality prediction model is corrected based on the test deviation to generate an updated criticality prediction model; Step 4: Repeat steps 1-3 using the updated criticality prediction model according to the preset maximum number of tests, and determine the final critical test cases.

2. The method according to claim 1, characterized in that, The process of determining key test cases from the initial test cases based on the key score and a preset key threshold, and predicting test deviations based on the actual test cases of the system under test and the key test cases, includes: If the criticality score is greater than the criticality threshold, the initial test case corresponding to the criticality score is determined as a critical test case. Based on the actual test cases of the system under test and the key test cases, predict the test deviation.

3. The method according to claim 2, characterized in that, The step of predicting test deviations based on the actual test cases of the system under test and the key test cases includes: If the offline real dataset of the actual system under test is obtained, the real test cases are extracted from the offline real dataset. Using a confusion matrix, a binary classification metric for the testing process is determined based on the predicted labels in the key test cases and the real labels in the real test cases. Based on the binary classification index, the test deviation is predicted.

4. The method according to claim 2, characterized in that, The step of predicting test deviations based on the actual test cases of the system under test and the key test cases includes: Without obtaining the offline real dataset of the actual system under test. Based on the agent prediction model, the actual system under test is tested using the key test cases to generate corresponding real test cases. Based on a preset, neural network-based keyness prediction function, the keyness scores of the key test cases, and the real labels of the real test cases, the cross-entropy loss is determined. The test bias is predicted based on the cross-entropy loss.

5. The method according to claim 4, characterized in that, The step of revising the criticality prediction model based on the test deviations, in the case of collecting the real test cases online, to generate an updated criticality prediction model, includes: With the real test cases collected online, a preset training strategy is used, with the cross-entropy loss as the loss function, to train the model in the training dataset for a preset number of fine-tuning rounds, so as to generate the updated agent prediction model. The preset training strategy is the same class balancing strategy used in the training method of the agent prediction model before the correction.

6. The method according to claim 2, characterized in that, The step of determining the initial test case corresponding to the criticality score as a critical test case when the criticality score is greater than the criticality threshold includes: If the criticality score is greater than the criticality threshold, sampling is performed from the natural distribution according to the sampling probability in the preset sampling strategy to determine the critical test cases from the initial test cases.

7. The method according to claim 6, characterized in that, When the criticality score is greater than the criticality threshold, sampling is performed from the natural distribution according to the sampling probability in the preset sampling strategy to determine the critical test cases from the initial test cases, including: If the criticality score is greater than the criticality threshold, sampling is performed from the natural distribution according to the sampling probability to determine the preliminary test cases from the initial test cases; Random numbers are drawn from a uniform distribution, and if the random numbers are less than the sampling probability, the preliminary test cases are determined as the key test cases; If the random number is greater than or equal to the sampling probability, updated preliminary test cases are continuously sampled from the natural distribution, and the criticality score of the updated preliminary test cases is compared with the criticality threshold.

8. The method according to claim 6, characterized in that, Also includes: If the criticality score is less than or equal to the criticality threshold, new initial test cases are obtained by resampling from the natural distribution according to the remaining sampling probability, and the criticality score of the new initial test cases is compared with the criticality threshold, wherein the remaining sampling probability is the difference between 1 and the sampling probability.

9. A resilient testing device for the differences in unmanned systems, characterized in that, include: The score determination module is used to input the initial test cases of the proxy prediction model corresponding to the actual system under test into the preset criticality prediction model in order to determine the criticality score of the initial test cases; The test deviation determination module is used to determine key test cases from the initial test cases based on the criticality score and the preset criticality threshold, and to predict test deviations based on the actual test cases of the system under test and the key test cases. The model correction module is used to correct the criticality prediction model based on the test deviation when the real test cases are collected online, so as to generate an updated criticality prediction model. The loop test correction module is used to repeatedly call the score determination module, the test deviation determination module, and the model correction module according to the preset maximum number of tests, and determine the final critical test cases.

10. An electronic device, characterized in that, include: processor; A memory storing a computer program that, when executed by the processor, causes the processor to perform the method as described in any one of claims 1-8.

Images