Self-adaptive multi-dimensional risk test strategy design method and system and storage medium

Abstract

The invention relates to the technical field of software engineering and automatic testing, and discloses a self-adaptive multi-dimensional risk testing strategy design method and system and a storage medium. The method comprises the following steps: triggering a risk assessment service through a CI / CD assembly line, preprocessing original data obtained by a plurality of heterogeneous data sources, and generating a risk feature vector; outputting a multi-dimensional risk index by using the risk assessment model, fusing the multi-dimensional risk index and the dynamic weight thereof based on an adaptive weighted fusion model, and calculating a comprehensive risk score; dynamically updating a risk score threshold value used for dividing the risk level, automatically generating a test strategy matched with the current risk level according to a preset mapping rule of the risk level and the test strategy, and calling the test suite to automatically execute the test strategy; and collecting a test execution result and online operation data as training data, and regularly optimizing parameters of the risk assessment model. The test strategy can be dynamically adjusted according to the real-time risk situation.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of software engineering and automated testing technology, and particularly relates to a self-adaptive multi-dimensional risk test strategy design method, system and storage medium. BACKGROUND

[0002] In the field of modern software engineering, the development and operation system DevOps with continuous integration (CI) and continuous deployment (CD) as the core has become an industry standard, aiming to significantly improve the delivery speed and iteration frequency of software. However, such a high-speed iteration development environment poses a great challenge to traditional software testing methods. Existing test strategies, whether manual testing or basic automated testing, are heavily dependent on the pre-experience of testers and static rule setting, and their response speed cannot match the rapid development rhythm, resulting in a serious disconnection between testing activities and development processes.

[0003] Specifically, the traditional test strategy has the following inherent defects: 1. It is static and lagging, usually formulated once at the beginning of the project or at a specific stage, relying on the personal experience of test engineers and the static understanding of requirements, and cannot be dynamically adjusted according to the frequent changes of code, architecture evolution and user behavior changes, resulting in a disconnection between test coverage and actual risk points; 2. The risk assessment dimension is single, and most automated test triggering conditions are simple, triggered only based on code changes, and the single-dimensional assessment method is easy to misjudge, which may waste test resources in low-risk areas and not test enough in high-risk areas, thus causing misallocation of test resources; 3. The process is fragmented, in a typical DevOps environment, although the test activities can be automatically executed, the adjustment of test strategy still needs human intervention, and there are barriers between data and decision-making processes among development, testing and operation, making it difficult for risk assessment results to automatically and immediately guide subsequent test activities, thus weakening the agile advantage of DevOps; 4. Lack of self-learning and evolution ability, the valuable data such as the execution results of tests and the faults found online cannot be systematically used to benefit and optimize the risk assessment model and test strategy in the early stage, resulting in stagnation of the efficiency of the test system.

[0004] Although the existing technology introduces some automated test tools, it is still essentially limited to executing preset scripts for testing, and lacks the ability to dynamically adjust the test range and depth according to the real-time risk situation of the software. SUMMARY

[0005] The application aims to provide a self-adaptive multi-dimensional risk test strategy design method, system and storage medium, embed risk assessment services and risk test services in CI / CD process execution, and continuously optimize risk assessment models by using a closed-loop feedback mechanism, so that the test strategy has dynamic adaptability, optimally configures test resources, and enhances the overall stability and security of a software system.

[0006] To achieve the above-mentioned purpose, the application provides the following technical solutions. A self-adaptive multi-dimensional risk test strategy design method comprises the following steps. Trigger a risk assessment service through a continuous integration CI / continuous deployment CD pipeline, acquire original data from multiple heterogeneous data sources in the whole life cycle of software development and operation, preprocess the original data, and generate a risk feature vector; Input the risk feature vector into a risk assessment model, output multi-dimensional risk indicators, and fuse the multi-dimensional risk indicators and their dynamic weights based on a self-adaptive weighted fusion model to calculate a comprehensive risk score; Dynamically update a risk score threshold for dividing risk levels, automatically generate a test strategy matching a current risk level according to a preset mapping rule of risk levels and test strategies, and call a test suite to automatically execute the test strategy; Collect test execution results and online running data as training data, and periodically optimize parameters of the risk assessment model.

[0007] A self-adaptive multi-dimensional risk test strategy design system comprises the following. A data processing module is configured to acquire original data from multiple heterogeneous data sources in the whole life cycle of software development and operation when a risk assessment service is triggered through a continuous integration CI / continuous deployment CD pipeline, preprocess the original data, and generate a risk feature vector; A risk prediction module is configured to input the risk feature vector into a risk assessment model, output multi-dimensional risk indicators, and fuse the multi-dimensional risk indicators and their dynamic weights based on a self-adaptive weighted fusion model to calculate a comprehensive risk score; A test execution module is configured to dynamically update a risk score threshold for dividing risk levels, automatically generate a test strategy matching a current risk level according to a preset mapping rule of risk levels and test strategies, and call a test suite to automatically execute the test strategy; A model optimization module is configured to collect test execution results and online running data as training data, and periodically optimize parameters of the risk assessment model.

[0008] Based on the same inventive concept, the application also provides a computer storage medium, wherein computer executable instructions are stored in the computer storage medium, and the computer executable instructions realize the adaptive multi-dimensional risk test strategy design method.

[0009] Technical effects and advantages of the application: (1) Realize accurate and comprehensive identification of software risks: comprehensively consider multiple source information such as code changes, historical defects, user influence, performance data, adopt an adaptive weighted fusion mechanism of multi-dimensional risk indicators, dynamically generate accurate risk assessment results, effectively overcome the limitations of single-dimensional evaluation, realize accurate positioning of high-risk modules and changes, and avoid test blind area; (2) Realize optimization and automatic configuration of test resources: deeply integrate the risk assessment service with the CI / CD process of DevOps, can automatically generate and execute differentiated test strategies according to the real-time calculation of the comprehensive risk score, for example, perform deep testing on high-risk and perform rapid testing on low-risk, thereby realizing on-demand allocation of test resources, greatly improving test efficiency and return on investment, and avoiding resource waste; (3) Endow the test system with continuous adaptive and evolution ability: design a closed-loop feedback mechanism, continuously use test results and online running data for feedback and training of the risk model, so that the prediction accuracy of the model is continuously improved over time, thereby having strong self-optimization ability.

[0010] Other features and advantages of the application will be described in the following description, and some will become apparent from the description, or will be understood by those skilled in the art. The purpose and other advantages of the application can be achieved and obtained by the structure indicated in the specification, claims and drawings. BRIEF DESCRIPTION OF DRAWINGS

[0011] In order to more clearly illustrate the technical solutions in the embodiments of the application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the application, and other drawings can be obtained by those skilled in the art without creative labor.

[0012] Figure 1 The flowchart of the adaptive multi-dimensional risk test strategy design method of the embodiment of the application; Figure 2 The structural schematic diagram of the adaptive multi-dimensional risk test strategy design system of the embodiment of the application. DETAILED DESCRIPTION

[0013] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the scope of the present application.

[0014] The embodiment of the present application discloses a self-adaptive multi-dimensional risk test strategy design method, as shown in the figure, the method comprises: Figure 1 S1, triggering a risk assessment service through a continuous integration CI / continuous deployment CD pipeline, obtaining original data from a plurality of heterogeneous data sources in a software development and operation life cycle, preprocessing the original data to generate a risk feature vector; S2, inputting the risk feature vector into a risk assessment model, outputting a multi-dimensional risk index, and fusing the multi-dimensional risk index and its dynamic weight based on a self-adaptive weighted fusion model to calculate a comprehensive risk score; S3, dynamically updating a risk score threshold for dividing risk levels, automatically generating a test strategy matching the current risk level according to a preset mapping rule of risk level and test strategy, and calling a test suite to automatically execute the test strategy; S4, collecting test execution results and online running data as training data, and periodically optimizing parameters of the risk assessment model.

[0015] In the embodiment of the present application, the risk assessment service and the risk test service are embedded in the CI / CD process for execution. During risk assessment, multi-source data is aggregated, multi-dimensional risk assessment is performed by using a risk assessment model, a test strategy matching the current risk level is automatically generated according to the multi-dimensional risk assessment result, and the test suite is called to automatically execute the test strategy. After that, the test execution result and the online running data are used as feedback to realize continuous self-learning and closed-loop optimization of the risk assessment model, so that the test strategy has dynamic adaptability. The present application can realize optimal allocation of test resources, and at the same time, enhance the overall stability and security of the software system.

[0016] The workflow of the present application scheme follows a clear, closed-loop logical sequence. The complete workflow from data input to strategy execution to feedback optimization is described below.

[0017] I. Multi-source heterogeneous data aggregation The original data is actively collected or passively received from a plurality of heterogeneous data sources in a software development and operation life cycle, including but not limited to: Obtaining code commit records, code change frequency, and code cyclomatic complexity from a code version control system (such as a distributed version control system Git);​ Obtain defect reports, requirement change history, defect density from project / defect tracking system (e.g. project management system JIRA); Obtain real-time performance indicators of online services, including program interface API response logs such as API delay, error rate, resource consumption, etc. from application performance management system APM; Collect user-submitted tickets, application store reviews, social media feedback, and other unstructured or semi-structured data from user feedback systems; Collect the dependency relationship of functional modules from software architecture definition files, including the user range of change impact, etc.

[0018] Based on this, the data source and derived feature table shown in Table 1 is generated.

[0019] Table 1

[0020] II. Data processing and feature engineering Since the collected raw data is multi-source, heterogeneous and contains noise, it cannot be directly used for model training, so the raw data needs to be preprocessed.

[0021] The preprocessed raw data includes: 1. Data cleaning, including filling missing values, correcting format errors, removing outliers and deleting duplicates.

[0022] This step aims to ensure the integrity and consistency of the data, including: using statistical methods (such as mean, median filling) to handle missing values, unifying the format of date, numerical and other fields, and using box plots and other methods to identify and handle outliers, detecting and deleting duplicate records through unique identifiers, etc.

[0023] 2. Feature encoding, including converting categorical features into numerical values using one-hot encoding or label encoding methods to make them suitable for machine learning models.

[0024] In this step, for categorical features (such as software module names, error types), one-hot encoding or label encoding methods are used to convert them into numerical format, where one-hot encoding is suitable for unordered categories to avoid introducing false order relationships, and label encoding is suitable for ordered categories or to reduce feature dimensions when the model allows.

[0025] 3. Data conversion, including using Z-score standardization method to normalize numerical features.

[0026] This step is to eliminate the model training bias caused by different dimensions between different features, and all numerical features are normalized using the Z-score standardization method.

[0027] 4. Feature selection, including extracting key features from the original data set, such as code complexity, module historical defect rate, change impact range, etc., and selecting a feature subset related to the target variable based on a mutual information statistical method, for example, selecting a feature set most related to software risk from a large number of features.

[0028] In order to improve the efficiency of the model and avoid overfitting, a mutual information (MI) based statistical method is used to select the features most related to software risk (such as defect occurrence probability) in this step. Mutual information MI is used to measure the amount of information contained in one random variable about another random variable, and the calculation formula is as follows:

[0029] wherein, X and Y represent the feature set and the target variable set, p ( x , y ) is the joint probability distribution of the feature subset x and the target variable y , p ( x ) and p ( y ) are the marginal probability distributions of x and y . By calculating the mutual information value M I( X , Y ) of each feature set and the target variable set, and selecting the feature subset with the highest value as the final feature subset.

[0030] In an embodiment of the present application, the risk feature vector generated after processing the original data includes: code change frequency, code complexity, historical defect density, API error rate, and negative user feedback.

[0031] III. Multi-dimensional risk assessment In order to analyze the risk from different levels and dimensions, the present application adopts a composite machine learning model composed of a random forest model, a gradient boosting decision tree model and a support vector machine model. These three models work together, each with its unique advantages, to improve the accuracy and robustness of risk assessment.

[0032] Random Forest (RF): This model plays the role of a "broad filter" and is efficient in handling high-dimensional data, assessing feature importance, and avoiding overfitting. RF is used for a preliminary, broad risk scan of the data, and is particularly good at identifying software modules with potential risks due to complex structures or frequent changes.

[0033] RF improves prediction accuracy and robustness by building multiple decision trees and taking a vote. The final prediction result (for regression problems) can be represented as:

[0034] where B is the total number of decision trees in the forest, T b is the prediction output of the y th decision tree for input features b . y

[0035] Gradient Boosting Decision Trees (GBDT): This model acts as a "deep analyzer" and performs more detailed risk mining based on the preliminary screening by RF. GBDT learns the complex non-linear dependencies between features by iteratively building decision trees, and can accurately locate function points with significantly increased error rates under specific input combinations or boundary conditions. The GBDT model update strategy is as follows:

[0036] where is the strong learner trained in the m th iteration, which aims to predict the target value x from input features , is the weak learner trained in the m th iteration, is the optimal weight or output value of the m th decision tree, is the learning rate, which controls the contribution of each decision tree to the final result and prevents overfitting.

[0037] Support Vector Machine (SVM): This model acts as a "precise classifier" and its core advantage is to find the optimal classification hyperplane to separate data points of different classes with the maximum margin. The SVM model is mainly used for accurate classification of risk levels, especially in distinguishing between medium and high risks, low and medium risks, and other ambiguous boundaries, providing more explicit and reliable decision-making basis.​

[0038] SVM model introduces slack variables and a regularization parameter C to tolerate partial misclassifications, whose optimization objective is:

[0039] The constraint condition is:

[0040] where, w is the normal vector of the hyperplane, b is the intercept of the hyperplane, x i is the risk feature vector of the i-th training sample, i i represents the true class label of the i-th training sample. y i The collaborative work of the three models enables the risk assessment process to have breadth, depth and accuracy, thereby generating a highly reliable risk profile.

[0041] Specifically, the above risk feature vector is input into the risk assessment model, which outputs multi-dimensional risk indicators, wherein the risk assessment model is a composite machine learning model composed of a random forest model, a gradient boosting decision tree model and a support vector machine model. Each model independently analyzes from the perspectives of breadth scanning, depth mining and accurate classification, and correspondingly outputs first, second and third risk indicators, such as code risk, test risk and operation and maintenance risk.

[0042] Further, the adaptive weighted fusion model is used to fuse the multi-dimensional risk indicators and their dynamic weights to calculate the comprehensive risk score, i.e., the first, second and third risk indicators are input into the adaptive weighted fusion model, and the weights of the risk indicators are dynamically adjusted according to the performance of the random forest model, the gradient boosting decision tree model and the support vector machine model, and the comprehensive risk score is output.

[0043] wherein the weight adjustment formula of the first, second and third risk indicators is:

[0044] is the weight of the j-th risk indicator at time t;

[0045] is the weight learning rate for controlling the step size of weight adjustment; is the observation value or deviation from the expected value of the j-th risk indicator at time t; is the weight learning rate for controlling the step size of weight adjustment; ​In the adaptive weighted fusion model, the calculation formula of the comprehensive risk score is:

[0046] is the comprehensive risk score at time t; is the normalized value of the jth risk indicator at time t.

[0047] In the present application, by defining a series of independent risk indicators such as login failure rate, system delay, security event log, etc., the weights of each indicator are dynamically adjusted according to real-time running data and historical performance, and then an adaptive weighting mechanism is used to fuse risk indicators from different dimensions and their dynamic weights, thereby generating a more comprehensive and more context-aware comprehensive risk score.

[0048] Further, a self-learning risk threshold mechanism is introduced to dynamically adjust the definition of risk by the system. The core idea of this mechanism is that the threshold used to divide the risk level should also be learned, rather than being hard-coded. The correlation between the risk prediction and the actual result (for example, whether a serious defect is really found in the module later) is continuously tracked, and the threshold is dynamically adjusted according to this correlation.

[0049] In the present application, the risk score threshold used to divide the risk level is dynamically updated, a test strategy matching the current risk level is automatically generated according to a preset mapping rule of risk level and test strategy, and the test suite is called to automatically execute the test strategy. Specifically, it comprises: an online gradient descent method is used to dynamically adjust the risk score threshold;

[0050] wherein, is the updated risk score threshold; is the old risk score threshold; is the score threshold learning rate, used to control the smoothness of the risk score threshold adjustment; is the deviation between the actual value and the predicted value of the risk score threshold.

[0051] According to the above self-learning risk threshold mechanism, the first risk score threshold and the second risk score threshold are calculated, and the risk score interval is divided. Specifically, the interval greater than the first risk score threshold is the high-risk interval; the interval greater than or equal to the second risk score threshold and less than or equal to the first risk score threshold is the medium-risk interval; and the interval less than the second risk score threshold is the low-risk interval.

[0052] If it is found that the current risk score threshold is too lenient (leading to risk miss) or too strict (leading to frequent false positives), the risk score threshold can be fine-tuned so that the risk judgment criteria become more accurate over time.

[0053] IV. CI / CD pipeline integration and automated test policy execution In order to deeply embed the risk assessment model into the integrated system of the modern software development and delivery life cycle, the risk assessment model is encapsulated into a standardized, network-accessible service (for example, a RESTful API or a containerized application) that interacts with mainstream CI / CD tools (such as Jenkins, GitLab CI / CD, Azure DevOps) through API calls or webhook mechanisms, and through integration with DevOps processes, abstract risk scores are converted into specific automated test actions.

[0054] The specific workflow of the integrated system is as follows: Configuration trigger point: define a special "risk assessment" stage in the CI / CD pipeline, which is usually set after the "code build" is successful and before the "deployment to test environment"; Automatic call: when the CI / CD pipeline executes to the risk assessment stage, the CI / CD tool automatically triggers a script that is responsible for collecting raw data (such as code commit ID, change file list, module, etc.) and sending these information to the risk assessment service module through API request; Synchronous waiting and result return: the CI / CD pipeline synchronously waits for the response of the risk assessment service module and receives a structured result returned by it, which contains the final quantitative risk score and other related diagnostic information.

[0055] This loose coupling integration allows the risk assessment service to flexibly access any CI / CD platform that supports standard interfaces, with good universality and scalability.

[0056] In the present application, the test policy engine triggers the corresponding test policy according to the risk level of the risk score interval: When the comprehensive risk score is greater than the first risk score threshold, it is a high risk level, triggering full regression testing, performance testing and security scanning testing; When the second risk score threshold is less than or equal to the comprehensive risk score, it is a medium risk level, triggering core function regression testing; When the comprehensive risk score is less than the second risk score threshold, it is a low risk level, triggering smoke testing or build verification BVT testing.

[0057] An example of mapping risk levels to test strategies is shown in Table 2, where the first risk score threshold is 0.7, the second risk score threshold is 0.4, and the test strategies include test scope, test type, and test resource allocation.

[0058] Table 2

[0059] V. Closed-loop feedback and continuous optimization A closed-loop feedback loop is constructed to learn from the results of each test execution, thereby evolving the entire system. The working principle of the closed-loop feedback mechanism is as follows: Result collection: After the automated test execution is completed, the test results (such as the pass / fail rate of test cases, code coverage, the number of newly discovered defects, and severity level) are automatically collected and recorded; Production monitoring: After the test software is deployed to the production environment, monitoring systems such as APM continuously collect runtime data (such as real user error rate, performance, system crash reports); Data backflow retraining: The results of the test phase and the performance of the production phase are automatically fed back to the data aggregation layer for incremental updating or periodic retraining of the risk assessment model.

[0060] The establishment of this closed-loop feedback mechanism produces a strong synergistic effect between the multi-dimensional risk assessment model and the DevOps system automation process. The risk assessment model provides intelligent decision-making "inputs" for the automation process, while the execution results of the automation process provide "feedback" for the optimization of the risk assessment model. The close integration of intelligent input and efficient execution framework transforms a static analysis model into a dynamic, co-evolutionary intelligent system with the software life cycle, thereby truly realizing risk-driven agile development and quality assurance.

[0061] Specifically, the test execution results and online running data are collected as training data, and the parameters of the risk assessment model are optimized periodically, including: During the process of invoking the test suite from the CI / CD pipeline to execute the corresponding test strategy, collect defect data related to test failures and performance bottleneck data in online production, and store them in the database; Clean and label the defect data and performance bottleneck data, convert them into a format suitable for machine learning model training, and build a training data set; When the next training cycle of the risk assessment model is triggered, input the training data set into the risk assessment model to adjust and optimize the model parameters.

[0062] To illustrate the working mechanism and practical value of the present application, an example of a high-risk, high-demand financial payment system is given to demonstrate the entire process from code submission to test execution.

[0063] Scenario: The core payment system of a certain financial technology company needs to be upgraded. The developer has completed a code modification on "optimizing transaction fee calculation logic" and submitted the code to the Git repository.

[0064] Step 1: CI / CD pipeline trigger The developer's git push operation automatically triggers the company's Jenkins CI / CD pipeline. The pipeline first performs code compilation and unit testing, and after all tasks are successfully completed, it enters the preset "risk assessment" stage. The Jenkins CI / CD pipeline calls the detailed information of this change (such as commit hash, change file path, author, etc.) through API, and sends it to the risk assessment service module.

[0065] Step 2: Multi-source data collection and feature engineering After receiving the request, the risk assessment service module immediately starts the data collection program: Git analysis: found that this change involves the core transaction module PaymentService.java, which has a high frequency of historical changes and a high complexity of this modification; JIRA analysis: found that the PaymentService.java module is associated with multiple high-priority payment failure defects in the past 6 months; APM system analysis: obtained the performance data of payment-related APIs in the production environment in the past 24 hours, found that the transaction success rate has a slight downward trend; User feedback analysis: through natural language processing, found that some users recently complained about "unclear fee display" in the community.

[0066] The collected raw data is preprocessed, including data cleaning, feature encoding, data transformation, feature selection, etc., to generate a series of feature vectors including "code change frequency", "code complexity", "historical defect density", "API error rate", and "negative user feedback".

[0067] Step 3: Risk assessment and scoring The feature vector is input into the composite risk assessment model composed of random forest model, gradient boosting decision tree model and support vector machine model: Random forest quickly identifies PaymentService.java as a high-risk file; Gradient Boosting Decision Tree further analysis found that in a specific high concurrency scenario, the new commission calculation logic may conflict with the old discount coupon logic, resulting in calculation errors; Support Vector Machine clearly classifies this change into the "high risk" category based on the comprehensive features.

[0068] Adaptive weighting: due to the monitoring of the API error rate in the production environment, the adaptive weighting mechanism of the system automatically increases the weights of the "API error rate" and "historical defect density" risk indicators, and generates a comprehensive risk score of 0.85 based on all analysis results, and returns it to the Jenkins CI / CD pipeline.

[0069] Step four: automatic test strategy generation After receiving the comprehensive risk score of 0.85, the Jenkins CI / CD pipeline queries the built-in strategy mapping table (Table 2) and matches it to the "high risk" level test strategy, automatically generating the following test plan: Test scope: execute the complete payment module regression test suite; Test type: includes functional testing, special data verification for commission calculation, stress testing simulating the Double Eleven peak traffic, and security scanning testing for transaction data; Resource allocation: automatically create and configure a large-scale performance testing environment with 10 nodes on the cloud.

[0070] Step five: automatic execution and feedback The Jenkins CI / CD pipeline schedules the automatic test framework to execute the test according to the generated plan; Results: functional testing found a commission calculation error under a boundary condition, and more importantly, stress testing successfully reproduced the performance bottleneck predicted by the Gradient Boosting Decision Tree when simulating high concurrency requests, and found that the new logic caused database lock contention, resulting in a large number of transaction timeout failures at peak times.

[0071] Action: due to the discovery of serious defects, the pipeline is automatically interrupted, the build status is marked as "failed", and a high-priority defect ticket is automatically created in JIRA, detailing the reproduction steps and logs, and assigned to the original code committer, at the same time, a detailed test report is sent to the development team and the test team.

[0072] The defect data found in this test, as well as the performance bottleneck data exposed in the stress test, are recorded by the system. In the next model training cycle, these valuable "failure cases" will be used to further optimize the model, so that it can make more accurate risk predictions in the future when facing similar code changes.

[0073] In the embodiments of the present application, there are two core mechanisms: Intelligent and dynamic risk assessment: By adaptive weighting mechanism, multi-dimensional risk indicators are integrated, and various machine learning models are used for deep analysis, which surpasses the traditional method relying on single and static indicators, realizes accurate and context-aware prediction of software risk, and the self-learning risk threshold and model optimization capability enable the risk assessment model to continuously evolve; Efficient and automated closed-loop execution framework: The risk assessment service is integrated into the CI / CD pipeline, realizing from risk identification to test strategy generation, and then to automatic execution of test strategy, and more importantly, through the closed-loop feedback mechanism, the test and operation results are converted into the input of the optimized model, forming strong adaptive ability.

[0074] The essence of the present application is that it changes the risk assessment model from an isolated analysis tool to an intelligent engine driving the entire software quality assurance process. The deep synergy of this evaluation intelligence and process automation is the key to solving the contradiction between speed and quality in modern software engineering. The specific embodiments fully demonstrate the great application value of the present application in key business scenarios such as financial payment, which can significantly improve test efficiency, optimize resource allocation, and actively prevent major defects and performance problems, providing strong technical support for enterprises to ensure software quality and improve user satisfaction in fierce market competition. Therefore, the present application has strong practicality and broad industrial application prospect.

[0075] As shown in Figure 2 The embodiments of the present application also provide an adaptive multi-dimensional risk test strategy design system, which comprises: A data processing module is configured to acquire original data from a plurality of heterogeneous data sources in the whole life cycle of software development and operation when a continuous integration (CI) and continuous deployment (CD) pipeline triggers a risk assessment service, preprocess the original data, and generate a risk feature vector; A risk prediction module is configured to input the risk feature vector into a risk assessment model, output multi-dimensional risk indicators, and fuse the multi-dimensional risk indicators and their dynamic weights based on an adaptive weighting fusion model to calculate a comprehensive risk score; A test execution module is configured to dynamically update a risk score threshold for dividing risk levels, automatically generate a test strategy matching the current risk level according to a preset mapping rule of risk levels and test strategies, and call a test suite to automatically execute the test strategy; A model optimization module is configured to collect test execution results and online operation data as training data, and periodically optimize parameters of the risk assessment model.

[0076] Further, the risk prediction module comprises: A composite machine learning module is configured to input the risk feature vector into a composite machine learning model composed of a random forest model, a gradient boosting decision tree model and a support vector machine model, each model independently analyzes from the perspective of breadth scanning, depth mining and accurate classification, and outputs a first risk indicator, a second risk indicator and a third risk indicator respectively; A weighted fusion module is configured to input the first risk indicator, the second risk indicator and the third risk indicator into an adaptive weighted fusion model, dynamically adjust the weights of the risk indicators according to the performances of the random forest model, the gradient boosting decision tree model and the support vector machine model, and output a comprehensive risk score. Wherein, the weight adjustment formula of the first risk indicator, the second risk indicator and the third risk indicator is:

[0077] is the weight of the jth risk indicator at time t; is the weight learning rate, used to control the step size of weight adjustment; is the observation value or the deviation from the expected value of the jth risk indicator at time t; In the adaptive weighted fusion model, the calculation formula of the comprehensive risk score is:

[0078] is the comprehensive risk score at time t; is the normalized value of the jth risk indicator at time t.

[0079] Further, the test execution module comprises: A threshold adjustment module is configured to realize dynamic adjustment of the risk score threshold by using an online gradient descent method;

[0080] Wherein, is the updated risk score threshold; is the old risk score threshold; is the score threshold learning rate, used to control the smoothness of the risk score threshold adjustment; is the deviation between the actual value and the predicted value of the risk score threshold; The test strategy mapping module is configured to divide risk score intervals according to the first risk score threshold and the second risk score threshold, and trigger corresponding test strategies according to risk levels of the risk score intervals; When the comprehensive risk score is greater than the first risk score threshold, the risk level is high, and full regression testing, performance testing and security scanning testing are triggered; When the second risk score threshold is less than or equal to the comprehensive risk score and less than or equal to the first risk score threshold, the risk level is medium, and core function regression testing is triggered; When the comprehensive risk score is less than the second risk score threshold, the risk level is low, and smoke testing or build verification BVT testing is triggered.

[0081] As to the system in the above embodiment, the specific manner in which each unit module performs the operation has been described in detail in the embodiment related to the method, and will not be described in detail here.

[0082] Based on the same inventive concept, the embodiments of the present application also provide an electronic device, comprising a memory and a processor, wherein the processor is configured to read and execute a computer program stored in the memory to implement the adaptive multi-dimensional risk test strategy design method.

[0083] Based on the same inventive concept, the embodiments of the present application also provide a computer storage medium, wherein the computer storage medium stores computer executable instructions, and the computer executable instructions implement the adaptive multi-dimensional risk test strategy design method when executed.

[0084] In several embodiments provided in the present application, it should be understood that the disclosed apparatus and method can be implemented in other manners. For example, the described apparatus embodiments are merely schematic, and the division of the modules is merely a logical function division, and there can be another division manner in actual implementation, for example, a plurality of modules or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed mutual coupling or direct coupling or communication connection can be indirect coupling or communication connection through some interfaces, apparatuses or modules, and can be electrical, mechanical or other forms.

[0085] The modules described as separate components may or may not be physically separate, and the components shown as modules may or may not be physical modules, i.e., may be located in one place, or may be distributed to multiple network modules. Part or all of the modules can be selected to achieve the purpose of the embodiment scheme according to actual needs. In addition, the functional modules in each embodiment of the present application can be integrated in one processing module, or each module can be physically present alone, or two or more modules can be integrated in one module. The integrated module can be realized in the form of hardware or in the form of a software function module.

[0086] The integrated module, if realized in the form of a software function module and sold or used as an independent product, can be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the present application essentially or the parts that contribute to the prior art or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0087] It should be noted that for the foregoing method embodiments, in order to facilitate description, they are all described as a combination of a series of actions, but those skilled in the art should know that the present application is not limited by the order of the described actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should know that the embodiments described in the specification all belong to preferred embodiments, and the actions and modules involved are not necessarily essential to the present application.

[0088] In the above embodiments, the description of each embodiment has its own emphasis, and the parts not described in detail in a certain embodiment can be referred to the related description of other embodiments. Finally, it should be pointed out that: the above only describes the preferred embodiments of the present application and is not intended to limit the present application. Although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments, or make equivalent replacements to some of the technical features, as long as they are within the spirit and principles of the present application. Any modification, equivalent replacement, improvement, etc. shall be included in the protection scope of the present application.

Claims

1. An adaptive multi-dimensional risk testing strategy design method, characterized in that, The method includes: Risk assessment services are triggered by continuous integration (CI) / continuous deployment (CD) pipelines, raw data is obtained from multiple heterogeneous data sources throughout the software development and operation lifecycle, and the raw data is preprocessed to generate risk feature vectors. The risk feature vector is input into the risk assessment model, which outputs multi-dimensional risk indicators. The multi-dimensional risk indicators and their dynamic weights are then fused based on an adaptive weighted fusion model to calculate a comprehensive risk score. The risk scoring threshold used to classify risk levels is dynamically updated. A test strategy matching the current risk level is automatically generated based on the preset mapping rules between risk levels and test strategies, and the test suite is invoked to automatically execute the test strategy. Collect test execution results and online operation data as training data, and regularly optimize the parameters of the risk assessment model.

2. The method according to claim 1, characterized in that, The process of acquiring raw data from multiple heterogeneous data sources throughout the software development and operation lifecycle includes: Obtain code commit history, code change frequency, and cyclomatic complexity from the code version control system; Obtain defect reports, requirement change history, and defect density from the defect tracking system; Obtain real-time performance metrics of online services from the application performance monitoring system, including API response logs. Collect user-submitted tickets, app store reviews, and social media feedback from the user feedback system; Collect the dependencies of functional modules from the software architecture definition document, including the user scope affected by changes.

3. The method according to claim 2, characterized in that, The preprocessing of the raw data includes: Data cleaning includes filling in missing values, correcting formatting errors, removing outliers, and deleting duplicates; Feature encoding includes converting categorical features into numerical features using one-hot encoding or label encoding methods; Data transformation, including normalizing numerical features using the Z-score standardization method; Feature selection includes extracting key features from the original dataset and filtering out a subset of features relevant to the target variable based on mutual information statistical methods; The generated risk feature vector includes: code change frequency, code complexity, historical defect density, API error rate, and negative user feedback.

4. The method according to claim 1, characterized in that, The process of inputting the risk feature vector into the risk assessment model, outputting multi-dimensional risk indicators, and calculating a comprehensive risk score by fusing the multi-dimensional risk indicators and their dynamic weights based on an adaptive weighted fusion model includes: The risk feature vector is input into a composite machine learning model consisting of a random forest model, a gradient boosting decision tree model, and a support vector machine model. Each model performs independent analysis from the perspectives of breadth scanning, depth mining, and precise classification, and outputs a first risk indicator, a second risk indicator, and a third risk indicator accordingly. The first, second, and third risk indicators are used as inputs to the adaptive weighted fusion model, and the weights of each risk indicator are dynamically adjusted based on the performance of the random forest model, gradient boosting decision tree model, and support vector machine model to output a comprehensive risk score. The weighting adjustment formulas for the first, second, and third risk indicators are as follows: It is the weight of the j-th risk indicator at time t; It is the weight learning rate, used to control the step size of weight adjustment; It is the observed value or deviation from the expected value of the j-th risk indicator at time t; In the adaptive weighted fusion model, the formula for calculating the comprehensive risk score is: It is a comprehensive risk score at time t; It is the normalized value of the j-th risk indicator at time t.

5. The method according to claim 4, characterized in that, The dynamic update is used to classify risk score thresholds into risk levels. Based on a preset mapping rule between risk levels and testing strategies, a testing strategy matching the current risk level is automatically generated, and the testing strategy is automatically executed by calling the test suite. This includes: The risk scoring threshold is dynamically adjusted using an online gradient descent method. in, This is the updated risk score threshold; It is the old risk scoring threshold; It is the learning rate for the scoring threshold, used to control the smoothness of risk scoring threshold adjustment; It is the deviation between the actual value and the predicted value of the risk scoring threshold; Based on the calculated first and second risk score thresholds, risk score intervals are divided, and corresponding test strategies are triggered according to the risk level of each risk score interval. When the overall risk score is greater than the first risk score threshold, it is considered a high-risk level, triggering comprehensive regression testing, performance testing, and security scanning testing. When the second risk score threshold ≤ the comprehensive risk score ≤ the first risk score threshold, it is a medium risk level, triggering the core function regression test; When the overall risk score is less than the second risk score threshold, it is considered a low-risk level, triggering a smoke test or a build-and-verify (BVT) test.

6. The method according to any one of claims 1 to 5, characterized in that, The collected test execution results and online operation data are used as training data to periodically optimize the parameters of the risk assessment model, including: During the process of calling the test suite from the CI / CD pipeline to execute the corresponding test strategy, defect data related to test failures and performance bottleneck data in online production are collected and stored in the database; The defect data and performance bottleneck data are cleaned and labeled, and then transformed into a format suitable for training machine learning models to build a training dataset. When the next training cycle of the risk assessment model is triggered, the training dataset is input into the risk assessment model to adjust and optimize the model parameters.

7. An adaptive multi-dimensional risk testing strategy design system, characterized in that, The system includes: The data processing module is used to obtain raw data from multiple heterogeneous data sources throughout the software development and operation lifecycle when the risk assessment service is triggered by the continuous integration (CI) / continuous deployment (CD) pipeline, and to preprocess the raw data to generate risk feature vectors. The risk prediction module is used to input the risk feature vector into the risk assessment model, output multi-dimensional risk indicators, and calculate a comprehensive risk score by fusing the multi-dimensional risk indicators and their dynamic weights based on an adaptive weighted fusion model. The test execution module is used to dynamically update the risk scoring threshold used to classify risk levels, automatically generate a test strategy that matches the current risk level according to the preset mapping rules between risk levels and test strategies, and call the test suite to automatically execute the test strategy. The model optimization module is used to collect test execution results and online running data as training data, and to periodically optimize the parameters of the risk assessment model.

8. The system according to claim 7, characterized in that, The risk prediction module includes: The composite machine learning module is used to input the risk feature vector into a composite machine learning model consisting of a random forest model, a gradient boosting decision tree model, and a support vector machine model. Each model performs independent analysis from the perspectives of breadth scanning, depth mining, and precise classification, and outputs a first risk indicator, a second risk indicator, and a third risk indicator accordingly. The weighted fusion module is used to take the first risk indicator, the second risk indicator, and the third risk indicator as input to the adaptive weighted fusion model, and dynamically adjust the weight of each risk indicator according to the performance of the random forest model, the gradient boosting decision tree model, and the support vector machine model, and output a comprehensive risk score. The weighting adjustment formulas for the first, second, and third risk indicators are as follows: It is the weight of the j-th risk indicator at time t; It is the weight learning rate, used to control the step size of weight adjustment; It is the observed value or deviation from the expected value of the j-th risk indicator at time t; In the adaptive weighted fusion model, the formula for calculating the comprehensive risk score is: It is a comprehensive risk score at time t; It is the normalized value of the j-th risk indicator at time t.

9. The system according to claim 7, characterized in that, The test execution module includes: The threshold adjustment module is used to dynamically adjust the risk score threshold using an online gradient descent method. in, This is the updated risk score threshold; It is the old risk scoring threshold; It is the learning rate for the scoring threshold, used to control the smoothness of risk scoring threshold adjustment; It is the deviation between the actual value and the predicted value of the risk scoring threshold; The test strategy mapping module is used to divide the risk score interval according to the calculated first risk score threshold and second risk score threshold, and trigger the corresponding test strategy according to the risk level of the risk score interval. When the overall risk score is greater than the first risk score threshold, it is considered a high-risk level, triggering comprehensive regression testing, performance testing, and security scanning testing. When the second risk score threshold ≤ the comprehensive risk score ≤ the first risk score threshold, it is a medium risk level, triggering the core function regression test; When the overall risk score is less than the second risk score threshold, it is considered a low-risk level, triggering a smoke test or a build-and-verify (BVT) test.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed, implement the method described in any one of claims 1-6.

Images