Automatic testing method, automatic testing system, computer equipment and medium

Abstract

The invention discloses an automatic testing method, an automatic testing system, computer equipment and a medium. The automatic test method comprises the steps that in response to a test starting instruction, a test instruction dynamic arrangement engine dynamically generates a test instruction sending strategy based on signal quality and sends a test instruction to tested equipment according to the test instruction sending strategy; response information about the test instruction returned by the tested device is received, and the multi-modal response analyzer matches a corresponding identification network model based on the type of the response information so as to analyze the response information by using the identification network model; and generating a test report based on the test instruction and the analyzed response information. Through the mode, the sending sequence and interval of the test instruction can be automatically optimized, manual intervention is eliminated, the test efficiency is improved, hybrid coding response analysis of the binary stream or the text is realized, the field adaptability is improved, the test accuracy is enhanced, the misjudgment rate is reduced, and the key defect detection rate is improved.

Description

Technical Field

[0001] The present application relates to the field of communication module research and development and production testing, and in particular to an automated testing method, an automated testing system, computer equipment and media. Background Art

[0002] In the field of communication module R&D and production testing, the AT command set, as the core interactive protocol for device control, has long relied on manual serial port tools for its testing and verification. Traditional testing methods require engineers to send commands one by one and visually compare the response results. This method has significant drawbacks such as low manual operation efficiency, low error tolerance, and difficulty in assessing test coverage. Although some automated testing tools have attempted to replace manual operations with script execution, their rigid testing processes are not adaptable to dynamic testing scenarios and face the following three major technical bottlenecks:

[0003] Poor environmental adaptability: Existing tools require customized development of adaptation layers for different communication protocols (such as UART / USB / SPI), resulting in high cross-platform deployment costs.

[0004] Weak defect traceability: When a test fails, only the original log output is provided, lacking causal reasoning of the abnormal event chain and generation of repair suggestions;

[0005] Lack of risk prediction mechanism: The potential risk level of test cases cannot be dynamically assessed based on historical test data, resulting in a high rate of missed detection of critical defects.

[0006] In recent years, the application of AI technology in software engineering has provided new insights into solving these problems. However, existing AI testing solutions primarily focus on general, protocol-independent scenarios and have yet to effectively address specific challenges in communication module testing, such as the strong timing dependencies of instruction sequences, multimodal parsing of response results (such as mixed binary stream and text encoding), and capturing transient hardware anomalies. Summary of the Invention

[0007] This application mainly provides an automated testing method, an automated testing system, computer equipment and media to solve the problems of high manual dependence, insufficient domain adaptability and lack of risk assessment in communication module testing.

[0008] To solve the above technical problems, the present application adopts a technical solution: providing an automated testing method. The method includes: responding to a test start instruction, a test instruction dynamic orchestration engine dynamically generating a test instruction sending strategy based on signal quality, and sending test instructions to the device under test according to the test instruction sending strategy;

[0009] receiving response information regarding the test instruction returned by the device under test, and matching a corresponding identification network model based on a type of the response information by a multimodal response parser to parse the response information using the identification network model;

[0010] A test report is generated based on the test instruction and the parsed response information.

[0011] In an optional implementation of the embodiment of the present application, before generating a test report based on the test instruction and the parsed response information, the automated testing method further includes:

[0012] Identify the exception code in the parsed response information, and generate a defect repair strategy for the exception code based on a knowledge graph; the knowledge graph includes a mapping relationship between test instructions, exception codes, and defect repair strategies;

[0013] The test report also includes the defect repair strategy for the abnormal code.

[0014] In an optional implementation of the embodiment of the present application, before generating a test report based on the test instruction and the parsed response information, the automated testing method further includes:

[0015] Generate a risk matrix based on the defect propagation probability model and the parsed response information; the risk matrix is ​​the risk weight of the test case, and the test instructions constitute the test case;

[0016] Calculate the system-level failure probability based on the risk matrix and vulnerability library to generate a risk assessment report;

[0017] The test report also includes the risk assessment report.

[0018] In an optional implementation of the embodiment of the present application, after generating the risk assessment report, the automated testing method further includes:

[0019] The test instruction dynamic scheduling engine optimizes the test instruction sending strategy based on signal quality and the risk assessment report.

[0020] In an optional implementation of the embodiment of the present application, before the test instruction dynamic orchestration engine generates a test instruction sending strategy based on signal quality, the automated testing method further includes:

[0021] The protocol analyzer sends code streams supported by several communication protocols to the device under test and collects return codes to identify the communication protocol supported by the device under test;

[0022] Establishing a communication connection with the device under test using a communication protocol supported by the device under test;

[0023] The signal quality of the communication with the device under test is obtained.

[0024] In an optional implementation of the embodiment of the present application, the test instruction sending strategy includes a test instruction sending interval and a test instruction sending sequence; the test instruction dynamic orchestration engine dynamically generates the test instruction sending strategy based on signal quality, including:

[0025] The test instruction dynamic scheduling engine dynamically generates the test instruction sending interval based on the signal quality;

[0026] The test instruction dynamic arrangement engine dynamically generates the test instruction sending sequence based on the signal quality and the instruction sequence optimization model; the instruction sequence optimization model is obtained based on historical test instruction training.

[0027] In an optional implementation of the embodiment of the present application, the recognition network model includes a text recognition network model and a binary stream recognition network model; the multimodal response parser matches the corresponding recognition network model based on the type of the response information to parse the response information using the recognition network model, including:

[0028] If the response information is text, the multimodal response parser matches the corresponding text recognition network model to parse the response information using the text recognition network model;

[0029] If the response information is a binary stream, the multimodal response parser matches the corresponding binary stream recognition network model to parse the response information using the binary stream recognition network model.

[0030] In an optional implementation manner of the embodiment of the present application, after receiving the response information regarding the test instruction returned by the device under test, the automated testing method further includes:

[0031] The multimodal response parser extracts the timing characteristics of the response information based on the time interval between the test instruction and the corresponding response information;

[0032] Identifying abnormal timing features among the timing features of the response information, and generating a defect repair strategy for the abnormal timing features based on a knowledge graph; the knowledge graph also includes a mapping relationship between the abnormal timing features and the defect repair strategy;

[0033] The test report also includes a defect repair strategy for the abnormal timing characteristics.

[0034] To solve the above technical problems, another technical solution adopted by this application is to provide an automated testing system, which includes a protocol parser, a dynamic orchestration engine, a multimodal response parser, a knowledge graph module and a risk assessment module:

[0035] The protocol analyzer is used to send code streams supported by several communication protocols to the device under test and collect return codes to identify the communication protocol supported by the device under test; establish a communication connection with the device under test using the communication protocol supported by the device under test; and obtain the signal quality of the communication with the device under test;

[0036] The test instruction dynamic orchestration engine is used to respond to the test start instruction, dynamically generate a test instruction sending strategy based on the signal quality, and send the test instruction to the device under test according to the test instruction sending strategy;

[0037] The multimodal response parser is configured to receive response information regarding the test instruction returned by the device under test, match a corresponding identification network model based on a type of the response information, and parse the response information using the identification network model;

[0038] The knowledge graph module is used to identify the exception code in the parsed response information and generate a defect repair strategy for the exception code based on the knowledge graph; the knowledge graph includes a mapping relationship between test instructions, exception codes and defect repair strategies;

[0039] The risk assessment module is used to generate a risk matrix based on the defect propagation probability model and the parsed response information; the risk matrix is ​​the risk weight of the test case, and the test instructions constitute the test case; based on the risk matrix and the vulnerability library, the system-level failure probability is calculated to generate a risk assessment report.

[0040] To solve the above technical problems, another technical solution adopted in this application is: providing a computer device, including a memory, a processor and a computer program stored in the memory, characterized in that the processor executes the computer program to implement the steps of the above-mentioned automated testing method.

[0041] In order to solve the above technical problems, another technical solution adopted in this application is: providing a computer-readable storage medium on which a computer program is stored, characterized in that when the computer program is executed by a processor, the steps of the above-mentioned automated testing method are implemented.

[0042] The beneficial effects of the present application are as follows: Different from the prior art, the present application discloses an automated testing method, an automated testing system, a computer device, and a medium. The automated testing method responds to a test start instruction, dynamically generates a test instruction sending strategy based on signal quality through a test instruction dynamic orchestration engine, and sends test instructions to the device under test according to the test instruction sending strategy, automatically optimizing the sending sequence and interval of the test instructions, realizing automated test execution, eliminating manual intervention, and improving test efficiency; by receiving response information regarding the test instruction returned by the device under test, a multimodal response parser matches a corresponding recognition network model based on the type of the response information, and uses the recognition network model to parse the response information, thereby realizing mixed encoding response parsing of binary streams or text, improving domain adaptability, enhancing test accuracy, reducing the false positive rate, and improving the key defect detection rate. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present application. Those skilled in the art can also derive other drawings based on these drawings without inventive work, among which:

[0044] Figure 1 This is a flow chart of the automated testing method according to the first embodiment of the automated testing method provided by the present application;

[0045] Figure 2 This is a protocol adaptation flow chart of Example 1 of the automated testing method provided by this application;

[0046] Figure 3 This is an instruction optimization flow chart of the first embodiment of the automated testing method provided by this application;

[0047] Figure 4 This is a schematic diagram of the repair strategy generation process of the first embodiment of the automated testing method provided by this application;

[0048] Figure 5 This is a schematic diagram of the risk assessment feedback process of Example 1 of the automated testing method provided by this application;

[0049] Figure 6 This is a schematic diagram of the virtual hierarchical division of automated testing in Example 1 of the automated testing method provided by this application;

[0050] Figure 7 This is a schematic diagram of the automated test data flow of the first embodiment of the automated test method provided by this application;

[0051] Figure 8 This is a schematic diagram of the structure of the automated testing system of Example 2 of the automated testing system provided in this application. DETAILED DESCRIPTION

[0052] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0053] The terms "first", "second" and "third" in the embodiments of the present application are only used for descriptive purposes and are not to be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first", "second" and "third" may explicitly or implicitly include at least one of the features. In the description of the present application, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device comprising a series of steps or units is not limited to the listed steps or units, but may optionally also include steps or units that are not listed, or may optionally also include other steps or units inherent to these processes, methods, products or devices.

[0054] References to "embodiments" herein mean that a particular feature, structure, or characteristic described in connection with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various places in the specification does not necessarily refer to the same embodiment, nor does it constitute an independent or alternative embodiment that is mutually exclusive of other embodiments. It is understood, both explicitly and implicitly, by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0055] Example 1

[0056] This application provides an automated testing method, referring to Figure 1 , Figure 1 : is a flow chart of an automated testing method according to Embodiment 1 of the automated testing method provided by the present application, and the automated testing method includes:

[0057] S10: In response to the test start instruction, the test instruction dynamic arrangement engine dynamically generates a test instruction sending strategy based on the signal quality, and sends the test instruction to the device under test according to the test instruction sending strategy.

[0058] In the above step S10, after responding to the test start instruction, refer to Figure 2 , Figure 2 This is a protocol adaptation flow chart of the first embodiment of the automated testing method provided by the present application. Before the test instruction dynamic orchestration engine generates a test instruction sending strategy based on signal quality, the automated testing method further includes:

[0059] S11: The protocol analyzer sends code streams supported by several communication protocols to the device under test and collects the return codes to identify the communication protocols supported by the device under test.

[0060] S12: Establishing a communication connection with the device under test using a communication protocol supported by the device under test.

[0061] S13: Obtain the signal quality of the communication with the device under test.

[0062] In an embodiment of the present application, after the test is started, the communication protocol needs to be adapted to the device under test first. In the above step S11, a code stream supported by several communication protocols is sent to the device under test through the protocol parser, such as code streams supported by communication protocols such as UART, USB and SPI; after sending it to the device under test, the protocol parser receives the return code returned by the device under test. For example, code streams supported by 6 different communication protocols are sent to the device under test, and the protocol parser receives a return code supported by one of the communication protocols, and then identifies the communication protocol type of the return code to determine the communication protocol supported by the device under test. Among them, the protocol parser can be a model that includes a pre-built protocol feature code library. When parsing the return code, it can find the corresponding protocol type in the protocol feature code library according to the characteristics of the return code (such as the first byte characteristics of the return code).

[0063] In step S12 above, after determining the communication protocols supported by the device under test, a communication connection is established with the device under test using the communication protocols supported by the device under test. The automated testing method provided by this application can support plug-and-play of multiple communication protocols such as UART, USB, and SPI. When establishing communication connections with multiple devices under test that support different communication protocols, different communication protocols can be quickly switched, thereby shortening the protocol switching time.

[0064] After establishing a communication connection with the device under test, the automated testing method provided in this application can also match the communication rate of the device under test. This can be done by determining the standard communication rate corresponding to the communication protocol based on the communication protocol, or by using a dynamic baud rate calibration algorithm based on the baud rate deviation of the acquired return code to automatically detect or adjust the baud rate during communication and scan for the optimal communication rate.

[0065] In step S13, the protocol analyzer also obtains the signal quality (i.e., channel quality) of the return code when parsing the return code, that is, the signal quality of the communication with the device under test. The signal quality includes signal strength, channel noise, baud rate deviation, and CRC error rate (through CRC error pattern recognition).

[0066] Unlike the prior art, in which a customized adaptation layer is required for each communication protocol to be tested for devices supporting different communication protocols, and the test process lacks dynamic adjustment capabilities, the automated testing method provided by this application uses a protocol parser to send code streams supported by several communication protocols to the device under test and collects the return code to identify the communication protocol supported by the device under test. This achieves rapid identification of communication protocols and supports plug-and-play of multiple communication protocols, allowing for rapid switching between different communication protocols, thereby shortening the protocol switching time and improving the dynamic adjustment capabilities of the test process.

[0067] In the embodiment of the present application, the test instruction sending strategy includes a test instruction sending interval and a test instruction sending sequence. Figure 3 , Figure 3 This is a flowchart of instruction optimization of the first embodiment of the automated testing method provided by this application. In step S10, the test instruction dynamic scheduling engine dynamically generates a test instruction sending strategy based on signal quality, including:

[0068] S14: The test instruction dynamic scheduling engine dynamically generates a test instruction sending interval based on the signal quality.

[0069] S15: The test instruction dynamic scheduling engine dynamically generates a test instruction sending sequence based on signal quality and an instruction sequence optimization model, wherein the instruction sequence optimization model is trained based on historical test instructions.

[0070] After the test is started, multiple test commands are sent to the device under test to test its response functions. For example, if the device under test is a mobile phone, the entire mobile phone testing process requires testing various functions and performance of the phone, such as the phone's call function, SMS function, video function, etc. For each function test, it is necessary to send several test commands (AT commands) to the phone under test and receive the corresponding response information one by one.

[0071] After obtaining the signal quality of the communication with the device under test, in step S14, the dynamic test command scheduling engine analyzes the signal quality and dynamically generates a test command sending interval based on the signal quality. For example, when testing the SMS function of a mobile phone, since the signal quality of the communication with the tested mobile phone has been obtained, the signal quality analysis reveals that the signal quality is poor, and it takes a long time to receive the return code. In this case, the dynamic test command scheduling engine will increase the test command sending interval. For another example, if the signal quality analysis finds that the signal quality is excellent and the return code can be received quickly, the dynamic test command scheduling engine will shorten the test command sending interval, thereby reducing the waiting time during the test process.

[0072] The test commands sent to the device under test must be sent one by one in a specific order to ensure successful execution. For example, when testing a mobile phone's SMS function, you first need to send a test command to confirm whether the SMS module in the tested phone is responding normally. Then, you need to send a test command to instruct the tested phone to set the SMS text mode. Then, you need to send a test command with the target number and SMS content to instruct the tested phone to send the SMS, and so on.

[0073] The automated testing method provided by the present application trains an instruction sequence optimization model based on successful execution cases of historical test instructions. In the above step S15, the test instruction dynamic orchestration engine inputs the generated initial test instructions into the trained instruction sequence optimization model, which can optimize the sequence of each initial test instruction, predict the priority of each test instruction, and obtain the test instruction sending sequence. Among them, the instruction sequence optimization model can be constructed based on a long short-term memory (LSTM) network, or it can be constructed based on an LSTM network and a Q-learning algorithm. The test instructions, historical instruction execution success rate, and current signal quality are input into the LSTM network, and the predicted test instruction sending sequence is output. At the same time, the test path is optimized in combination with the Q-learning algorithm.

[0074] Different from the technical solutions of automated testing tools in the prior art, which cannot dynamically optimize the instruction sending strategy according to the test environment (such as network jitter and hardware anomalies, etc.), rely on manual experience, and need to execute all use cases sequentially, the test efficiency is low. The automated testing method provided by this application dynamically generates the test instruction sending interval based on signal quality through the test instruction dynamic orchestration engine, and dynamically generates the test instruction sending sequence based on the signal quality and instruction sequence optimization model, thereby realizing the automatic optimization of the test instruction sequence and the sending interval. Compared with traditional tools, the test time is reduced, 7×24 hours of unattended testing is supported, the test efficiency is improved, invalid test paths are avoided, the misjudgment rate is reduced, the fixed test process of traditional tools is broken through, and an environment-adaptive test strategy is realized.

[0075] S20: receiving response information regarding the test instruction returned by the device under test, and the multimodal response parser matches the corresponding recognition network model based on the type of the response information to parse the response information using the recognition network model.

[0076] In step S20, the recognition network model includes a text recognition network model and a binary stream recognition network model. The multimodal response parser matches the corresponding recognition network model based on the type of the response information to parse the response information using the recognition network model, specifically including:

[0077] If the response information is text, the multimodal response parser matches the corresponding text recognition network model to parse the response information using the text recognition network model;

[0078] If the response information is a binary stream, the multimodal response parser matches the corresponding binary stream recognition network model to parse the response information using the binary stream recognition network model.

[0079] During testing, although a communication protocol has been adapted, the protocol's transmission stream still has multiple formats. For example, serial communication includes both binary and text streams. Therefore, the automated testing method provided in this application builds a multimodal response parser to determine the format type of the response information and parse the text and binary stream responses separately.

[0080] The multimodal response parser includes two recognition network models: a text recognition network model and a binary stream recognition network model. After the response information is input into the multimodal response parser, the type of the response information is first determined. This can be done by using the characteristics of the first byte of the response information to distinguish between text-type and binary-stream-type response information. The response information is then parsed using the corresponding recognition network model to obtain the parsed response information. The multimodal response parser can not only identify and parse text-type or binary-stream-type response information, but also identify and parse response information that is a mixture of text and binary-stream formats.

[0081] Among them, the multimodal response parser can be constructed using a BERT+CNN dual-channel hybrid architecture, and collaborative parsing can be achieved through a dynamic routing mechanism. The text recognition network model can be a Bidirectional Encoder Representation (BERT) fine-tuned model, which extracts keywords from the response information of the test instruction through the entity recognition layer (such as "ERROR 502" is mapped to a memory overflow error) to obtain a semantic feature vector; the binary stream recognition network model can be a feature extraction network of a convolutional neural network (CNN), which identifies the data frame header features (synchronization word, length field) of the response information of the test instruction through the frame structure parsing layer to obtain a binary feature vector. Among them, the dynamic routing mechanism refers to automatically switching the parsing path according to the characteristic value of the first byte of the response information (such as 0xA5 for binary frame, 0x24 for ASCII text), avoiding the limitation of the traditional model that requires pre-specified data types.

[0082] Unlike the existing technology, which requires pre-specifying the type of response information for parsing response information and cannot achieve mixed parsing, the automated testing method provided by this application constructs a multimodal response parser, matches the corresponding recognition network model based on the type of response information, and uses the recognition network model to parse the response information. It can automatically identify text and binary stream type response information and parse them separately, achieving multimodal mixed parsing, solving the problem of parsing mixed responses of text and binary streams in communication module testing, and improving test efficiency.

[0083] In the present application, refer to Figure 4 , Figure 4 This is a schematic diagram of a repair strategy generation process of the first embodiment of the automated testing method provided by the present application. Before generating a test report based on the test instructions and the parsed response information, the automated testing method further includes:

[0084] S21: Identify the exception code in the parsed response information and generate a defect repair strategy for the exception code based on the knowledge graph, wherein the knowledge graph includes a mapping relationship between test instructions, exception codes, and defect repair strategies.

[0085] In an embodiment of the present application, the test report also includes a defect repair strategy for the abnormal code.

[0086] In the above-mentioned step S21, after obtaining the response information after parsing, the response information after parsing is compared and an exception is marked, and the exception in the response information after parsing is identified. After the response information after parsing is marked as abnormal, the exception code (error code) corresponding to the abnormality can be found in the historical test database. For example, a test instruction for obtaining positioning is sent to the tested mobile phone, the tested mobile phone obtains the current positioning and returns the corresponding response information, and the positioning obtained by parsing the response information is compared with the positioning of the tested mobile phone in fact. If the comparison finds that the error is within the preset range, it can be considered that there is no abnormality. If the error exceeds the preset range, then find and mark which test link has an abnormality, for example, it may be that an abnormality occurred when starting the positioning module, or an abnormality occurred when obtaining the positioning.

[0087] After identifying the exception code in the parsed response information, the defect repair strategy for the exception code can be found based on the knowledge graph, and the search results of the exception code and the corresponding defect repair strategy can be added to the test report. Among them, the knowledge graph is constructed through a graph neural network (GCN) and historical test data. It constructs a knowledge graph containing the mapping relationship between test instructions (test instruction grammatical rules), exception codes and defect repair strategies corresponding to the exception codes. Among them, the test instructions are mainly used to compare the grammatical rules of the test instructions to determine whether there are defects in the grammatical rules of each test instruction in this test. In addition, if the defect repair strategy corresponding to the exception code cannot be found, the defect repair strategy corresponding to the closest exception code can be found to generate repair suggestions.

[0088] Different from the existing technology that relies on manual experience to judge the response information of the test instruction, manually identify the anomalies in the response information and judge the corresponding repair strategy, which has a high misjudgment rate and a low key defect detection rate. The automated testing method provided by this application automatically identifies the exception code in the parsed response information through a knowledge graph with a large number of built-in fault repair strategies, generates a defect repair strategy for the exception code based on the knowledge graph, and automatically outputs an executable repair plan (such as firmware upgrade instructions and hardware adjustment suggestions, etc.), realizing the intelligent mapping of error codes to solutions, improving the efficiency of fault root cause location, reducing fault diagnosis time, increasing the adoption rate of repair strategies, improving test accuracy, and improving the key defect detection rate. It does not rely on manual experience, saves labor costs, and reduces the misjudgment rate.

[0089] In the present application, refer to Figure 4 , Figure 4 : is a schematic diagram of a repair strategy generation process of the first embodiment of the automated testing method provided by the present application. Before generating a test report based on the test instruction and the parsed response information, the automated testing method further includes:

[0090] S22: The multimodal response parser extracts the timing characteristics of the response information based on the time interval between the test instruction and the corresponding response information.

[0091] S23: Identify abnormal timing features in the timing features of the response information, and generate a defect repair strategy based on the abnormal timing features based on the knowledge graph. The knowledge graph also includes a mapping relationship between the abnormal timing features and the defect repair strategy.

[0092] In an embodiment of the present application, the test report also includes a defect repair strategy for the abnormal timing characteristics.

[0093] During the test process, in addition to comparing the results and marking exceptions on the parsed response information to obtain the results of the functional test, you can also obtain the results of the performance test.

[0094] In step S22, the time interval between the test instruction and the corresponding response information is used to extract the timing characteristics of the response information and obtain the performance test results. The time interval between the test instruction and the corresponding response information can be obtained from the test log. For example, if the time interval between the test instruction and the corresponding response information exceeds a preset threshold, it is considered that the time interval between the test instruction and the corresponding response information is too large, and the test performance result does not meet the requirements.

[0095] In the above step S23, the abnormal timing features in the extracted timing features of the response information are marked, and the defect repair strategies corresponding to the abnormal timing features are also searched in the knowledge graph. The extracted timing features of the response information and the defect repair strategies corresponding to the abnormal timing features can be added to the test report.

[0096] Different from the existing technology that relies on manual experience to analyze the timing features in the test log and determine the corresponding defect repair strategy, which has a high false positive rate, the automated testing method provided by this application uses a multimodal response parser to extract the timing features of the response information based on the time interval between the test instruction and the corresponding response information. Based on the timing features, it automatically identifies the defect repair strategy corresponding to the abnormal timing features in the knowledge graph and outputs executable repair suggestions, which improves the test accuracy and reduces the false positive rate caused by manual intervention.

[0097] It should be noted that the execution order of the above-mentioned step S21 and steps S22 to S23 can be executed successively or in parallel, which is not limited here.

[0098] In the present application, refer to Figure 5 , Figure 5 This is a risk assessment feedback flow diagram of the first embodiment of the automated testing method provided by the present application. Before generating a test report based on the test instructions and the parsed response information, the automated testing method further includes:

[0099] S24: Generate a risk matrix based on the defect propagation probability model and the parsed response information, wherein the risk matrix is ​​the risk weight of the test case, and the test instructions constitute the test case.

[0100] S25: Calculate the system-level failure probability based on the risk matrix and vulnerability library to generate a risk assessment report.

[0101] In an embodiment of the present application, the test report also includes a risk assessment report.

[0102] In step S24, after obtaining the parsed response information, the parsed response information is input into the trained defect propagation probability model to generate a risk matrix. It should be noted that for a device under test, such as a mobile phone, multiple tests are typically required, including testing the phone's phone function, SMS function, and positioning function, among others. During the testing of a single function, multiple test instructions are typically required to test each test point of that function. After all test points for all test items in all functions of a device under test have been tested, a risk matrix is ​​generated based on all the parsed response information obtained.

[0103] The principle of generating a risk matrix based on the defect propagation probability model is to obtain the corresponding specific failure information based on the failure results (abnormal results) in all parsed response information, analyze the failed steps and the functions tested by the corresponding test instructions based on the failure information, and finally calculate the diffusion path of the defects in each failure result based on the state transition probability matrix through correlation calculation, derive the range of possible impact, dynamically calculate the risk weight of the test case, and generate a risk matrix. The state transition probability matrix is ​​generated based on historical defect data training, and the defect propagation probability model can be constructed using a Markov model. The risk levels of multiple test items are preset in the defect propagation probability model, and the risk levels are quantified.

[0104] In step S25, after generating the risk matrix, the system-level failure probability can be calculated based on the risk matrix and the associated vulnerability library. Finally, a risk assessment report is generated based on the risk matrix and the system-level failure probability. The principle of calculating the system-level failure probability is to reversely query the vulnerability library based on the failure results in all parsed response information to find the same or most relevant historical issues. Based on the failure of the historical issues, the impact of the failure results in the parsed response information is directly derived. The system-level failure probability is calculated to provide early warning of the failure risk of key modules and identify potential security risks.

[0105] Different from the existing technology that only implements single-link automated testing (such as use case generation or log analysis), and does not build a full-link system from defect prediction to root cause location to quality assessment. The automated testing method provided by this application generates a risk matrix based on the defect propagation probability model and the parsed response information, and calculates the system-level failure probability based on the risk matrix and the vulnerability library to generate a risk assessment report. It builds a full-link closed-loop risk assessment system from defect prediction to root cause location to quality assessment, improves the key defect detection rate and the accuracy of system-level failure risk prediction, reduces the quality scoring error, and improves the risk prediction capability.

[0106] In the present application, refer to Figure 5 , Figure 5 This is a schematic diagram of the risk assessment feedback process of the first embodiment of the automated testing method provided by this application. After generating the risk assessment report, the automated testing method further includes:

[0107] S26: The test instruction dynamic scheduling engine generates and optimizes the test instruction sending strategy based on the signal quality and risk assessment report.

[0108] The automated testing method provided herein incorporates a feedback adjustment mechanism. After generating a risk assessment report, in step S26, the dynamic test command orchestration engine automatically optimizes and adjusts the test command sending strategy based on the risk assessment results. For example, if the risk assessment results indicate poor network quality, the dynamic test command orchestration engine automatically adjusts the test command sending interval or data length to improve system stability.

[0109] Furthermore, throughout the entire test process, the dynamic test command orchestration engine analyzes the current signal instructions in real time based on the test command response information, dynamically adjusting the test command sending interval and retry strategy. For example, during a test, if the current default network status is excellent, the dynamic test command orchestration engine adjusts the test command sending interval to 1 second, and the retry strategy is that 2 out of 3 successful tests indicate a successful test. However, after analyzing the test command response information, it is found that the current network signal is weak. In this case, the dynamic test command orchestration engine automatically adjusts the test command sending interval to 2 seconds, thereby increasing the device buffering time, and reduces the retry strategy to detect a success after 1 reply.

[0110] Unlike conventional testing tools in the prior art, which cannot adjust test strategies based on real-time signal quality and have fixed test strategies requiring manual adjustment, the automated testing method provided by this application optimizes test instruction sending strategies based on signal quality and risk assessment reports through a dynamic test instruction orchestration engine. This dynamically optimizes test strategies through risk assessment result feedback, builds a closed-loop risk assessment system, improves dynamic testing capabilities, and reduces maintenance costs.

[0111] S30: Generate a test report based on the test instruction and the parsed response information.

[0112] In the above step S30, a test report is generated based on the parsed response information obtained in the above step S20 and the corresponding test instructions, wherein the test report may also include the exception code corresponding to the abnormal result in the parsed response information, and the defect repair strategy corresponding to the exception code; it may also include the timing characteristics extracted by the time interval between the test instruction and the corresponding response information; it may also include a risk matrix, a system-level failure probability, and a risk assessment report generated by calculating the system-level failure probability based on the risk matrix and the vulnerability library.

[0113] In a specific embodiment, referring to Figure 6 , Figure 6 This is a diagram of the virtual hierarchical division of the automated test in the first embodiment of the automated test method provided by this application. The automated test method provided by this application can be performed as follows according to the test process: Figure 6 The virtual layer division is divided into the communication adaptation layer, AI engine layer, and decision-making center layer. The communication adaptation layer is divided into the protocol analysis module and the instruction transceiver engine, the AI ​​engine layer is divided into the dynamic orchestration module and the multimodal analysis module, and the decision-making center layer is divided into the knowledge graph module and the risk assessment module.

[0114] Among them, the protocol parsing module performs operations such as steps S11 to S13, that is, realizes the communication protocol self-adaptation and parses the return code to obtain the signal quality; in the instruction transceiver engine and dynamic arrangement module, the test instruction dynamic arrangement engine performs operations such as steps S14 to S15, that is, the test instruction dynamic arrangement engine dynamically generates a test instruction sending strategy based on the signal quality, and sends the test instruction to the device under test according to the test instruction sending strategy; the multimodal parsing module performs operations such as step S20, that is, receives the response information about the test instruction returned by the device under test, and the multimodal response parser responds to the response information based on the response information The type of the error is matched with the corresponding recognition network model, so as to utilize the recognition network model to parse the response information; the knowledge graph module performs operations such as steps S21 to S23, that is, generating defect repair strategies for exception codes and repair strategies corresponding to timing feature problems based on the knowledge graph; the risk assessment module performs operations such as steps S24 to S25, that is, generating a risk matrix based on the defect propagation probability model and the parsed response information, and calculating the system-level failure probability based on the risk matrix and the vulnerability library to generate a risk assessment report, and the risk assessment report can also be fed back to the instruction transceiver engine to dynamically adjust the test instruction sending interval.

[0115] In a specific embodiment, referring to Figure 7 , Figure 7This is a data flow diagram of the automated testing method of the first embodiment of the automated testing method provided by the present application. The automated testing method may include the following steps S1 to S12:

[0116] S1: Test started.

[0117] S2: Perform communication protocol adaptation to match the communication protocol of the device under test and connect to the device under test.

[0118] S3: Dynamically generate a test instruction sequence through a test instruction dynamic orchestration engine and determine the test instruction sending interval.

[0119] S4: Send test instructions to the device under test and collect response information.

[0120] S5: Determine the response information type through the multimodal response parser.

[0121] S6: Parse the text and binary stream response information using the corresponding recognition network model to obtain parsed response information.

[0122] S7: Compare the parsed response information with the exception mark and determine the corresponding error code.

[0123] S8: Extract the timing features of the test instructions in the log and mark the abnormal timing features in the timing features.

[0124] S9: Match defects in the knowledge graph using error codes and abnormal timing features to determine repair strategies for the same or similar historical defects.

[0125] S10: Generate defect repair strategies corresponding to error codes and abnormal timing characteristics based on the knowledge graph, and generate repair suggestions.

[0126] S11: Input the parsed response information obtained from all test items into the defect propagation probability model to generate a risk matrix, and calculate the system-level failure probability based on the risk matrix and the associated vulnerability library.

[0127] S12: Generate a test report based on the test instructions, the parsed response information, the defect repair strategy of the exception code, the timing characteristics of the response information and the corresponding repair strategy and risk assessment report.

[0128] Different from the problems of high manual dependence, insufficient field adaptability and lack of risk assessment in the test methods in the prior art. The automated testing method provided by the present application responds to the test start instruction, dynamically generates a test instruction sending strategy based on signal quality through the test instruction dynamic orchestration engine, and sends the test instruction to the device under test according to the test instruction sending strategy, automatically optimizes the sending sequence and interval of the test instruction, realizes automated test execution, eliminates manual intervention, and improves test efficiency; by receiving the response information about the test instruction returned by the device under test, the multimodal response parser matches the corresponding recognition network model based on the type of the response information, and uses the recognition network model to parse the response information, thereby realizing mixed encoding response parsing of binary streams or texts, improving field adaptability, enhancing test accuracy, reducing the false positive rate, and improving the key defect detection rate.

[0129] Example 2

[0130] Based on the same inventive concept, the present application also provides an automated testing system, referring to Figure 8 , Figure 8 : This is a schematic diagram of the structure of the automated testing system of the second embodiment of the automated testing system provided by the present application. The automated testing system includes a protocol parser 10, a test instruction dynamic orchestration engine 20, a multimodal response parser 30, a knowledge graph module 40, a risk assessment module 50 and a device under test 60:

[0131] The protocol analyzer 10 is used to send code streams supported by several communication protocols to the device under test and collect return codes to identify the communication protocol supported by the device under test 60; establish a communication connection with the device under test 60 using the communication protocol supported by the device under test 60; and obtain the signal quality of the communication with the device under test 60;

[0132] The test instruction dynamic scheduling engine 20 is used to respond to the test start instruction, dynamically generate a test instruction sending strategy based on the signal quality, and send the test instruction to the device under test 60 according to the test instruction sending strategy;

[0133] The multimodal response parser 30 is used to receive the response information regarding the test instruction returned by the device under test 60, match the corresponding recognition network model based on the type of the response information, and parse the response information using the recognition network model;

[0134] The knowledge graph module 40 is used to identify the exception code in the parsed response information and generate a defect repair strategy for the exception code based on the knowledge graph; the knowledge graph includes a mapping relationship between test instructions, exception codes and defect repair strategies;

[0135] The risk assessment module 50 is used to generate a risk matrix based on the defect propagation probability model and the parsed response information; the risk matrix is ​​the risk weight of the test case, and the test instructions constitute the test case; based on the risk matrix and the vulnerability library, the system-level failure probability is calculated to generate a risk assessment report.

[0136] In the second embodiment, the specific process of implementing automated testing through interaction between the protocol parser 10, the test instruction dynamic orchestration engine 20, the multimodal response parser 30, the knowledge graph module 40, the risk assessment module 50 and the device under test 60 in the automated testing system can refer to the detailed description of steps S10 to S30, S11 to S13, S21 to S23 and S24 to S26 in the automated testing method of the first embodiment above, and any repetitions will not be repeated here.

[0137] Example 3

[0138] Based on the same inventive concept, an embodiment of the present application further provides a computer device, comprising a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the automated testing method described in the above embodiment 1.

[0139] Example 4

[0140] Based on the same inventive concept, an embodiment of the present application further provides a computer-readable storage medium on which a computer program / instruction is stored. When the computer program / instruction is executed by a processor, the automated testing method described in the above embodiment 1 is implemented.

[0141] The above description is merely an embodiment of the present application and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. An automated testing method, characterized in that: include: In response to the test start instruction, the test instruction dynamic orchestration engine dynamically generates a test instruction sending strategy based on the signal quality, and sends the test instruction to the device under test according to the test instruction sending strategy; receiving response information regarding the test instruction returned by the device under test, and matching a corresponding identification network model based on a type of the response information by a multimodal response parser to parse the response information using the identification network model; A test report is generated based on the test instruction and the parsed response information.

2. The automated testing method according to claim 1, wherein: Before generating a test report based on the test instruction and the parsed response information, the automated testing method further includes: Identify the exception code in the parsed response information, and generate a defect repair strategy for the exception code based on a knowledge graph; the knowledge graph includes a mapping relationship between test instructions, exception codes, and defect repair strategies; The test report also includes the defect repair strategy for the abnormal code.

3. The automated testing method according to claim 1 or 2, wherein: Before generating a test report based on the test instruction and the parsed response information, the automated testing method further includes: Generate a risk matrix based on the defect propagation probability model and the parsed response information; the risk matrix is ​​the risk weight of the test case, and the test instructions constitute the test case; Calculate the system-level failure probability based on the risk matrix and vulnerability library to generate a risk assessment report; The test report also includes the risk assessment report.

4. The automated testing method according to claim 3, wherein: After generating the risk assessment report, the automated testing method further includes: The test instruction dynamic scheduling engine optimizes the test instruction sending strategy based on signal quality and the risk assessment report.

5. The automated testing method according to claim 3, wherein: Before the test instruction dynamic scheduling engine generates a test instruction sending strategy based on signal quality, the automated testing method further includes: The protocol analyzer sends code streams supported by several communication protocols to the device under test and collects return codes to identify the communication protocol supported by the device under test; Establishing a communication connection with the device under test using a communication protocol supported by the device under test; The signal quality of the communication with the device under test is obtained.

6. The automated testing method according to claim 1, wherein: The test instruction sending strategy includes a test instruction sending interval and a test instruction sending sequence; The test instruction dynamic scheduling engine dynamically generates a test instruction sending strategy based on signal quality, including: The test instruction dynamic scheduling engine dynamically generates the test instruction sending interval based on the signal quality; The test instruction dynamic arrangement engine dynamically generates the test instruction sending sequence based on the signal quality and the instruction sequence optimization model; the instruction sequence optimization model is obtained based on historical test instruction training.

7. The automated testing method according to claim 1, wherein: The recognition network model includes a text recognition network model and a binary stream recognition network model; The multimodal response parser matches a corresponding recognition network model based on the type of the response information to parse the response information using the recognition network model, including: If the response information is text, the multimodal response parser matches the corresponding text recognition network model to parse the response information using the text recognition network model; If the response information is a binary stream, the multimodal response parser matches the corresponding binary stream recognition network model to parse the response information using the binary stream recognition network model.

8. The automated testing method according to claim 1, wherein: Before generating a test report based on the test instruction and the parsed response information, the automated testing method further includes: The multimodal response parser extracts the timing characteristics of the response information based on the time interval between the test instruction and the corresponding response information; Identifying abnormal timing features among the timing features of the response information, and generating a defect repair strategy for the abnormal timing features based on a knowledge graph; the knowledge graph also includes a mapping relationship between the abnormal timing features and the defect repair strategy; The test report also includes a defect repair strategy for the abnormal timing characteristics.

9. An automated testing system, characterized in that: The automated testing system includes a protocol parser, a test instruction dynamic orchestration engine, a multimodal response parser, a knowledge graph module, and a risk assessment module: The protocol analyzer is used to send code streams supported by several communication protocols to the device under test and collect return codes to identify the communication protocol supported by the device under test; establish a communication connection with the device under test using the communication protocol supported by the device under test; and obtain the signal quality of the communication with the device under test; The test instruction dynamic orchestration engine is used to respond to the test start instruction, dynamically generate a test instruction sending strategy based on the signal quality, and send the test instruction to the device under test according to the test instruction sending strategy; The multimodal response parser is configured to receive response information regarding the test instruction returned by the device under test, match a corresponding identification network model based on a type of the response information, and parse the response information using the identification network model; The knowledge graph module is used to identify the exception code in the parsed response information and generate a defect repair strategy for the exception code based on the knowledge graph; the knowledge graph includes a mapping relationship between test instructions, exception codes and defect repair strategies; The risk assessment module is used to generate a risk matrix based on the defect propagation probability model and the parsed response information; The risk matrix is ​​the risk weight of the test case, and the test instructions constitute the test case; The system-level failure probability is calculated based on the risk matrix and vulnerability library to generate a risk assessment report.

10. A computer device comprising a memory, a processor, and a computer program stored in the memory, wherein: The processor executes the computer program to implement the steps of the automated testing method according to any one of claims 1 to 8.

11. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the automated testing method according to any one of claims 1 to 8 are implemented.

Images