Automatic log analysis method and system based on keywords
By adopting keyword-based automatic log analysis method and system in embedded software testing, and evaluating log analysis results with automatic analysis scripts, the problem of low testing efficiency and accuracy in embedded software testing is solved, and efficient and accurate Test process.
Patent Information
- Application Number
- CN202411468960.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-21
- Publication Date
- 2025-02-11
Smart Images
Figure CN119415489A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of rail transportation, and in particular to a keyword-based automatic log analysis method and system. Background Art
[0002] With the increasing scale and complexity of software, coupled with the increasingly stringent requirements of users for software real-time, reliability, security and availability, back-end development and testing are facing unprecedented challenges. This requires us not only to respond quickly to market changes, but also to ensure the best user experience. The characteristics of embedded software testing are particularly prominent. Its interface types are diverse, including but not limited to digital signals, analog signals and various bus data (such as M-LVDS, CAN, UART), coupled with the complexity of the structure (such as redundant structure MooN) and strict timing requirements, which significantly increase the complexity of the testing process. During the test, it is necessary to accurately control a variety of stimulus signals, and there is often a strict time series dependency between these signals. At the same time, embedded testing is usually limited by resource constraints, such as limited memory, response time and processing speed, which together constitute the key challenges and difficulties of embedded software testing. In summary, in the face of growing software and hardware requirements, back-end testing must ensure software quality while taking into account efficiency and user experience. Especially in the embedded field, its unique testing requirements and resource constraints require it to adopt a more sophisticated and efficient testing strategy.
[0003] Currently, embedded software testing mainly adopts two methods: QEMU semi-virtual simulation environment and target board environment verification. The advantage of QEMU semi-virtual simulation is its asset-light characteristics, which can provide a highly configurable and customizable simulation environment with strong flexibility. However, it has high requirements for the development of the virtual layer kernel, and there are differences between it and the real target machine environment, which may introduce potential risks in industries with extremely strict security requirements. In view of this, for industries with higher security requirements, target board environment verification has become the preferred solution.
[0004] like Figure 1As shown in the figure, in the field of rail transit, the target board verification method is widely used in the testing of software hierarchical structures. The software hierarchical structure includes: application software layer, middleware layer, operating system layer, board support package and driver, and hardware resources. The application software layer includes several application systems for providing use cases to users; the middleware layer provides low-level component services that are not platform-dependent; the operating system layer includes specific platform software and computer network infrastructure, etc.; the board support package and driver are used for hardware initialization and driver access; the hardware resources include the physical components and devices of the computer, which provide the necessary support and environment for software operation. Since changes in the underlying software or hardware resources may have unpredictable effects on the upper application software layer, acceptance testing needs to be implemented in a variety of environments to enhance confidence in the stability and compatibility of the operating system layer. However, the target board environment verification has the disadvantages of uncontrollable data and difficult monitoring. The data in the test process is highly dynamic and unpredictable, and a large amount of data needs to be monitored in real time, which affects the test accuracy and test efficiency. Therefore, a method for automatic log analysis with simple calculation method is needed to improve the test efficiency. Summary of the invention
[0005] The purpose of the present invention is to provide a keyword-based log automatic analysis method and system, which reduces the time consumption of manual calculation, improves the efficiency and accuracy of log analysis results, and enhances the reusability of log automatic analysis scripts.
[0006] In order to achieve the above-mentioned purpose, the present invention provides a keyword-based automatic log analysis method, which writes an automatic analysis script that can characterize and test the input and output relationship of the test case, uses the automatic analysis script to test the test case, obtains the log analysis results, and determines whether the test case passes the test by screening the log analysis results.
[0007] Optionally, the automatic analysis script includes: a precondition script, a hit condition script, and expected results; the precondition is an input stimulus, which can calculate the expected value based on the input and output relationship of the test case; the expected result is the value of the preselected data obtained after the input variable is calculated through the test case; the hit condition script can compare the expected value with the preselected data value to obtain the log analysis result.
[0008] Optionally, the method of calculating the expected value and performing log analysis using the precondition script includes:
[0009] Mark the input variable related to the test case as the first key variable, and input the first key variable into the test case to obtain the expected result of the test case output;
[0010] Mark the variable with the same representation content as the first key variable as the second key variable, and input the second key variable into the precondition script to calculate the expected value calculated by the program;
[0011] The expected result is compared with the expected value calculated by the program through the hit condition script; when the expected result is consistent with the expected value calculated by the program, the output log analysis result is "test passed".
[0012] Optionally, the method of calling the automatic analysis script to obtain the log analysis result further includes: manually calculating the expected value and performing log analysis; the method of manually calculating the expected value and performing log analysis includes:
[0013] Mark the input variable related to the test case as the first key variable, and input the first key variable into the test case to obtain the expected result of the test case output;
[0014] Mark the variable with the same representation content as the first key variable as the second key variable, perform manual calculation on the second key variable according to the input and output relationship of the test case to obtain the manual calculation expected value, and input the manual calculation expected value into the automatic analysis script;
[0015] The expected result is compared with the manually calculated expected value through the hit condition script; when the expected result is consistent with the manually calculated expected value, the output log analysis result is "test passed".
[0016] Optionally, when the expected result is inconsistent with the expected value, the log analysis result is output as "failed the test", and fault value error processing is performed at the same time, and the cause of the fault is output.
[0017] Optionally, the method for obtaining the log analysis result of the test case includes:
[0018] The test case includes: a plurality of steps and a plurality of operation cycles; an automatic analysis script is called for each step in each operation cycle of the test case, and a log analysis result of each step in each operation cycle of the test case is recorded.
[0019] Optionally, the method for filtering the log analysis results includes:
[0020] Filter the steps in each running cycle whose log analysis results are "passed the test" and record the step number and cycle number of the step;
[0021] Traverse each step, filter out the cycle number in each step whose log analysis result is "passed test", and establish a cycle number set in each step whose log analysis result is "passed test"; determine whether the cycle number set of each step includes a cycle that satisfies the user-defined association relationship;
[0022] When the cycle number sets of all steps respectively include the cycles corresponding to the steps that satisfy the user-defined association relationship, the test case passes the test; when the cycle number set of a step does not include the cycle that satisfies the user-defined association relationship, the test case fails the test.
[0023] Optionally, input variables related to the test case are extracted from OMAP dictionary data; wherein the OMAP dictionary data includes: all input variables and output variables required by the test object including several test cases, and the input and output variables of the test object are stored in the form of key-value pairs.
[0024] In addition, another object of the present invention is to provide a keyword-based log automatic analysis system, comprising:
[0025] A simulator, wherein the simulator is an external stimulus for a test object including several test cases, and the variables of the external stimulus are stored in the memory of the log automatic analysis system of the vehicle software testing platform in the form of key-value pairs. The simulator triggers variable assignment at a preset time point to achieve the purpose of inputting variables to the test object;
[0026] A log manager, in which an automatic analysis script is run to detect the object under test; the log manager receives output variables sent by the object under test, stores and processes them in the form of key-value pairs, generates log analysis results, and determines whether the object under test passes the test.
[0027] Optionally, the variables of the external excitation in the simulator are input and output variables related to the object under test.
[0028] Compared with the prior art, the technical solution of the present invention has at least the following beneficial effects:
[0029] The present invention extracts key variables from the full OMAP data of the tested object, writes analysis scripts according to the requirements of the tested case, and uses automatic analysis scripts to evaluate log analysis results, directly outputs the test status of the tested case, effectively reducing the workload of manual inspection, shortening the analysis time of the log, and significantly improving the test efficiency.
[0030] The operation of the automatic analysis script of the present invention is highly dependent on the standardization and stability of the variable names in the log. When the variable requirements and names output by OMAP remain unchanged and a software system is used to store key-value pairs, the automatic analysis script can be reused, which enhances the reusability of the automatic analysis script and provides strong support for the inspection work of testers.
[0031] The automatic analysis script design of the present invention is close to natural language, and clearly integrates the precondition script, hit condition script and expected results. When testing the object under test, the log manager automatically calls and parses these scripts, and the user can perform logical processing on the acquired data and compare the expected results with the preset values. For the part of the expected result that requires complex calculations, the automatic analysis script introduces the automatic calculation function of code logic, and compares it with the expected results to obtain the test results of the object under test. It effectively reduces the time consumption of manual calculations, reduces the errors introduced by manual operations, and thus reduces the error costs, ensuring the accuracy and efficiency of the test process. BRIEF DESCRIPTION OF THE DRAWINGS
[0032] Figure 1 It is a software layered structure diagram in the background technology.
[0033] Figure 2 This is a structural diagram of the keyword-based automatic log analysis system.
[0034] Figure 3 A flowchart of log analysis for the automatic analysis script in the keyword-based log automatic analysis method.
[0035] Figure 4 The log analysis flow chart for the test case steps in the keyword-based automatic log analysis method.
[0036] Figure 5 This is a flowchart for judging the steps and results of test cases in the keyword-based automatic log analysis method. DETAILED DESCRIPTION
[0037] The technical solution of the present invention will be described clearly and completely below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0038] In the description of the present invention, it should be noted that the terms "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc., indicating the orientation or positional relationship, are based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first", "second", and "third" are used for descriptive purposes only, and cannot be understood as indicating or implying relative importance.
[0039] In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.
[0040] like Figure 2 As shown, the present invention utilizes a keyword-based automatic log analysis system (i.e., an on-board software testing platform) to perform acceptance testing on the object under test. The automatic log analysis system includes: various simulators and a log manager. The simulator is an external stimulus for the object under test. The variables of the external stimulus (i.e., input and output variables related to the object under test in the object under test) are stored in the memory of the automatic log analysis system in the form of key-value pairs. The simulator can trigger variable assignment at a preset time point to achieve the purpose of inputting variables to the object under test; the log manager contains a log analysis program for the object under test, which can detect the object under test. The log manager can receive output variables sent by the object under test, and store and process them in the form of key-value pairs to generate feedback logs to determine whether the object under test passes the test. When using the automatic log analysis system for testing, the simulator sends external stimuli to the object under test and assigns values to input variables in the external stimuli at a preset time point; after receiving the input variables, the object under test outputs output variables according to its calculation logic (i.e., the logic required for the object under test to complete a specific function); the log manager stores the output variables in the form of key-value, generates log analysis results based on the output variables of the object under test, and determines whether the object under test passes the test based on the log analysis results.
[0041] The present invention provides a keyword-based log automatic analysis method, which adopts a keyword-driven script writing method for the log manager of the log automatic analysis system, refines complex test steps into a series of concise keywords, and simplifies the process of acceptance test of the object under test. By writing an automatic analysis script that can characterize and test the input and output relationship of the test case, the test case is tested using the automatic analysis script to obtain the log analysis result, and by screening the log analysis result, it is determined whether the test case passes the test. The specific steps include:
[0042] Step 1, initialize the use case step and set the keyword library.
[0043] Initialize the loop count and set OMAP (Open Multimedia Application Platform) dictionary data, wherein the OMAP dictionary data includes all input variables and output variables required by the object under test (i.e., OMAP full data), and the input and output variables of the object under test are stored in the memory of the automatic log analysis system in the form of key-value pairs; since the object under test includes several use cases, when actually testing a use case in the object under test, only the input variables and output variables related to the use case under test need to be provided, therefore, a keyword library is established, and the input variables or output variables related to the use case under test are used as key variables, and the collection of all the key variables is the keyword library.
[0044] Step 2, use keyword variables to write automatic analysis scripts.
[0045] Key variables are divided into two categories according to their uses, namely, the first key variable that can be directly used as the expected value and the second key variable that needs to be calculated to obtain the expected check value. The first key variable and the second key variable have the same representation content, but the way to obtain the expected value is different; the first key variable directly obtains the expected value by inputting the test case, and the second key variable obtains the expected value through the automatic analysis script.
[0046] The automatic analysis script is written using key variables, and the automatic analysis script includes: precondition script, hit condition script, and expected result. Among them, the precondition script is the input stimulus, which represents the relationship between the input and output of the test case; the expected result is the expected output value, that is, the value that can be obtained by the test case through logical calculation of the input variable, which is the target of the automatic analysis script calculation; the hit condition is the standard for verifying whether the automatic analysis script can achieve the expected result.
[0047] When using an automatic analysis script, the first key variable and the second key variable are input into the automatic analysis script, the expected value of the first key variable obtained according to the test case is used as the expected result, the second key variable is input into the precondition script to obtain the expected value calculated by the program, and the hit condition script is used to determine whether the expected result is consistent with the expected value calculated by the program.
[0048] Step 3: Call the automatic analysis script to perform log analysis.
[0049] Step 3.1, establish and call the pre-selected database.
[0050] like Figure 3 As shown, the pre-selected database contains all second key variables, and variables with the same characterization content as those in the pre-selected database are extracted from the OMAP dictionary data as first key variables.
[0051] Step 3.2, use the program to calculate the expected value and perform log analysis.
[0052] The first key variable is input into the test case, and after the logic calculation of the test case, the value of the pre-selected data output by the test case (ie, the expected result in the automatic analysis script) is obtained.
[0053] Enter the second key variable into the precondition script and calculate the expected value calculated by the program.
[0054] The hit condition script compares the value of the pre-selected data with the expected value calculated by the program. When the value of the pre-selected data is consistent with the expected value calculated by the program, the output log analysis result is "passed test", indicating that the inspection item of the stimulus has passed; when the value of the pre-selected data is inconsistent with the expected value calculated by the program, the output log analysis result is "failed test", indicating that the inspection item of the stimulus has failed, and the fault value error processing is performed at the same time, the fault cause is output and recorded in the log manager.
[0055] Among them, when the representation content of the first key variable and the second key variable changes according to project requirements, if the input and output relationship of the test case is determined and the names of the first key variable and the second key variable remain unchanged, the automatic analysis script does not need to be rewritten and can be reused.
[0056] Step 3.3, manually calculate the expected value and perform log analysis.
[0057] The first key variable is input into the test case, and the value of the pre-selected data output by the test case (ie, the expected result in the automatic analysis script) is obtained.
[0058] According to the logical relationship between the input and output of the test case, the expected value is manually calculated and input into the automatic analysis script.
[0059] The value of the pre-selected data is compared with the manually calculated expected value through the hit condition. When the value of the pre-selected data is consistent with the manually calculated expected value, the output log analysis result is "passed test", indicating that the inspection item of the stimulus has passed; when the value of the pre-selected data is inconsistent with the manually calculated expected value, the output log analysis result is "failed test", indicating that the inspection item of the stimulus has not passed, and the fault value error processing is performed at the same time, the fault cause is output and recorded in the log manager.
[0060] Among them, the step 3.2 and step 3.3 can be calculated simultaneously or separately. Users can make decisions based on their own needs and resources. Using automatic analysis scripts to calculate expected values can reduce the situation of human calculation errors and make the program more adaptable. When manually calculating expected values, the script writer does not need to understand the code, but only needs to understand the logic between the input value and output value of the test case.
[0061] Step 4: Analyze the step log of the test case.
[0062] like Figure 4 As shown, when the test case has N steps and runs K cycles, the specific steps of the step log analysis process of the test case include:
[0063] Step 4.1 According to the test case, set the number of steps to N and the end cycle to K;
[0064] Step 4.2, perform the log analysis in step 3 on the first step of the first running cycle of the test case, and record the log analysis result ("passed the test" or "failed the test") in the log manager.
[0065] Step 4.3, traverse the N steps of the test case, and record the log analysis and judgment results of each step; and traverse the K running cycles of the test case, and record the log analysis results of each step in each running cycle.
[0066] Step 5: Determine the step results of the test case.
[0067] like Figure 5 As shown, the use case log analysis result recorded in step 4.3 is used as the input value to judge the detection result of the use case step. The specific steps include:
[0068] Step 5.1, filter the steps in the test case whose log analysis results are "passed". Traverse the log analysis judgment results of N steps in K running cycles, extract the steps whose log analysis results in step 4.3 are "passed", and record the step number and cycle number corresponding to the step.
[0069] Step 5.2, traverse each step of the test case in step 5.1, filter out the cycle number in each step whose log analysis result is "passed the test", and establish a set of cycle numbers in each step whose log analysis result is "passed the test"; determine whether the cycle number set of each step includes a cycle that satisfies the user-defined association relationship.
[0070] When the cycle number sets of N steps respectively include the cycles corresponding to the steps that satisfy the user-defined association relationship, the output result is "CaseResult=Pass", indicating that the test case passes the test; when the cycle number set of the steps does not include the cycle that satisfies the user-defined association relationship, the output result is "CaseResult=Fail", indicating that the test case fails the test.
[0071] The detection result of the test case (ie, "CaseResult=Pass" or "CaseResult=Fail") will be output through the log manager of the automatic log analysis system and fed back to the user to complete the test of the test case.
[0072] In summary, the present invention extracts key variables from the full OMAP data of the object under test, writes analysis scripts according to the requirements of the test case, and uses automatic analysis scripts to evaluate the log analysis results, and directly outputs the test status of the test case under test, which effectively reduces the workload of manual inspection, shortens the log analysis time, and significantly improves the test efficiency.
[0073] Although the content of the present invention has been described in detail through the above preferred embodiments, it should be appreciated that the above description should not be considered as a limitation of the present invention. After reading the above content, it will be apparent to those skilled in the art that various modifications and substitutions of the present invention will occur. Therefore, the protection scope of the present invention should be limited by the appended claims.
Claims
1. A keyword-based automatic log analysis method, characterized in that: By writing an automatic analysis script that can characterize and test the input and output relationship of the test case, the test case is tested using the automatic analysis script to obtain log analysis results, and then judging whether the test case passes the test by screening the log analysis results.
2. The keyword-based automatic log analysis method according to claim 1 is characterized in that: The automatic analysis script includes: a precondition script, a hit condition script, and an expected result; the precondition is an input stimulus, which can calculate the expected value based on the input and output relationship of the test case; the expected result is the value of the preselected data obtained after the input variable is calculated through the test case; the hit condition script can compare the expected value with the preselected data value to obtain the log analysis result.
3. The keyword-based automatic log analysis method according to claim 2 is characterized in that: Methods for using precondition scripts to calculate expected values and perform log analysis include: Mark the input variable related to the test case as the first key variable, and input the first key variable into the test case to obtain the expected result of the test case output; Mark the variable that represents the same content as the first key variable as the second key variable, and input the second key variable into the precondition script to calculate the expected value calculated by the program; The expected result is compared with the expected value calculated by the program through the hit condition script; when the expected result is consistent with the expected value calculated by the program, the output log analysis result is "passed the test".
4. The keyword-based automatic log analysis method according to claim 2 is characterized in that: The method of calling the automatic analysis script to obtain the log analysis result also includes: manually calculating the expected value and performing log analysis; the method of manually calculating the expected value and performing log analysis includes: Mark the input variable related to the test case as the first key variable, and input the first key variable into the test case to obtain the expected result of the test case output; Mark the variable with the same representation content as the first key variable as the second key variable, perform manual calculation on the second key variable according to the input and output relationship of the test case to obtain the manual calculation expected value, and input the manual calculation expected value into the automatic analysis script; The expected result is compared with the manually calculated expected value through the hit condition script; when the expected result is consistent with the manually calculated expected value, the output log analysis result is "passed the test".
5. The keyword-based automatic log analysis method according to claim 3 or 4, characterized in that: When the expected result is inconsistent with the expected value, the output log analysis result is "failed the test", and the fault value error processing is performed and the fault cause is output.
6. The keyword-based automatic log analysis method according to claim 5 is characterized in that: Methods for obtaining log analysis results of the test case include: The test case includes: a plurality of steps and a plurality of operation cycles; an automatic analysis script is called for each step in each operation cycle of the test case, and a log analysis result of each step in each operation cycle of the test case is recorded.
7. The keyword-based automatic log analysis method according to claim 6 is characterized in that: Methods for filtering log analysis results include: Filter the steps in each running cycle whose log analysis results are "passed test" and record the step number and cycle number of the step; Traverse each step, filter out the cycle number in each step whose log analysis result is "passed test", and establish a cycle number set in each step whose log analysis result is "passed test"; determine whether the cycle number set of each step includes a cycle that satisfies the user-defined association relationship; When the cycle number sets of all steps respectively include the cycles corresponding to the steps that satisfy the user-defined association relationship, the test case passes the test; when the cycle number set of a step does not include the cycle that satisfies the user-defined association relationship, the test case fails the test.
8. The keyword-based automatic log analysis method according to claim 1, characterized in that: The input variables related to the test cases are extracted from the OMAP dictionary data; wherein the OMAP dictionary data includes: all input variables and output variables required by the test object including several test cases, and the input and output variables of the test object are stored in the form of key-value pairs.
9. A system using the keyword-based automatic log analysis method according to claims 1-8, characterized in that: include: A simulator, wherein the simulator is an external stimulus of a test object including several test cases, and the variables of the external stimulus are stored in the memory of the automatic log analysis system in the form of key-value pairs. The simulator triggers variable assignment at a preset time point to achieve the purpose of inputting variables to the test object; A log manager, in which an automatic analysis script is run to detect the object under test; the log manager receives output variables sent by the object under test, stores and processes them in the form of key-value pairs, generates log analysis results, and determines whether the object under test passes the test.
10. The system for automatic log analysis based on keywords according to claim 9, characterized in that: The variables of the external excitation in the simulator are input and output variables related to the object under test.