Artificial intelligence-based test chain status tracking analysis method and system, and storage medium

Abstract

This application relates to the field of data processing technology, and in particular to a test chain state tracking and analysis method, system, and storage medium based on artificial intelligence. The test chain state tracking and analysis method includes: acquiring raw logs of continuous integration and continuous deployment corresponding to a target test chain; preprocessing the raw logs to obtain preprocessed log text; inputting the preprocessed log text into a large language model to obtain structured information representing the overall state of the target test chain; and updating the state information associated with the target test chain based on the structured information. This application can accurately and quickly extract complex information, solving the problem of unreliable extraction of complex information in existing technologies.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a test chain state tracking and analysis method, system and storage medium based on artificial intelligence. Background Technology

[0002] In the software development field, Continuous Integration and Continuous Deployment (CI / CD) is a crucial part of the software development process, used to automate code building, testing, and deployment. Complete end-to-end testing typically involves multiple sequentially executed CI / CD pipelines, such as from source build to test task generation, test execution, and result collection. However, existing test chain state tracking methods suffer from the following problems in practical applications:

[0003] First, status information is scattered and fragmented: Traditional methods typically monitor each CI / CD pipeline independently, obtaining its status (e.g., success, failure, running). Due to the lack of an automatic integration mechanism, the statuses of multiple related pipelines are scattered, making it difficult to form a unified global view of the test chain. This necessitates manually associating upstream and downstream relationships logically using identifiers such as build numbers when analyzing the status of the entire test chain, which is inefficient and prone to errors.

[0004] Secondly, information extraction is complex and fragile: When obtaining test chain state information, most methods rely on constructing log keyword searches or regular expression matching to extract key information. However, the constructed logs are usually unstructured text with complex and varied formats. Relying on regular expressions for information extraction is not only inefficient but also difficult to cover all format variations, resulting in high maintenance costs. Furthermore, this method cannot understand the semantics of the logs, limiting the accuracy and completeness of information extraction.

[0005] Third, insufficient real-time performance: It is impossible to obtain the current progress of the executing test chain in real time, which affects the ability of developers and testers to monitor and intervene in the testing process.

[0006] Fourth, the reason for termination is unclear: when the test chain is terminated, it is difficult to quickly and automatically determine the specific reason for the termination. For example, it is impossible to distinguish whether the test is due to a timeout or manual cancellation, and it is also impossible to record the operator's relevant information, which brings difficulties to the troubleshooting and analysis of the problem.

[0007] Fifth, lack of global interrelationships: In multi-pipeline testing environments, there are complex upstream and downstream dependencies between different pipelines. Existing technologies require manual sorting of these dependencies, making it difficult to automatically establish the interrelationships of the test chain, resulting in complex paths and low efficiency when tracing problems. Summary of the Invention

[0008] To address the aforementioned technical issues, this application provides a test chain state tracking and analysis method and system based on artificial intelligence, which can accurately and quickly extract complex information, thus solving the problem of complex and fragile information extraction in the prior art.

[0009] In a first aspect, this application provides an artificial intelligence-based test chain state tracking and analysis method, comprising: acquiring raw logs of continuous integration and continuous deployment corresponding to a target test chain; preprocessing the raw logs to obtain preprocessed log text; inputting the preprocessed log text into a large language model to obtain structured information representing the overall state of the target test chain; and updating the state information associated with the target test chain based on the structured information.

[0010] In some embodiments, the step of inputting the preprocessed log text into a large language model to obtain structured information representing the overall state of the target test chain includes: inputting the preprocessed log text into a preset prompt word template to generate prompt words; inputting the prompt words into the large language model to obtain structured information in a predetermined format based on the prompt words and representing the overall state of the target test chain.

[0011] In some embodiments, the prompt word template is used to guide the large language model to output a data object in a predetermined format, the data object including at least state information for identifying the overall state of the target test chain.

[0012] In some embodiments, the status information is determined according to a preset status judgment rule, the status judgment rule includes semantic logic, and the status information includes at least one of running, success, failure, timeout termination, and manual cancellation.

[0013] In some embodiments, the data object further includes: upstream association information, which is used to indicate the identifier of the upstream task or upstream test chain that triggers the execution of the target test chain.

[0014] In some embodiments, the method further includes: establishing an association between the target test chain and the upstream task or the upstream test chain based on the upstream association information, and updating the test chain topology view.

[0015] In some embodiments, the structured information includes: a test trigger type, which includes at least one of the following: anonymous trigger, user trigger, or upstream task trigger.

[0016] In some embodiments, the method further includes: establishing an association between the target test chain and the upstream task or the upstream test chain based on the upstream association information, and updating the test chain topology view.

[0017] In some embodiments, the method further includes: degradation processing, wherein when the large language model service is unavailable or analysis fails, a degradation parsing mode is switched; in the degradation parsing mode, the original log or the preprocessed log text is parsed using predefined regular expression rules to obtain the status information associated with the target test chain.

[0018] Secondly, this application also provides an artificial intelligence-based test chain state tracking and analysis system, applicable to any of the test chain state tracking and analysis methods described above, wherein the test chain state tracking and analysis system includes:

[0019] The log acquisition module is used to acquire the raw logs of continuous integration and continuous deployment corresponding to the target test chain;

[0020] The log preprocessing module is used to preprocess the original log to obtain preprocessed log text;

[0021] The intelligent analysis module is used to input the preprocessed log text into the large language model and obtain structured information representing the overall state of the target test chain;

[0022] The status management module is used to update the status information associated with the target test chain based on the structured information.

[0023] Thirdly, this application also provides a computer storage medium that stores a program or instructions that cause a computer to execute the test chain state tracking and analysis method described in the first aspect above.

[0024] The technical solution provided in this application has the following advantages compared with the prior art:

[0025] The AI-based test chain state tracking and analysis method provided in this application obtains the original logs corresponding to the target test chain, preprocesses the original logs, and then inputs the preprocessed original logs into a large language model to generate structured information (such as a unified test chain state view) representing the overall state of the test chain. This allows for accurate and rapid extraction of complex information, solving the problem of unreliable information extraction of complex information in existing technologies. Based on the structured information, the state information associated with the target test chain is updated, thereby improving the efficiency and accuracy of test chain state tracking and analysis.

[0026] Furthermore, the AI-based test chain status tracking and analysis method provided in this application constructs the relationship between upstream and downstream pipelines, simplifies the problem tracing path, and can automatically establish the relationship between test chains by inferring the build number and source project name of upstream and downstream tasks from the logs. Attached Figure Description

[0027] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0028] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, those skilled in the art can obtain other drawings based on these drawings without any creative effort.

[0029] Figure 1 A flowchart illustrating the AI-based test chain state tracking and analysis method provided in this application embodiment;

[0030] Figure 2 A logical diagram illustrating the AI-based test chain state tracking and analysis method provided in this application embodiment;

[0031] Figure 3 A schematic diagram of the structure of the AI-based test chain state tracking and analysis system provided in this application embodiment;

[0032] Figure 4 This is a schematic diagram of the structure of the computer storage medium provided in the embodiments of this application. Detailed Implementation

[0033] To better understand the above-mentioned objectives, features, and advantages of this application, the solution of this application will be further described below. It should be noted that, unless otherwise specified, the embodiments and features described in these embodiments can be combined with each other.

[0034] Many specific details are set forth in the following description in order to provide a full understanding of this application, but this application may also be implemented in other ways different from those described herein; obviously, the embodiments in the specification are only some embodiments of this application, and not all embodiments.

[0035] The following description, in conjunction with the accompanying drawings, provides an exemplary description of the artificial intelligence-based test chain state tracking and analysis method, system, and storage medium provided in the embodiments of this application.

[0036] Figure 1 This is a flowchart illustrating the AI-based test chain state tracking and analysis method provided in this application embodiment. Figure 2 A logical diagram illustrating the AI-based test chain state tracking and analysis method provided in this application embodiment. (Refer to...) Figure 1 and Figure 2 As shown, the test chain state tracking and analysis method 100 includes the following steps:

[0037] S101. Obtain the raw logs of continuous integration and continuous deployment corresponding to the target test chain.

[0038] Specifically, identify the target test chain to be analyzed and the specific build / deployment version (e.g., identified by a unique build number), and extract the original log file associated with that specific build number, containing detailed information about the entire build / deployment process, from the corresponding continuous integration and continuous deployment (CI / CD) system or log store.

[0039] The build number can be used as a query key to retrieve and extract raw log files associated with a specific build number from pre-defined CI / CD systems (such as Jenkins, GitLab CI / CD, AzureDevOps, etc.) or log repositories (such as ELK Stack, Splunk, log services provided by cloud service providers, etc.).

[0040] It is understood that the above description of obtaining raw logs is merely exemplary, and those skilled in the art can select and set the method for obtaining raw logs according to actual needs, as long as the technical principles of this application can be achieved.

[0041] S102. Preprocess the original log to obtain the preprocessed log text.

[0042] Specifically, the preprocessing process includes intelligent truncation of the raw logs, extracting key information segments, and removing redundant content. For example, the first 3000 characters can be extracted from the raw logs; this section typically contains trigger and start information for the test chain, which is crucial for understanding the initial conditions and context of the test. The last 8000 characters can also be extracted; this section usually contains test results and statistics, key data for evaluating test execution and final state. For excessively long sections of the log, which may contain a large amount of repetitive or non-critical information, these sections can be omitted, retaining only contextual hints, thereby reducing the overall log length and improving the efficiency of subsequent processing.

[0043] It is understood that the above description of the preprocessing of raw logs is merely exemplary, and those skilled in the art can select and set the preprocessing of raw logs according to actual needs, as long as the technical principles of this application can be achieved.

[0044] S103. Input the preprocessed log text into the large language model to obtain structured information representing the overall state of the target test chain.

[0045] Specifically, Large Language Models (LLMs) are natural language processing tools with powerful semantic understanding and text generation capabilities. When preprocessed log text is input into an LLM, it uses internal preset algorithms and pre-trained semantic knowledge to perform semantic parsing, identify key information, and transform this key information into a structured data format. This structured information can be a JSON object or another easily processed data format.

[0046] In some embodiments, the step of inputting the preprocessed log text into a large language model to obtain structured information representing the overall state of the target test chain includes: inputting the preprocessed log text into a preset prompt word template to generate prompt words; inputting the prompt words into the large language model to obtain structured information in a predetermined format based on the prompt words and representing the overall state of the target test chain.

[0047] Specifically, while the preprocessed log text has extracted key information, it remains in unstructured text format, hindering efficient state analysis. To accurately and efficiently extract and parse key information, prompts generated from pre-defined prompt templates are input into a large language model to obtain structured information in a predetermined format representing the overall state of the target test chain.

[0048] The large language model is a pre-trained model that can parse instructions and log content in prompt words, extract key information, and generate structured information according to the format specified in the prompt word template.

[0049] For example, the prompt word template is used to guide the large language model to output a data object in a predetermined format, the data object including at least state information for identifying the overall state of the target test chain.

[0050] Specifically, prompt word templates guide large language models through explicit instructions and a structured framework, specifying which key information to extract from log text and in what format to output that information. For example, prompt word templates might guide large language models to output a fixed-format JSON object. This JSON object could include, for example, status information identifying the overall state of the target test chain, as well as pipeline build numbers, test case statistics, trigger types, and so on.

[0051] In some embodiments, the status information is determined according to a preset status judgment rule, the status judgment rule includes semantic logic, and the status information includes at least one of running, success, failure, timeout termination, and manual cancellation.

[0052] Specifically, the state information can be determined according to preset state judgment rules, which, combined with semantic logic, accurately determine the overall state of the test chain. In particular, the state judgment rules are logical instructions embedded in the prompt word templates, used to guide the Large Language Model (LLM) to judge and output accurate state information based on the semantic content of the log text.

[0053] The state determination rule can be the following logic:

[0054] If the log contains information about an operation being performed (such as semantic information like "running" or "task in progress"), the status information is determined to be "running".

[0055] If the log contains semantic information such as "execution completed" or "success", and the test case pass rate exceeds a predetermined value (such as 95%), then the status information is judged as "successful".

[0056] If the log contains semantic information such as "execution failed" or "error", the status information is judged as "failure";

[0057] If the log contains information indicating that the test stopped due to exceeding the preset time, the status information is judged as "timeout aborted";

[0058] If the log contains semantic information such as "Aborted by [user]" or "manually canceled", the status information is judged as "manually canceled".

[0059] In some embodiments, the data object may further include: upstream association information, which is used to indicate the identifier of the upstream task or upstream test chain that triggers the execution of the target test chain.

[0060] Specifically, data objects can also include upstream association information, which can clearly define the source and dependencies of the target test chain. For example, in a continuous integration / continuous deployment (CI / CD) environment, the execution of a test chain may be triggered by upstream tasks (such as code submissions, build tasks, or other test chains). By using preset regular expression patterns, the build number, source project name, and other information of the upstream task generation (GenTaskFile) and test steps (TestStep) can be automatically extracted from the logs to automatically establish the association relationship of the test chain.

[0061] In some embodiments, the structured information may include: a test trigger type, which includes at least one of the following: anonymous trigger, user trigger, or upstream task trigger.

[0062] Specifically, the test trigger type is used to specify how the test chain is initiated. The test trigger type includes at least one of the following: anonymous trigger, user trigger, or upstream task trigger.

[0063] Anonymous triggering refers to a test chain starting not directly by a specific user or upstream task, but through some automated mechanism or system default behavior. For example, some test chains may start automatically when the system detects specific conditions (such as scheduled tasks or automatic triggering mechanisms in the code repository).

[0064] User-triggered test chains are test chains manually initiated by a specific user. In this case, the execution of the test chain can be explicitly initiated by developers, testers, or other authorized users through interface operations, command-line commands, or other interactive methods. User-triggered test chains are typically used for verification in specific scenarios, such as testing new features or fixes during development.

[0065] Upstream task triggering refers to a test chain being triggered by other tasks (such as build tasks, deployment tasks, or other test chains). In a CI / CD pipeline, test chains typically depend on the output of upstream tasks; for example, a test task might be automatically triggered after a successful code build. Upstream task-triggered test chains reflect the dependencies between test chains, facilitating the construction of a complete test flow view. By recording the identifiers of upstream tasks (such as build number, task ID, etc.), the origin of the test chain can be traced.

[0066] S104. Update the status information associated with the target test chain based on the structured information.

[0067] Specifically, to enable real-time monitoring and dynamic management of the test chain status, structured information generated by a large language model is used to update and maintain the status of the target test chain, thereby ensuring that the current status of the test chain is accurately reflected. For example, status fields (such as "success", "failure", "timeout") in the structured information can be updated in the status record of the target test chain.

[0068] Furthermore, the test chain state tracking and analysis method may also include: establishing an association between the target test chain and the upstream task or the upstream test chain based on the upstream association information, and updating the test chain topology view.

[0069] Specifically, the test chain state tracking and analysis method provided in this application can also establish the relationship between the target test chain and upstream tasks or upstream test chains based on upstream association information, and update the topology view of the test chains according to the relationship. This topology view can intuitively display the hierarchical structure and dependencies between test chains, helping developers and testers quickly understand the flow logic of the entire testing process. By automatically identifying and recording the identifiers of upstream tasks or test chains, the triggering source and execution path of the test chain can be displayed.

[0070] Furthermore, the test chain state tracking and analysis method may also include: degradation processing, wherein when the large language model service is unavailable or analysis fails, switching to degradation parsing mode; in the degradation parsing mode, parsing the original log or the preprocessed log text using predefined regular expression rules to obtain the state information associated with the target test chain.

[0071] Specifically, degradation processing ensures that the system can still function normally and obtain status information of the target test chain even when the Large Language Model (LLM) service is unavailable or analysis fails. In this scenario, predefined regular expression rules can be used to parse the raw logs or preprocessed log text. These predefined regular expression rules are designed based on common log formats and key information, and can match and extract fields such as status information, trigger type, and upstream association information. For example, regular expressions can match keywords such as "SUCCESS," "FAILED," and "Aborted by [user]" in the logs to determine the status of the target test chain.

[0072] Furthermore, the analysis results can be persistently cached. For test chains in the "running" state, the status information can be cached for a short period (e.g., 60 seconds). Since the status of a running test chain may change frequently (e.g., switching from one test case to the next), a short cache time allows for timely updates to the status information, providing users with the latest test progress.

[0073] For completed test chains (such as success, failure, timeout, etc.), their status information can be cached for a relatively long time (e.g., 5 minutes). Completed states usually do not change, so a longer caching time can reduce repeated queries to the original logs or large language models, reducing computational and storage burdens.

[0074] In addition, it supports historical query functionality, allowing users to check the status information of test chains at a specific point in time, facilitating troubleshooting, performance analysis, and historical data comparison. For example, users can view the status changes of a test chain over the past week and analyze the trend of failure rate changes.

[0075] Figure 2 A logical diagram illustrating the AI-based test chain state tracking and analysis method provided in this application embodiment. The following is in conjunction with... Figure 2 This paper provides a detailed explanation of the test chain state tracking and analysis method based on artificial intelligence.

[0076] Reference Figure 2 First, the testchain build number is input. Based on this number, the relevant raw logs are retrieved and preprocessed (e.g., intelligent truncation), extracting the log portions containing key information. Structured hints are then constructed to guide the large language model in semantic analysis. The large language model can determine whether to parse the logs and return the parsed results (e.g., JSON objects) or to use a regular expression-based fallback method to extract key information. Key information may include, for example, the testchain status, upstream task information, test summary, and trigger and termination reasons. This key information is used to generate a complete testchain status object. This testchain status object can be cached and stored to support fast access and historical queries. The output structured status information can be, for example, a unified testchain status view, providing users with a clear and real-time overview of the testchain status.

[0077] In summary, the AI-based test chain state tracking and analysis method 100 provided in this application obtains the original logs corresponding to the target test chain, preprocesses the original logs, and then inputs the preprocessed original logs into a large language model to generate structured information (such as a unified test chain state view) representing the overall state of the test chain. This allows for accurate and rapid extraction of complex information, solving the problem of unreliable extraction of complex information in existing technologies. Based on the structured information, the state information associated with the target test chain is updated, thereby improving the efficiency and accuracy of test chain state tracking and analysis.

[0078] Furthermore, the AI-based test chain status tracking and analysis method provided in this application constructs the relationship between upstream and downstream pipelines, simplifies the problem tracing path, and can automatically establish the relationship between test chains by inferring the build number and source project name of upstream and downstream tasks from the logs.

[0079] Based on the same inventive concept, this application also provides an artificial intelligence-based test chain state tracking and analysis system 200. Figure 3 This is a schematic diagram of the structure of the AI-based test chain state tracking and analysis system provided in this application embodiment. Figure 3 As shown, the AI-based test chain state tracking and analysis system 200 includes:

[0080] Log acquisition module 201 is used to acquire the raw logs of continuous integration and continuous deployment C corresponding to the target test chain;

[0081] Log preprocessing module 202 is used to preprocess the original log to obtain preprocessed log text;

[0082] The intelligent analysis module 203 is used to input the preprocessed log text into the large language model and obtain structured information representing the overall state of the target test chain;

[0083] The state management module 204 is used to update the state information associated with the target test chain based on the structured information.

[0084] The test chain state tracing analysis system provided in the above embodiments can execute the test chain state tracing analysis method provided in the above embodiments and has the same or corresponding beneficial effects, which will not be described in detail here.

[0085] This application also provides a computer storage medium 300, Figure 4 This is a schematic diagram of the structure of the computer storage medium provided in an embodiment of this application. The computer storage medium 300 stores a computer program 301, which, when executed by a processor, can implement the test chain state tracking and analysis method described in any of the embodiments above in this application.

[0086] It should be understood that, in this embodiment, the aforementioned computer storage medium may be located at at least one of the multiple network servers in a computer network. Optionally, in this embodiment, the aforementioned storage medium may include, but is not limited to, various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0087] It should be noted that the order of the embodiments described above is merely for descriptive purposes and does not represent the superiority or inferiority of the embodiments.

[0088] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of those different embodiments or examples.

[0089] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "a plurality of" means two or more, unless otherwise explicitly specified.

[0090] In this application, unless otherwise stated, directional terms such as "up" and "down" are generally used in relation to the direction shown in the accompanying drawings, or in relation to the vertical, perpendicular, or gravitational direction; similarly, for ease of understanding and description, "left" and "right" are generally used in relation to the left and right shown in the accompanying drawings; "inner" and "outer" refer to the inner and outer contours of each component itself, but the above directional terms are not intended to limit this application.

[0091] The above description is merely an exemplary embodiment of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various variations or substitutions within the technical scope described in this application, and these should all be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An artificial intelligence-based test chain status tracking analysis method, characterized by, include: Obtain the raw logs of continuous integration and continuous deployment corresponding to the target test chain; The original log is preprocessed to obtain the preprocessed log text; The preprocessed log text is input into a large language model to obtain structured information representing the overall state of the target test chain; Update the state information associated with the target test chain based on the structured information.

2. The test chain status tracking analysis method of claim 1, wherein, The step of inputting the preprocessed log text into a large language model to obtain structured information representing the overall state of the target test chain includes: Input the preprocessed log text into a preset prompt word template to generate prompt words; The prompt words are input into the large language model to obtain structured information in a predetermined format that represents the overall state of the target test chain based on the prompt words.

3. The test chain state tracking and analysis method according to claim 2, characterized in that, The prompt word template is used to guide the large language model to output a data object in a predetermined format. The data object includes at least state information used to identify the overall state of the target test chain.

4. The test chain status tracking analysis method of claim 3, wherein, The status information is determined according to a preset status judgment rule, which includes semantic logic, and the status information includes at least one of running, success, failure, timeout termination, and manual cancellation.

5. The test chain status tracking analysis method of claim 4, wherein, The data object also includes: upstream association information, which is used to indicate the upstream task or the identifier of the upstream test chain that triggered the execution of the target test chain.

6. The test chain status tracking analysis method of claim 5, wherein, The method further includes: Based on the upstream association information, establish the association between the target test chain and the upstream task or the upstream test chain, and update the test chain topology view.

7. The test chain status tracking analysis method of claim 6, wherein, The structured information includes: Test trigger type, which includes at least one of the following: anonymous trigger, user trigger, or upstream task trigger.

8. The test chain status tracking and analysis method of claim 1, wherein, The method further includes: degradation processing, wherein... When the large language model service is unavailable or analysis fails, switch to degraded parsing mode; In the downgraded parsing mode, the original log or the preprocessed log text is parsed using predefined regular expression rules to obtain the status information associated with the target test chain.

9. An artificial intelligence-based test chain state tracking and analysis system, applicable to the test chain state tracking and analysis method according to any one of claims 1 to 8, wherein the test chain state tracking and analysis system comprises: The log acquisition module is used to acquire the raw logs of continuous integration and continuous deployment corresponding to the target test chain; The log preprocessing module is used to preprocess the original log to obtain preprocessed log text; The intelligent analysis module is used to input the preprocessed log text into the large language model and obtain structured information representing the overall state of the target test chain; The status management module is used to update the status information associated with the target test chain based on the structured information.

10. A computer storage medium, characterized in that, The computer storage medium stores a program or instructions that cause the computer to execute the test chain state tracking and analysis method as described in any one of claims 1 to 8.

Images